Lane line detection method and system, computer equipment and vehicle

Abstract

The invention provides a lane line detection method and system, computer equipment and a vehicle, and the method comprises the steps: obtaining a target image and three-dimensional point cloud data, carrying out the grouping of lane line segments in the target image according to the positive and negative slope, and carrying out the color marking according to a grouping result, color features and gradient features are extracted from the target image after color marking is completed, pixel points with the color features and the gradient features at the same time serve as target pixel points, and lane line candidate points are formed according to the target pixel points; and based on the lane line candidate points and the three-dimensional point cloud data, performing inverse perspective transformation on the target image to obtain a bird's-eye view image, and determining a lane line position according to the bird's-eye view image. According to the method, the lane line detection precision can be improved, the potential lane lines are identified according to the color features and the gradient features, higher robustness can be achieved for illumination changes, local shielding and ground noise, the method can adapt to lane line detection in a complex environment, and therefore the detection noise is effectively reduced.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of intelligent driving, and particularly relates to a lane line detection method and system, a computer device and a vehicle. BACKGROUND

[0002] According to relevant investigations, about 50% of car accidents are caused by vehicles not driving in the specified lane. Therefore, an advanced driving assistance system (ADAS) is increasingly concerned by researchers, such as a lane departure warning, a lane keeping, an adaptive cruise control and the like. In the advanced driving assistance system, lane line detection is essential.

[0003] Lane detection is an important part of a vehicle auxiliary driving system, but lane line detection is still a challenging task in an environment of road signs or vehicle interference. Especially for different weather and driving environments, such as in rain, snow, fog, haze, strong light, shadow shielding and lane line wear and the like, the performance of existing related detection methods sharply decreases, and it is difficult to meet the stringent requirements of reliability of a high-level intelligent driving system. Therefore, how to detect lane lines in a complex environment is still a great challenge. SUMMARY

[0004] The present application provides a lane line detection method and system, a computer device and a vehicle to solve or alleviate the above-described problems.

[0005] The lane line detection method provided by the present application comprises the following steps: obtaining a target image and three-dimensional point cloud data corresponding to the target image, the target image comprising lane line segments of different slopes; grouping the lane line segments according to the positive and negative of the slopes, and color marking the lane line segments in the target image according to the grouping result, so that the lane line segments of different groups have different colors; extracting color features and gradient features from the target image after color marking, taking the pixel points with both color features and gradient features as target pixel points, and forming lane line candidate points according to the target pixel points; performing inverse perspective transformation on the target image based on the lane line candidate points and the three-dimensional point cloud data, obtaining a bird's eye view image corresponding to the target image, and determining a lane line position according to the bird's eye view image.

[0006] In an embodiment of the present application, the process of determining the lane line position according to the bird's eye view image comprises: performing an inverse projection transformation on the bird's eye view image, and dividing the bird's eye view image after the inverse projection transformation into a plurality of block images with the same area; wherein the dividing comprises horizontal dividing and / or vertical dividing; generating a corresponding pixel histogram based on pixel distribution of the block image, and taking a position of a highest point of the pixel histogram as a lane line position.

[0007] In an embodiment of the present application, before generating a corresponding pixel histogram based on pixel distribution of the block image, the method further comprises: calculating a proportion of effective pixels in each block image, and recording a corresponding block image as an abnormal block image when the proportion is greater than a preset proportion threshold, and replacing the abnormal block image with an adjacent block image; wherein the effective pixels include pixels corresponding to lane line segments.

[0008] In an embodiment of the present application, the method further comprises: performing clustering and fitting based on the block images, and distributing continuous or discontinuous lane lines into one category; According to the clustering and fitting results of the block images, taking the current frame target image as prior knowledge of the next frame target image, and performing lane line tracking based on the prior knowledge and received lane line prior information; wherein the lane line prior information is provided by a roadside unit.

[0009] In an embodiment of the present application, after obtaining the target image, the method further comprises: taking the lane line segments as straight lines, and allowing all straight lines in one group to intersect with all straight lines in another group according to the grouping result to determine all intersection points; calculating a distance between each intersection point and each straight line, and selecting a shortest intersection point as a vanishing point, and taking a vertical coordinate of the vanishing point in an image coordinate system as a vanishing line; wherein the image coordinate system is determined based on the target image; performing denoising on the target image based on the vanishing point and the vanishing line, and deleting all or part of the line segments in the target image that have no association with lane lines.

[0010] In an embodiment of the present application, the process of color labeling the lane line segments in the target image according to the grouping result comprises: obtaining weather data through a sunlight and rainfall sensor, and dynamically adjusting a color threshold and a contrast enhancement parameter based on the weather data; color labeling the lane line segments in the target image according to the grouping result, the color threshold and the contrast enhancement parameter.

[0011] In an embodiment of the present application, after the lane line position is determined according to the bird's-eye view image, the method further comprises: in response to an automatic driving request or an assisted driving request, performing path planning based on the determined lane line position, and controlling the current vehicle to automatically drive or assist driving according to the path planning result.

[0012] The present application also provides a lane line detection system, which comprises: a data acquisition module configured to acquire a target image and three-dimensional point cloud data corresponding to the target image, the target image comprising lane line segments with different slopes; a color marking module configured to group the lane line segments according to the positive and negative of the slope, and mark the lane line segments in the target image with different colors according to the grouping result, so that the lane line segments in different groups have different colors; a feature extraction module configured to extract color features and gradient features from the target image after color marking, and take the pixel points with both color features and gradient features as target pixel points, and form lane line candidate points according to the target pixel points; a lane line detection module configured to perform inverse perspective transformation on the target image according to the lane line candidate points and the three-dimensional point cloud data, to obtain a bird's-eye view image corresponding to the target image, and determine a lane line position according to the bird's-eye view image.

[0013] The present application also provides a computer device comprising a memory, a processor and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the lane line detection method according to any one of the above embodiments.

[0014] The present application also provides a computer readable storage medium having a computer program stored thereon, wherein the computer program is executed by a processor to implement the steps of the lane line detection method according to any one of the above embodiments.

[0015] The present application also provides a vehicle comprising the lane line detection system as described above, or comprising the computer device as described above.

[0016] The beneficial effects of the present application: the present application provides a lane line detection method and system, a computer device and a vehicle, by acquiring a target image and three-dimensional point cloud data corresponding to the target image, the texture and color information in the target image, the distance and spatial coordinates in the three-dimensional point cloud data can be fully utilized in the lane line detection process, so as to complement the target image and the three-dimensional point cloud data, greatly improving the lane line detection accuracy. At the same time, by extracting the color feature and gradient feature of the lane line segment, and then identifying the potential lane line in the target image according to the color feature and gradient feature, the double feature fusion mechanism not only has stronger robustness to light change, local shielding and ground noise in the lane line detection process, but also can adapt to lane line detection in complex environment, so as to effectively reduce detection noise. And the present application constructs a three-dimensional lane line topology structure by fusing the image and the three-dimensional point cloud data, and generates lane line candidate points in three-dimensional space, and then generates a bird's eye view image by inverse perspective transformation of the target image to determine the lane line position, further improving the lane line recognition accuracy. In addition, by combining the lane line prior information provided by the roadside unit in the lane line detection process, the lane line detection result can be continuously and stably output. BRIEF DESCRIPTION OF DRAWINGS

[0017] The drawings incorporated into the specification and forming a part thereof, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the application. It is to be expressly understood that the drawings are only some embodiments of the present application, and other drawings can be obtained from these drawings without creative labor for those skilled in the art.

[0018] In the drawings: Figure 1 The flowchart of the lane line detection method provided by an embodiment of the present application is shown in the figure; Figure 2 The image after color labeling provided by an embodiment of the present application is shown in the figure; Figure 3 The image after preprocessing provided by an embodiment of the present application is shown in the figure; Figure 4 The image after HSV color space conversion provided by an embodiment of the present application is shown in the figure Figure 5 The image when gradient feature extraction is performed is shown in the figure provided by an embodiment of the present application; Figure 6 The image fused with three-dimensional point cloud data provided by an embodiment of the present application is shown in the figure; Figure 7 The bird's eye view image provided by an embodiment of the present application is shown in the figure; Figure 8This is a schematic diagram of a segmented image provided in one embodiment of this application; Figure 9 A schematic diagram of the hardware structure of a lane detection system provided in an embodiment of this application; Figure 10 This is a schematic diagram of the hardware structure of a vehicle provided in one embodiment of this application; Figure 11 This is a schematic diagram of the architecture of a lane line perception fusion controller provided in one embodiment of this application; Figure 12 This is a schematic diagram of the hardware structure of a computer device suitable for implementing one or more embodiments of this application. Detailed Implementation

[0019] The following specific examples illustrate the implementation of this application. Those skilled in the art can easily understand other advantages and effects of this application from the content disclosed in this specification. This application can also be implemented or applied through other different specific embodiments. Various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of this application. In the absence of conflict, the following embodiments and features in the embodiments can be combined with each other.

[0020] It is understood that the illustrations provided in the following embodiments are only schematic representations of the basic concept of this application. The drawings only show the components related to this application and are not drawn according to the actual number, shape and size of the components in the actual implementation. In the actual implementation, the form, quantity and proportion of each component can be arbitrarily changed, and the layout of the components may also be more complex.

[0021] In the following description, numerous details are explored to provide a more thorough explanation of embodiments of the present application. However, it will be apparent to those skilled in the art that embodiments of the present application may be practiced without these specific details. In other embodiments, well-known structures and devices are shown in block diagram form rather than in detail to avoid obscuring embodiments of the present application.

[0022] Figure 1 A flowchart illustrating a lane line detection method is shown. Specifically, in an exemplary embodiment, as... Figure 1 As shown, this embodiment provides a lane line detection method, including the following steps: S110, acquire the target image and the corresponding 3D point cloud data. The target image includes lane line segments with different slopes. S120, group the lane line segments according to the positive or negative of the slope, and color-mark the lane line segments in the target image according to the grouping results so that the lane line segments in different groups have different colors; S130, extracting color features and gradient features from the target image after color marking is completed respectively, taking the pixel points with both color features and gradient features as target pixel points, and forming lane line candidate points according to the target pixel points; S140, based on the lane line candidate points and the three-dimensional point cloud data, performing inverse perspective transformation on the target image to obtain a bird's eye view image corresponding to the target image, and determining the lane line position according to the bird's eye view image.

[0023] In some example embodiments, the process of step S110 acquiring the target image can include: capturing real-time video streams or images during the driving process of the vehicle through the vehicle-mounted camera, and selecting images including lane line segments with different slopes as target images from the real-time video streams or images.

[0024] In some example embodiments, since the images captured by the vehicle-mounted camera contain not only lane lines but also a large number of invalid areas such as sky, trees, buildings, etc. These invalid areas account for almost half of the image area, and the lane line area becomes difficult to identify. Therefore, in order to better extract lane information, the target image can be preprocessed to improve the lane detection accuracy. Since the lane line or lane line segment can be regarded as a straight line in a two-dimensional image, the lane line segments can be divided into two groups according to the positive and negative slopes of the segments, then the segments with slopes within a certain range are selected, and then the lane line segments are marked with different colors according to the grouping results. In some examples, the length of the lane line segment and the distance of the lane line segment to the vertical center line of the target image can be selected as the target for selecting the lane line segment. In addition, in order to avoid color recognition errors under different weather conditions, such as rainy days, cloudy days, etc., weather data can be obtained through a sunlight and rainfall sensor when color marking is performed, and then the color threshold and the contrast enhancement parameter are dynamically adjusted based on the weather data; then the lane line segments in the target image are color marked according to the grouping results, the color threshold and the contrast enhancement parameter, so as to ensure the stability of color marking under different lighting conditions. In addition, the weather data obtained by the sunlight and rainfall sensor can also be used to adaptively adjust the length of the lane line segment and the distance of the lane line segment to the vertical center line of the target image to cope with the interference of rain on the edge features of the image. As an example, the image after marking the lane line segment is as shown in Figure 2

[0025] ​In some example embodiments, the process of pre-processing the target image can include: taking the lane line segments as straight lines, and according to the grouping result, making all straight lines of one group intersect with all straight lines of another group to determine all intersection points; calculating the distance of each intersection point to each straight line, and selecting the intersection point with the shortest distance as the vanishing point, and taking the ordinate of the vanishing point in the image coordinate system as the vanishing line; wherein the image coordinate system is determined based on the target image; and based on the vanishing point and the vanishing line, performing denoising on the target image, and deleting all or part of the line segments in the target image that are not associated with the lane lines. Specifically, taking Figure 2 for example, the process of deleting all or part of the line segments in Figure 2 that are not associated with the lane lines can be: ; formula (1) ; formula (2) In formula (1) and formula (2), represents the slope, represents the abscissa of the th lane line segment in the first group, represents the ordinate of the th lane line segment in the first group, represents the abscissa of the th lane line segment in the second group, represents the ordinate of the th lane line segment in the second group, represents the influence factor of the distance of the th lane line segment to the center of the target image, and are both preset constants; represents the length of the th lane line segment, represents the length of the th lane line segment.

[0026] After pre-processing, Figure 2 the red vanishing point is accurately detected, and most of the irrelevant line segments in the image are removed. The image after removing the vanishing point is shown in Figure 3 .

[0027] In some example embodiments, in the target image, the lane lines are usually white or yellow, arranged regularly, with clear edges, and form a strong contrast with the surrounding environment. In view of these characteristics, two feature extraction methods can be used to extract the lane lines. Therefore, the process of step S120 of extracting color features and gradient features from the target image after color labeling, and taking the pixel points with both color features and gradient features as target pixel points, and forming lane line candidate points according to the target pixel points can include: Since HSV images are easier to extract the desired color, in order to better extract the white line or yellow line from the image, the RGB image can be converted to the HSV color space. For example, taking Figure 3 as an example, Figure 4 , Figure 3 the image after HSV color space conversion.

[0028] In addition to the lane line, there is also a similar color of the extracted lane line color, so the lane line edge can be extracted based on the gradient method, but since the edge information extracted by the gradient is very narrow, morphological expansion can be performed on it. In some examples, the Canny algorithm can be used for lane line gradient feature extraction. For example, taking Figure 4 as an example, Figure 5 shows the gradient detection result of Figure 4 .

[0029] Since both color features and gradient features will contain a lot of noise, if the two color features and gradient features are directly added, although the lane features will be increased, more noise will be superimposed, which will greatly increase the difficulty of subsequent work. Therefore, in order to be able to eliminate the noise of color features and gradient features from each other, rather than superimposing noise, color features and gradient features can be extracted respectively, and then the intersection of the pixel points is taken as a new feature map. This process greatly eliminates noise, because only the pixel points that have both color features and gradient features will be retained. Generally, the pixels detected based on gradient features are the edges of objects, which are usually a combination of slender pixels. Therefore, in order to increase the number of lane line gradient features, morphological dilation can be performed on the pixels based on the gradient features, thereby greatly increasing the number and range of pixel points. Then the pixel points that have both color features and gradient features are taken as target pixel points, and lane line candidate points are formed according to all target pixel points. Since the feature fusion process is very strict, when extracting color features and gradient features, the restrictions can be appropriately relaxed to avoid too few or even no features being extracted. The specific process of feature extraction can be referred to related technologies, which will not be described here.

[0030] In some example embodiments, since the de-perspective transformation can remove the perspective effect, the originally parallel lane lines can be represented. Therefore, in order to better cluster and fit the lane lines, the image area containing the lane lines in the image can be taken as a region of interest, and then the region of interest is de-perspective transformed based on the pixel points to obtain the bird's eye view image of the image. Specifically, the process of de-perspective transforming the target image based on the lane line candidate points and the three-dimensional point cloud data to obtain the bird's eye view image corresponding to the target image can include: combining the image with the three-dimensional point cloud data provided by the millimeter wave radar, after the two-dimensional coordinates of the image and the three-dimensional point cloud data provided by the millimeter wave radar are time and space aligned, the two are jointly input into a fusion core (such as a Kalman filter), the image provides rich texture and color information, and the three-dimensional point cloud data provides accurate distance and spatial coordinates, the two complement each other and jointly construct a more accurate three-dimensional lane line topology. The image of fusing the two-dimensional coordinates of the image and the three-dimensional point cloud data of the millimeter wave radar is shown in Figure 6 , the bird's eye view image obtained by de-perspective transforming based on Figure 6 is shown in Figure 7 . Therefore, the noise of the image formed by simultaneously extracting and fusing color features and gradient features is significantly reduced, thereby greatly improving the lane line detection accuracy.

[0031] In some example embodiments, the process of determining the lane line position according to the bird's eye view image includes: de-projecting the bird's eye view image, and dividing the bird's eye view image after de-projecting into a plurality of block images with the same area; wherein the division includes horizontal division and / or vertical division; generating a corresponding pixel histogram based on the pixel distribution of the block image, and taking the position of the highest point of the pixel histogram as the lane line position. In addition, before generating the corresponding pixel histogram based on the pixel distribution of the block image, it can also include: calculating the proportion of effective pixels in each block image, and when the proportion is greater than a preset proportion threshold, taking the corresponding block image as an abnormal block image, and replacing the abnormal block image with adjacent block images; wherein the effective pixels include the pixels corresponding to the lane line segments. In some examples, since the bird's eye view image after de-projecting still contains some noise, in order to eliminate this noise and extract more accurate lane lines, the image can be divided into four equal block images, as shown in Figure 8 , and Figure 8 shows Figure 7the image segmentation result. After the image is segmented, the curvature of the lane line in each image is reduced and tends to be straighter, and its representation on the pixel histogram is also more dense. Therefore, the position of the highest point of the pixel histogram can be regarded as the position of the lane line, and by selecting the pixel point of the highest point of the pixel histogram, the possibility of extracting noise can be greatly reduced. In addition, under normal circumstances, the amount of lane line pixels in each line segment is roughly the same, but in special cases, such as words and signs on the road, which have the same characteristics as the lane line, they will also be extracted, and the total number of pixels in the line segment is much larger than that in other line segments. When this happens, the adjacent line segment can be used for replacement. Since the lane line is continuous and equally divided, the change of each segment is very limited, and the error before and after replacement can be ignored. In addition, the description of the pixel points of the block image can be as follows: Equation (3) In equation (3), represents the proportion of valid pixels contained in the segmented block image, represents the valid pixels contained in each block image, represents the valid pixels contained in the th lane line segment of each block image, represents the number of segmented block images. When is greater than a predetermined value, the corresponding block image will be replaced by the adjacent block image.

[0032] In some example embodiments, the lane line detection method can further include: based on the clustering and fitting of the block images, assigning continuous or discontinuous lane lines into a category; based on the clustering and fitting results of the block images, taking the current frame target image as prior knowledge of the next frame target image, and performing lane line tracking based on the prior knowledge and received lane line prior information; wherein the lane line prior information is provided by a road side unit. In some examples, lane line pixels are regularly clustered and arranged, so this property can be used to remove noise, for example, selecting DBSCAN as a clustering method for lane line clustering and fitting. DBSCAN is a density-based clustering method that does not need to specify the number of categories, and secondly it can find clusters of arbitrary shape, and by setting reasonable parameters it can effectively assign continuous or discontinuous lane lines to a category. Since lane line detection is a continuous process, the lane line curvature between adjacent two frames changes little or not at all, so the previous frame image can provide prior knowledge for the next frame image. When performing lane line tracking, Kalman filtering or particle filtering can be used. At the same time, lane line prior information (such as curvature, position) provided by a road side unit (RSU) through C-V2X (Cellular-Vehicle-to-Everything, C-V2X) can be used to verify and guide the above image recognition results. For example, when the performance of lane line detection through images is degraded due to strong glare, more reliable prior information provided by V2X (Vehicle to everything, V2X) can be relied on to maintain continuous and stable output of lane lines.

[0033] In some example embodiments, after determining the lane line position according to the bird's eye view image, the lane line detection method can further include: in response to an autonomous driving request or an assisted driving request, performing path planning based on the determined lane line position, and controlling the current vehicle to perform autonomous driving or assisted driving according to the path planning result. The specific process of path planning based on the lane line position can be referred to related technologies, which will not be described here.

[0034] In summary, the present application proposes a lane line detection method, which obtains a target image and three-dimensional point cloud data corresponding to the target image, the target image including lane line segments with different slopes; then groups the lane line segments according to the positive and negative of the slopes, and color marks the lane line segments in the target image according to the grouping result, so that the lane line segments in different groups have different colors; then extracts color features and gradient features from the target image after color marking, and takes the pixel points with both color features and gradient features as target pixel points, and forms lane line candidate points according to the target pixel points; then performs inverse perspective transformation on the target image based on the lane line candidate points and the three-dimensional point cloud data, obtains a bird's eye view image corresponding to the target image, and determines the lane line position according to the bird's eye view image. It can be seen that, by obtaining the target image and the three-dimensional point cloud data corresponding to the target image, the present method can make full use of the texture and color information in the target image, and the distance and spatial coordinates in the three-dimensional point cloud data in the lane line detection process, so as to fuse and complement the target image and the three-dimensional point cloud data, greatly improving the lane line detection precision. At the same time, by extracting the color features and gradient features of the lane line segments, and then identifying the potential lane lines in the target image according to the color features and gradient features, the double-feature fusion mechanism not only has stronger robustness to light changes, local occlusions and ground noise in the lane line detection process, but also can adapt to lane line detection in complex environments, thereby effectively reducing detection noise. Moreover, the present method fuses the image and the three-dimensional point cloud data to construct a three-dimensional lane line topology structure, and generates lane line candidate points in a three-dimensional space, then generates a bird's eye view image by inverse perspective transformation on the target image to determine the lane line position, further improving the lane line recognition accuracy. In addition, by combining the lane line prior information provided by the roadside unit in the lane line detection process, the lane line detection result can be continuously and stably output.

[0035] In another example embodiment of the present application, as shown in Figure 9 The embodiment also provides a lane line detection system, which includes: The data acquisition module 910 is configured to obtain a target image and three-dimensional point cloud data corresponding to the target image, the target image including lane line segments with different slopes; The color marking module 920 is configured to group the lane line segments according to the positive and negative of the slopes, and color mark the lane line segments in the target image according to the grouping result, so that the lane line segments in different groups have different colors; The feature extraction module 930 is configured to extract color features and gradient features from the target image after color marking, and take the pixel points with both color features and gradient features as target pixel points, and form lane line candidate points according to the target pixel points; The lane line detection module 940 is configured to perform inverse perspective transformation on the target image according to the lane line candidate points and the three-dimensional point cloud data, to obtain a bird's eye view image corresponding to the target image, and determine the lane line position according to the bird's eye view image.

[0036] It can be understood that the lane line detection system provided by the above-mentioned embodiments and the lane line detection method provided by the above-mentioned embodiments belong to the same concept, wherein the specific manner in which the lane line detection method performs operations has been described in detail in the above-mentioned method embodiments, which will not be repeated here. In actual application, the lane line detection system provided by the above-mentioned embodiments can assign the above-mentioned functions to be completed by different functional modules according to needs, that is, the internal structure of the lane line detection system is divided into different functional modules, and then all or part of the functions of the corresponding functional modules are implemented by the lane line detection method described in the above-mentioned embodiments, for example, all or part of the functions of the data acquisition module 910 can be implemented by the relevant execution process of step S110, all or part of the functions of the color marking module 920 can be implemented by the relevant execution process of step S120, all or part of the functions of the feature extraction module 930 can be implemented by the relevant execution process of step S130, and all or part of the functions of the lane line detection module 940 can be implemented by the relevant execution process of step S140. This will not be specifically limited here.

[0037] In summary, the present application provides a lane line detection system, which obtains a target image and three-dimensional point cloud data corresponding to the target image, the target image including lane line segments with different slopes; then groups the lane line segments according to the positive and negative of the slopes, and color marks the lane line segments in the target image according to the grouping result, so that the lane line segments in different groups have different colors; then extracts color features and gradient features from the target image after color marking, and takes the pixel points with both color features and gradient features as target pixel points, and forms lane line candidate points according to the target pixel points; then performs inverse perspective transformation on the target image based on the lane line candidate points and the three-dimensional point cloud data, obtains a bird's eye view image corresponding to the target image, and determines the lane line position according to the bird's eye view image. As can be seen, the system can fully utilize the texture and color information in the target image, the distance and spatial coordinates in the three-dimensional point cloud data in the lane line detection process, so as to complement the target image and the three-dimensional point cloud data, greatly improving the lane line detection accuracy. At the same time, by extracting the color features and gradient features of the lane line segments, and then identifying the potential lane lines in the target image according to the color features and gradient features, the double feature fusion mechanism not only has stronger robustness to light changes, local occlusions and ground noise in the lane line detection process, but also can adapt to lane line detection in complex environments, thereby effectively reducing detection noise. Moreover, the system fuses the image and the three-dimensional point cloud data to construct a three-dimensional lane line topology structure, and generates lane line candidate points in three-dimensional space, then generates a bird's eye view image by inverse perspective transformation on the target image to determine the lane line position, further improving the lane line recognition accuracy. In addition, by combining the lane line prior information provided by the roadside unit in the lane line detection process, the lane line detection result can be continuously and stably output.

[0038] In another example embodiment of the present application, the embodiment also provides a vehicle including the lane line detection system as described in the above embodiments, or including the computer device as described in the above embodiments. Since the specific manner in which the lane line detection system and the computer device perform operations has been described in detail in the embodiments, the technical functions and effects possessed by the vehicle provided by the present embodiment can be referred to the above embodiments, and will not be described here.

[0039] In some examples, as Figure 10As shown, the vehicle includes a data input layer, an image processing module, and an intelligent driving module. The data input layer includes a vehicle-mounted camera, a sunlight and rainfall sensor, a millimeter wave radar, a roadside unit, and a communication module. The image processing module includes a data acquisition and analysis unit, a historical data storage unit, a multi-source data ETL (Extraction Transformation Loading) unit, an algorithm input / output data storage unit, an algorithm real-time calculation unit, and a lane line output unit. The intelligent driving module includes a path planning unit and an automatic driving unit. The vehicle-mounted camera can be used to capture real-time video streams or images during vehicle driving. The sunlight and rainfall sensor is used to obtain weather data. The millimeter wave radar can form three-dimensional point cloud data. The roadside unit is used to provide lane line prior information (such as curvature and position). The communication module is used to transmit data of the vehicle-mounted camera, the sunlight and rainfall sensor, the millimeter wave radar, and the roadside unit to the image processing module. The data is stored by the historical data storage unit in the image processing module and analyzed by the data acquisition and analysis unit. Then, the analysis results are transmitted to the algorithm input / output data storage unit and the algorithm real-time calculation unit through the multi-source data ETL unit. The algorithm real-time calculation unit executes the steps of the lane line detection method described in some embodiments above to output lane lines. Meanwhile, the output results are transmitted to the algorithm input / output data storage unit for storage by the algorithm real-time calculation unit. The automatic driving unit in the intelligent driving module generates an automatic driving request according to the lane lines output by the lane line output unit, and the path planning unit performs path planning according to the output lane lines.

[0040] In some examples, the image processing module can be composed of a lane perception fusion controller (LPFC). The core architecture of the lane perception fusion controller can be as follows Figure 11As shown, the vehicle lane line detection device includes a multi-source data access interface module, a core processing module, and an output and communication module. Specifically, the multi-source data access interface module includes a video interface, a radar interface, a V2X communication module, and a CAN communication module. The video interface supports multiple GMSL2 / GMSL3 interfaces and is used to access a vehicle-mounted camera with a maximum of 8 million pixels. The radar interface supports multiple CAN-FD and vehicle Ethernet (100BASE-T1) interfaces and is used to access a millimeter wave radar to receive point cloud and target list data. The V2X communication module integrates a C-V2X module (supporting PC5 direct communication) and is used to interact with a roadside unit (RSU) for low-latency data exchange to receive priori map and lane line information. The CAN communication module is used to receive analog / digital signals provided by a light sensor and a rain sensor. The core processing module includes a main processor (MCU) that adopts a high-performance multi-core ARM Cortex-A series processor, is responsible for running an operating system, task scheduling, traditional visual algorithms, and fusion logic, and integrates a lane line recognition algorithm. The output and communication module includes a vehicle communication network interface and a debugging and diagnosis interface. The vehicle communication network interface sends structured lane line information to an automatic driving decision planning MDC, a body control module (BCM), and a human-machine interaction (HMI) unit through CAN-FD and vehicle Ethernet (1000BASE-T1). The debugging and diagnosis interface provides an Ethernet interface for system debugging, log recording, and OTA software updating.

[0041] In an example embodiment of the present application, a computer device is also provided, which can include a memory, a processor, and a computer program stored on the memory. The processor can execute the computer program to cause the computer device to perform the steps of the lane line detection method as described above. Figure 1 The steps of the lane line detection method as shown. Figure 12 A structural schematic diagram of a computer device 1000 is shown. Referring to Figure 12 As shown, the computer device 1000 includes a processor 1010, a memory 1020, a power supply 1030, a display unit 1040, and an input unit 1060.

[0042] The processor 1010 is the control center of the computer device 1000, connects various components through various interfaces and lines, and performs various functions of the computer device 1000 by running or executing computer programs / instructions stored in the memory 1020, thereby overall monitoring the computer device 1000. In some embodiments, when the processor 1010 calls the computer program stored in the memory 1020, it can perform the steps of the lane line detection method as described above. Figure 1The steps of the lane line detection method are shown. Optionally, the processor 1010 can include one or more processing units; preferably, the processor 1010 can integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, the user interface and the application, etc., and the modem processor mainly processes the wireless communication. In some embodiments, the processor 1010 and the memory 1020 can be implemented on a single chip, and in other embodiments, they can also be implemented on separate chips respectively.

[0043] The memory 1020 mainly includes a program storage area and a data storage area, wherein the program storage area can store the operating system, various applications, etc.; the data storage area can store instruction data created according to the use of the computer device 1000, etc. In addition, the memory 1020 can include a high-speed random access memory and a non-volatile memory, for example, including at least one magnetic disk storage device, a flash memory device or other non-volatile solid-state memory device, etc.

[0044] The computer device 1000 further includes a power supply 1030 (such as a battery) for supplying power to various components. The power supply can be logically connected to the processor 1010 through a power management system, so as to realize the function management of charging, discharging and power consumption through the power management system.

[0045] The display unit 1040 can be used to display information input by the user or information provided to the user, and can also be used to display various menus of the computer device 1000, etc. In the embodiments of the present application, it is mainly used to display the display interface of each application in the computer device 1000 and the objects such as text, pictures, etc. displayed in the display interface. The display unit 1040 can include a display panel 1050. The display panel 1050 can be configured in the form of a liquid crystal display (LCD), an organic light-emitting diode (OLED), etc.

[0046] The input unit 1060 can be used to receive information such as numbers or characters input by the user. The input unit 1060 can include a touch panel 1070 and other input devices 1080. The touch panel 1070 can also be referred to as a touch screen, and the touch panel 1070 can collect touch operations of the user on or near it (such as the user's operation on or near the touch panel 1070 using a finger, a stylus or any suitable object or accessory).

[0047] Specifically, the touch panel 1070 can detect a user's touch operation, detect signals caused by the touch operation, and convert the signals into touch coordinates and send the touch coordinates to the processor 1010. In addition, the touch panel 1070 can also be implemented in various types such as resistive, capacitive, infrared, and surface acoustic wave, etc. The other input devices 1080 include, but are not limited to, one or more of a physical keyboard, function keys (such as volume control keys, power on / off keys, etc.), trackballs, mice, and joysticks.

[0048] Of course, the touch panel 1070 can also cover the display panel 1050. When the touch panel 1070 detects a touch operation on or near it, it can transmit to the processor 1010 to determine the type of touch event, and then the processor 1010 provides corresponding visual output on the display panel 1050 according to the type of touch event. Although in the above description, the touch panel 1070 and the display panel 1050 are implemented as two independent components to realize the input and output functions of the computer device 1000, in some embodiments, the touch panel 1070 and the display panel 1050 can be integrated to realize the input and output functions of the computer device 1000. Figure 12

[0049] The computer device 1000 can also include one or more sensors, such as a pressure sensor, a gravity acceleration sensor, a proximity light sensor, etc. Of course, according to the needs of specific application scenarios, the above computer device 1000 can also include a camera and other components.

[0050] In an exemplary embodiment of the present application, a computer readable storage medium is also provided, the storage medium stores computer programs / instructions, and the computer programs / instructions are executed by a processor to enable the above computer device to perform the steps of the lane line detection method as shown in the present application. Figure 1

[0051] Those skilled in the art can understand that, Figure 12 The above is only an example of a computer device and does not constitute a limitation on the device, which can include more or fewer components than shown, or combine certain components, or different components. For the convenience of description, each part is described as a module (or unit) according to its function. Of course, when implementing the present application, the functions of each module (or unit) can be implemented in the same or multiple software or hardware. For example, as some examples, the aforementioned computer device can be a vehicle, a car machine, etc.

[0052] ​​Those skilled in the art will understand that the present invention can take the form of a computer program product embodied on one or more computer-readable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code. The present invention is described according to flowchart illustrations and / or block diagrams of lane detection methods, lane detection systems, and computer program products based on some embodiments. It should be understood that each block of the flowchart illustrations and / or block diagrams, as well as combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be applied to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing device, generate instructions for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The functions specified in one or more boxes. These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable apparatus for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0053] It is understood that the above embodiments, in collecting, storing, using, processing, transmitting, providing, disclosing, deleting, etc., related data (such as target images), are completed with or with the user's consent. For example, the target image is obtained with the user's knowledge and consent; or it is provided voluntarily by the user after reading the relevant instructions; or it is actively authorized / provided / uploaded by the user when using some or all of the functions described in the above embodiments; or it is obtained through other means or channels with the user's consent.

[0054] It can be understood that, although the terms first, second, third, etc. can be used in this application to describe the preset ranges, etc., these terms are only used to distinguish the preset ranges from each other. For example, the first preset range can also be referred to as the second preset range, and similarly, the second preset range can also be referred to as the first preset range, without departing from the scope of the embodiments of the present application.

[0055] The above embodiments are only illustrative of the principles and effects of the present application, and are not intended to limit the present application. Any person skilled in the art can modify or change the above embodiments without departing from the spirit and scope of the present application. Therefore, all equivalent modifications or changes made by those skilled in the art without departing from the spirit and technical ideas disclosed in the present application should be covered by the claims of the present application.

Claims

1. A lane line detection method, characterized in that, The method includes the following steps: Acquire a target image and corresponding 3D point cloud data, wherein the target image includes lane line segments with different slopes; The lane line segments are grouped according to the sign of their slope, and the lane line segments in the target image are color-coded according to the grouping results so that the lane line segments in different groups have different colors. After color marking, extract color features and gradient features from the target image, and use the pixels that have both color features and gradient features as target pixels, and form lane line candidate points based on the target pixels. Based on the lane line candidate points and the three-dimensional point cloud data, the target image is subjected to inverse perspective transformation to obtain a bird's-eye view image corresponding to the target image, and the lane line position is determined based on the bird's-eye view image.

2. The lane line detection method according to claim 1, characterized in that, The process of determining the lane line position based on the bird's-eye view includes: The bird's-eye view image is subjected to back-projection transformation, and the bird's-eye view image after back-projection transformation is segmented into multiple block images of the same area; wherein, the segmentation includes horizontal segmentation and / or vertical segmentation; A corresponding pixel histogram is generated based on the pixel distribution of the block image, and the position of the highest point of the pixel histogram is used as the position of the lane line.

3. The lane line detection method according to claim 2, characterized in that, Before generating the corresponding pixel histogram based on the pixel distribution of the block image, the method further includes: calculating the proportion of effective pixels in each block image, and when the proportion is greater than a preset proportion threshold, recording the corresponding block image as an abnormal block image, and replacing the abnormal block image with an adjacent block image; wherein, the effective pixels include pixels corresponding to lane line segments.

4. The lane line detection method according to claim 2 or 3, characterized in that, The method further includes: Clustering and fitting are performed based on the block images to assign continuous or discontinuous lane lines to a category; Based on the clustering and fitting results of the block images, the target image of the current frame is used as prior knowledge for the target image of the next frame, and lane line tracking is performed based on the prior knowledge and the received lane line prior information; wherein, the lane line prior information is provided by the roadside unit.

5. The lane line detection method according to any one of claims 1 to 3, characterized in that, After acquiring the target image, the method further includes: Treat the lane line segments as straight lines, and according to the grouping results, make all the straight lines in one group intersect with all the straight lines in another group to determine all the intersection points; Calculate the distance between each intersection point and each straight line, and select the intersection point with the shortest distance as the vanishing point, and use the ordinate of the vanishing point in the image coordinate system as the vanishing line; wherein, the image coordinate system is determined based on the target image; The target image is denoised based on the vanishing point and the vanishing line, and all or part of the line segments in the target image that are not associated with the lane lines are deleted.

6. The lane line detection method according to any one of claims 1 to 3, characterized in that, The process of color-coding lane line segments in the target image based on the grouping results includes: Weather data is acquired through a sunlight and rainfall sensor, and the color threshold and contrast enhancement parameters are dynamically adjusted based on the weather data. Based on the grouping results, color threshold, and contrast enhancement parameters, the lane line segments in the target image are color-marked.

7. The lane line detection method according to any one of claims 1 to 3, characterized in that, After determining the lane line position based on the bird's-eye view image, the method further includes: responding to an autonomous driving request or an assisted driving request, performing path planning based on the determined lane line position, and controlling the current vehicle to perform autonomous driving or assisted driving according to the path planning result.

8. A lane line detection system, characterized in that, The system includes: The data acquisition module is used to acquire a target image and corresponding 3D point cloud data, wherein the target image includes lane line segments with different slopes. The color marking module is used to group the lane line segments according to the positive or negative slope, and to mark the lane line segments in the target image with color according to the grouping results, so that the lane line segments in different groups have different colors. The feature extraction module is used to extract color features and gradient features from the target image after color marking, and to take the pixels that have both color features and gradient features as target pixels, and to form lane line candidate points based on the target pixels. The lane line detection module is used to perform a reverse perspective transformation on the target image based on the lane line candidate points and the three-dimensional point cloud data to obtain a bird's-eye view image corresponding to the target image, and to determine the lane line position based on the bird's-eye view image.

9. A computer device, characterized in that, The method includes a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the lane detection method according to any one of claims 1 to 7.

10. A vehicle, characterized in that, The vehicle includes the lane detection system as described in claim 8, or the computer equipment as described in claim 9.

Images