Lane line detection method, device and equipment

Abstract

The present application provides a lane line detection method, device and equipment, the method comprising: determining a target point cloud, a target line type and a target sorting value of a target lane line based on a current frame image; determining a distance matching degree based on a target point cloud and a historical point cloud of a historical lane line; determining a sequence matching degree based on a target sorting value and a historical sorting value of a historical lane line; determining a line type matching degree based on a target line type and a historical line type of a historical lane line; determining a target matching degree between a target lane line and a historical lane line based on a distance matching degree, a sequence matching degree and a line type matching degree; determining an associated historical lane line that matches a target lane line from multiple historical lane lines based on a target matching degree between a target lane line and each historical lane line, and fusing the target lane line with the associated historical lane line. Through the present application scheme, the stability and accuracy of lane line detection can be improved.

Description

Technical Field

[0001] The present application relates to the field of perception and mapping technology, and in particular to a lane line detection method, device and equipment. Background Art

[0002] A vehicle is usually equipped with multiple cameras (such as a front-view camera, surround-view camera, rear-view camera, etc.), which can be used to capture images. For example, the front-view camera can be used to capture images, and lane lines can be detected based on the images captured by the front-view camera. For example, the image can be input into a neural network model, and the physical position of the lane line in front of the vehicle can be detected in real time based on the neural network model to achieve lane line detection.

[0003] However, since the lane line detection performance is affected by factors such as lighting, environment, ground slope, obstacle occlusion or decreased lane line contrast, errors may occur in the lane line detection results, resulting in the inability to accurately detect the lane line, that is, the lane line detection results are inaccurate and the front and rear detection results of the lane line are inconsistent. Summary of the invention

[0004] The present application provides a lane line detection method, the method comprising:

[0005] Determine the target point cloud, target line type and target sort value of the target lane line based on the current frame image;

[0006] Based on the target point cloud and the historical point cloud of the historical lane line, determine the distance matching degree between the target lane line and the historical lane line; based on the target sorting value and the historical sorting value of the historical lane line, determine the order matching degree between the target lane line and the historical lane line; based on the target line type and the historical line type of the historical lane line, determine the line type matching degree between the target lane line and the historical lane line; based on the distance matching degree, the order matching degree and the line type matching degree, determine the target matching degree between the target lane line and the historical lane line; wherein the historical point cloud, the historical line type and the historical sorting value are determined based on the historical frame image before the current frame image;

[0007] Based on the target matching degree between the target lane line and each historical lane line, an associated historical lane line that matches the target lane line is determined from multiple historical lane lines, and the target lane line is merged with the associated historical lane line; or, based on the target matching degree between the target lane line and each historical lane line, the target lane line is determined to be a new lane line.

[0008] The present application provides a lane line detection device, the device comprising:

[0009] A detection module, used to determine a target point cloud of a target lane line, a target line type of the target lane line, and a target ranking value of the target lane line based on a current frame image;

[0010] A processing module, for determining a distance matching degree between the target lane line and the historical lane line based on the target point cloud and the historical point cloud of the historical lane line; determining a sequence matching degree between the target lane line and the historical lane line based on the target sorting value and the historical sorting value of the historical lane line; determining a line type matching degree between the target lane line and the historical lane line based on the target line type and the historical line type of the historical lane line; determining a target matching degree between the target lane line and the historical lane line based on the distance matching degree, the sequence matching degree and the line type matching degree; wherein the historical point cloud, the historical line type and the historical sorting value are determined based on the historical frame image preceding the current frame image;

[0011] A matching and fusion module is used to determine, from multiple historical lane lines, an associated historical lane line that matches the target lane line based on the target matching degree between the target lane line and each historical lane line, and fuse the target lane line with the associated historical lane line; or, based on the target matching degree between the target lane line and each historical lane line, determine that the target lane line is a new lane line.

[0012] The present application provides an electronic device, comprising: a processor and a machine-readable storage medium, wherein the machine-readable storage medium stores machine-executable instructions that can be executed by the processor; the processor is used to execute the machine-executable instructions to implement the above-mentioned lane line detection method.

[0013] The present application provides a computer program product, which includes a computer program. When the computer program is executed by a processor, the lane line detection method described above is implemented.

[0014] The present application provides a machine-readable storage medium, which stores machine-executable instructions that can be executed by a processor; wherein the processor is used to execute the machine-executable instructions, and when the machine-executable instructions are executed, the above-mentioned lane line detection method is implemented.

[0015] It can be seen from the above technical solutions that in the embodiment of the present application, lane line detection is realized by combining the point cloud, line type and ranking value of the lane line, that is, lane line detection is realized by parameters of multiple dimensions of the lane line, which can improve the stability and accuracy of lane line detection, ensure the continuous and complete output of lane lines around the vehicle, and even if the lane line detection performance is affected by factors such as illumination, environment, ground slope, obstacle occlusion or decreased contrast of the lane line, it can also avoid errors in the lane line detection results, accurately detect the lane line, and the front and back detection results of the lane line are consistent. For example, the point cloud of the lane line represents multiple three-dimensional points of the lane line, which can reflect the distance (depth) between the lane line and the vehicle, the line type of the lane line indicates whether the lane line is a solid line or a dotted line, and the ranking value of the lane line indicates which lane line is around the vehicle. When the point cloud, line type and ranking value of the lane line are used to realize lane line detection, the line type and ranking value can be used to review the detection results of the point cloud, thereby improving the accuracy of lane line detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 is a flow chart of a lane line detection method in one embodiment of the present application;

[0017] Figure 2 is a schematic diagram of the FOV of a forward-looking camera in one embodiment of the present application;

[0018] Figure 3 is a flow chart of a lane line detection method in one embodiment of the present application;

[0019] Figure 4 is a schematic diagram of a target ranking value of a target lane line in one embodiment of the present application;

[0020] Figure 5 is a schematic diagram of the sequential matching degree between the target lane line and the historical lane line in this application;

[0021] Figure 6 is a schematic diagram of a lane grid update process in one embodiment of the present application;

[0022] Figure 7 It is a schematic diagram of splicing near and far key points in one embodiment of the present application;

[0023] Figure 8 It is a schematic diagram of the process of online lane mapping in one embodiment of the present application;

[0024] Fig.9A is a schematic structural diagram of a lane line detection device in one embodiment of the present application;

[0025] Fig. 9B It is a hardware structure diagram of an electronic device in one embodiment of the present application. DETAILED DESCRIPTION

[0026] In the embodiment of the present application, a lane line detection method is proposed, which can be applied to electronic devices, see Figure 1 FIG. 1 is a flow chart of the lane line detection method, which may include:

[0027] Step 101: Determine a target point cloud, a target line type, and a target sort value of a target lane line based on a current frame image, wherein the target point cloud may include a plurality of target three-dimensional points, and the target line type may be a solid line or a dotted line.

[0028] Step 102: determine the distance matching degree between the target lane line and the historical lane line based on the target point cloud and the historical point cloud of the historical lane line; determine the order matching degree between the target lane line and the historical lane line based on the target sorting value and the historical sorting value of the historical lane line; determine the line type matching degree between the target lane line and the historical lane line based on the target line type and the historical line type of the historical lane line; determine the target matching degree between the target lane line and the historical lane line based on the distance matching degree, the order matching degree and the line type matching degree; wherein the historical point cloud, the historical line type and the historical sorting value are determined based on the historical frame image before the current frame image.

[0029] Step 103: Based on the target matching degree between the target lane line and each historical lane line, determine the associated historical lane line that matches the target lane line from multiple historical lane lines, and merge the target lane line with the associated historical lane line; or, based on the target matching degree between the target lane line and each historical lane line, determine that the target lane line is a new lane line, and the subsequent process will merge the lane line with the target lane line.

[0030] Exemplarily, the target point cloud includes multiple target three-dimensional points. Based on the target point cloud and the historical point cloud of the historical lane line, the distance matching degree between the target lane line and the historical lane line is determined, which may include but is not limited to: determining the curve equation of the historical lane line based on the historical point cloud of the historical lane line; sampling multiple reference three-dimensional points based on the curve equation of the historical lane line; for each target three-dimensional point, calculating the distance between the target three-dimensional point and each reference three-dimensional point, and selecting the minimum distance based on the distance between the target three-dimensional point and each reference three-dimensional point; determining the distance matching degree based on the minimum distance corresponding to each target three-dimensional point.

[0031] Exemplarily, based on the target line type and the historical line type of the historical lane line, the line type matching degree between the target lane line and the historical lane line is determined, which may include but is not limited to: if the target line type is the same as the historical line type, then the line type matching degree can be determined to be a first value; if the target line type is different from the historical line type, then the line type matching degree can be determined to be a second value, and the second value can be greater than the first value.

[0032] Exemplarily, based on the target ranking value and the historical ranking value of the historical lane line, determining the order matching degree between the target lane line and the historical lane line may include but is not limited to: determining the order matching degree based on the absolute value of the difference between the target ranking value and the historical ranking value; wherein, the ranking value of each lane line may be decreased from left to right to obtain the target ranking value of the target lane line; or, the ranking value of each lane line may be decreased from right to left to obtain the target ranking value of the target lane line.

[0033] Exemplarily, fusing the target lane line with the associated historical lane line may include: updating the first grid queue of the associated historical lane line based on the target point cloud to obtain an updated second grid queue; wherein the first grid queue includes multiple first grids, and the key points corresponding to each first grid are determined based on the historical point cloud, and the second grid queue includes multiple second grids, and the key points corresponding to each second grid are determined based on the historical point cloud and the target point cloud; obtaining a high-confidence point set based on the second grid queue; the high-confidence point set includes the key points corresponding to each second grid in the second grid queue; determining curve equation parameters based on the high-confidence point set; updating the curve equation of the associated historical lane line based on the curve equation parameters, and the curve equation of the associated historical lane line is used to determine the distance matching degree corresponding to the next frame image of the current frame image.

[0034] Exemplarily, updating a first grid queue of associated historical lane lines based on a target point cloud to obtain an updated second grid queue may include: generating a reference curve based on the target point cloud, and rasterizing the reference curve according to a preset step size to obtain multiple second grids; for each target three-dimensional point in the target point cloud, determining a projection point of the target three-dimensional point on the reference curve, and using the second grid to which the projection point belongs as the second grid of the target three-dimensional point; for each second grid of the reference curve, determining a first grid corresponding to the second grid in the first grid queue; determining a key point based on all target three-dimensional points corresponding to the second grid and all historical three-dimensional points corresponding to the first grid, and updating the key point to the key point of the second grid; and constructing a second grid queue, the second grid queue including the key point of each second grid.

[0035] In a possible implementation, determining the first grid corresponding to the second grid in the first grid queue may include but is not limited to: for each first grid in the first grid queue, determining the first key point corresponding to the first grid based on all historical three-dimensional points corresponding to the first grid; determining the second key point corresponding to the second grid based on all target three-dimensional points corresponding to the second grid; and determining the first grid corresponding to the second grid from the first grid queue based on the distance between the second key point and the first key point corresponding to each first grid in the first grid queue.

[0036] Exemplarily, determining the curve equation parameters based on the high confidence point set may include, but is not limited to: determining the obstacle point cloud around the target lane line based on the current frame image, determining the road trend point based on the obstacle point cloud, and obtaining the low confidence point set based on the road trend point. On this basis, the curve equation parameters may be determined based on the high confidence point set and the low confidence point set.

[0037] Exemplarily, determining the curve equation parameters based on the high confidence point set and the low confidence point set may include, but is not limited to: determining the curve equation parameters using the following formula:

[0038]

[0039] x i ,y i , z i represents the i-th 3D point in the high confidence point set, x j ,y j , z j represents the jth 3D point in the low confidence point set, Lx, Ly, Lz represent the loss value, W A Represents the weight of the high confidence point set, W B Represents the weight of the low confidence point set; a k , b k and c k represents the curve equation parameter, s i represents the mileage of the i-th 3D point in the high confidence point set, s j Represents the mileage of the j-th 3D point in the low confidence point set.

[0040] It can be seen from the above technical solutions that in the embodiment of the present application, lane line detection is realized by combining the point cloud, line type and sorting value of the lane line, that is, lane line detection is realized by parameters of multiple dimensions of the lane line, which can improve the stability and accuracy of lane line detection and ensure the continuous and complete output of lane lines around the vehicle. Even if the lane line detection performance is affected by factors such as illumination, environment, ground slope, obstacle occlusion or decreased contrast of the lane line, it can also avoid errors in the lane line detection results, and the lane line can be accurately detected, and the front and back detection results of the lane line are consistent. A lane line detection fusion method based on sliding grid is proposed, which can fuse the target lane line with the associated historical lane line based on the sliding grid, thereby realizing the smooth fusion of the target lane line and the associated historical lane line. With the grid as the granularity, the grid of the target lane line is matched with the grid of the associated historical lane line, and then the grids at the same position are found, and the grids at the same position are fused to improve the fusion quality. A near-far fusion scheme is proposed, which divides the original fitting points into high and low confidence points. The high-confidence point set includes the key points corresponding to the grid of the lane line, and the low-confidence point set includes the road trend points corresponding to the obstacle point cloud around the lane line. In this way, the far-end trend (i.e., the road trend points corresponding to the obstacle point cloud) is introduced as a constraint on the fitting, which improves the stability of the final curve equation.

[0041] The lane line detection method of the embodiment of the present application is described below in conjunction with specific embodiments.

[0042] In order to realize the function of autonomous driving, the vehicle needs to support lane line detection. For example, for autonomous driving functions such as ICC (Intelligent Cruise Control), NOA (Navigate on Autopilot), RTM (Real-time Mapping), ADAS (Advanced Driving Assistance System), etc., lane line detection is required.

[0043] In order to support lane line detection, the cameras deployed on the vehicle (such as front-view cameras, surround-view cameras, rear-view cameras, etc.) can collect images around the vehicle (such as front-view images collected by the front-view camera, surround-view images collected by the surround-view camera, rear-view images collected by the rear-view camera, etc., taking the front-view image collected by the front-view camera as an example). On this basis, the front-view image collected by the front-view camera can be input into the neural network model, and the physical position of the lane line in front of the vehicle can be detected in real time based on the neural network model to achieve lane line detection.

[0044] However, since the lane line detection performance is affected by factors such as lighting, environment, ground slope, obstacle occlusion or decreased lane line contrast, errors may occur in the lane line detection results, resulting in the inability to accurately detect the lane line, that is, the lane line detection results are inaccurate and the front and rear detection results of the lane line are inconsistent.

[0045] In addition, see Figure 2 As shown in FIG. 1 , a schematic diagram of the FOV (Field Of View) of the front-view camera is shown. The FOV range of real-time detection is narrow and does not meet the real-time detection requirements of the surrounding lane lines.

[0046] In view of the above findings, a lane line detection method is proposed in the embodiment of the present application, which can realize online mapping detection of rasterized lane lines, improve the stability of lane line detection, improve the accuracy of lane line detection, and ensure continuous and complete output of lane lines around the vehicle. Figure 3 FIG. 1 is a flow chart of a lane line detection method. The method can be applied to an electronic device, and the electronic device can be a vehicle-mounted device, such as a vehicle-mounted device supporting an automatic driving function. The lane line detection method can include the following steps:

[0047] Step 301: While the vehicle is traveling, collect images of the surroundings of the vehicle through a camera.

[0048] For example, during the driving process of the vehicle, the camera can periodically collect images. Each time an image is collected, the currently collected image can be referred to as a current frame image, and lane line detection needs to be performed based on the current frame image. In addition, the image before the current frame image (such as one frame image or multiple frames of images) can be referred to as a historical frame image, and the image after the current frame image can be referred to as a next frame image.

[0049] Exemplarily, the image around the vehicle can be a front view image captured by a front-view camera, a surround view image captured by a surround view camera, a rear view image captured by a rear-view camera, or a fused image of multiple images (such as a fused image of a front view image and multiple surround view images), and there is no limitation on this.

[0050] Step 302: Determine a target point cloud, a target line type, and a target sort value of a target lane line (a target lane line is any lane line obtained based on the current frame image) based on the current frame image.

[0051] Exemplarily, after obtaining the current frame image, the current frame image can be input into the neural network model, and the neural network model determines the category of each pixel point of the current frame image, and the category of the pixel point indicates that the pixel point belongs to the lane line or the pixel point does not belong to the lane line. Regarding the network structure and training process of the neural network model, this embodiment does not impose any restrictions, and it is sufficient to be able to determine the category of each pixel point.

[0052] After knowing the category of each pixel point of the current frame image, all the pixel points belonging to the lane line can be determined, and the line formed by these pixel points is the target lane line. Obviously, when there are multiple lane lines around the vehicle, multiple target lane lines can be known based on all the pixel points belonging to the lane line.

[0053] Exemplarily, after obtaining the target lane line, for each pixel point of the target lane line (i.e., the pixel point of the current frame image), depth prediction can be performed on the pixel point to obtain the three-dimensional point (i.e., three-dimensional coordinates) of the pixel point in the physical space. There is no restriction on this depth prediction process, and it is sufficient to obtain the three-dimensional point corresponding to the pixel point. The three-dimensional point can include the physical position (x, y) and depth d.

[0054] After depth prediction of each pixel point of the target lane line, a three-dimensional point of each pixel point in the physical space can be obtained, and all three-dimensional points constitute the target point cloud of the target lane line, that is, the target point cloud can include multiple three-dimensional points of the target lane line, and these three-dimensional points are called target three-dimensional points.

[0055] For example, after obtaining the target lane line, the target line type of the target lane line can also be determined, and the target line type can be a solid line or a dashed line. For example, the target line type of the target lane line can be determined by a neural network model, or by analyzing the current frame image to determine the target line type of the target lane line. There is no limitation to this, as long as the target line type of the target lane line can be determined.

[0056] Exemplarily, after obtaining the target lane line, a target ranking value of the target lane line may also be determined, and the target ranking value is used to indicate which lane line around the vehicle the target lane line is.

[0057] For example, the target ranking value of the target lane line can be obtained by decreasing the ranking value of each lane line from left to right. Based on this, in the direction from left to right, assuming that there are 4 target lane lines in total, the target ranking value of the first target lane line is 4, the target ranking value of the second target lane line is 3, the target ranking value of the third target lane line is 2, and the target ranking value of the fourth target lane line is 1. Alternatively, the target ranking value of the first target lane line is 2, the target ranking value of the second target lane line is 1, the target ranking value of the third target lane line is -1, and the target ranking value of the fourth target lane line is -2. Alternatively, the target ranking value of the first target lane line is 6, the target ranking value of the second target lane line is 4, the target ranking value of the third target lane line is 2, and the target ranking value of the fourth target lane line is 0. Of course, the above are just a few examples, and there is no limitation to this, as long as the ranking value decreases.

[0058] See also Figure 4 As shown, it is a schematic diagram of the target ranking values ​​of the target lane lines. From left to right, the target ranking values ​​of the four target lane lines are 2, 1, -1, and -2 respectively.

[0059] For example, the target ranking value of the target lane line can be obtained by decreasing the ranking value of each lane line from right to left. Based on this, in the direction from right to left, assuming that there are 4 target lane lines in total, the target ranking value of the first target lane line is 2, the target ranking value of the second target lane line is 1, the target ranking value of the third target lane line is -1, and the target ranking value of the fourth target lane line is -2. Of course, the above is just an example and there is no limitation to this, as long as the ranking value decreases.

[0060] In summary, the target point cloud, target line type and target sorting value of the target lane line can be determined. Taking four target lane lines as an example, the target point cloud, target line type and target sorting value of the target lane line a1, the target point cloud, target line type and target sorting value of the target lane line a2, the target point cloud, target line type and target sorting value of the target lane line a3, and the target point cloud, target line type and target sorting value of the target lane line a4 are obtained.

[0061] Step 303: Determine the distance matching degree (also referred to as similarity matching degree) between the target lane line and the historical lane line based on the target point cloud of the target lane line and the historical point cloud of the historical lane line.

[0062] For example, during vehicle driving, assuming that image 1, image 2, and image 3 are collected in sequence, when image 2 is used as the current frame image, the lane line obtained based on image 2 is called the target lane line, and the point cloud of the target lane line is called the target point cloud, while the lane line obtained based on image 1 is called the historical lane line, and the point cloud of the historical lane line is called the historical point cloud. When image 3 is used as the current frame image, the lane line obtained based on image 3 is called the target lane line, and the point cloud of the target lane line is called the target point cloud, while the lane line obtained based on image 1 and image 2 is called the historical lane line, and the point cloud of the historical lane line is called the historical point cloud.

[0063] Exemplarily, the distance matching degree is used to reflect the distance between the target point cloud and the historical point cloud. If the distance between the target point cloud and the historical point cloud is larger, the distance matching degree is larger, indicating that the target point cloud and the historical point cloud are more dissimilar, that is, the probability that the target lane line and the historical lane line are the same lane line is smaller. If the distance between the target point cloud and the historical point cloud is smaller, the distance matching degree is smaller, indicating that the target point cloud and the historical point cloud are more similar, that is, the probability that the target lane line and the historical lane line are the same lane line is larger.

[0064] In a possible implementation, the following steps may be used to determine the distance matching degree between the target lane line and the historical lane line. This is just an example and there is no limitation on this determination method.

[0065] Step S11: determining a curve equation of the historical lane line based on the historical point cloud of the historical lane line.

[0066] For example, when image 2 is used as the current frame image, the curve equation of the target lane line can be determined based on the target point cloud of the target lane line. The determination method is shown in the subsequent steps. On this basis, when image 3 is used as the current frame image, the curve equation of the historical lane line has been determined based on the historical point cloud of the historical lane line.

[0067] Step S12: sampling a plurality of reference three-dimensional points based on the curve equation of the historical lane line.

[0068] For example, the curve equation of the historical lane line can be shown in formula (1):

[0069]

[0070] In formula (1), a i , b i and c i represents the curve equation parameters, the curve equation parameters of the curve equation of the historical lane line are known values, and s represents the mileage value. For example, when i is 0, configure one or more mileage values ​​s, and the mileage value s can be configured arbitrarily. For each mileage value s, substitute the mileage value s into formula (1), and you can get the reference three-dimensional point (x, y, z) corresponding to the mileage value s. When i is 1, configure one or more mileage values ​​s, for each mileage value s, substitute the mileage value s into formula (1), and you can get the reference three-dimensional point (x, y, z) corresponding to the mileage value s. Similarly, when i is n, configure one or more mileage values ​​s, for each mileage value s, substitute the mileage value s into formula (1), and you can get the reference three-dimensional point (x, y, z) corresponding to the mileage value s. On this basis, multiple reference three-dimensional points can be obtained based on the curve equation.

[0071] Step S13: The target point cloud of the target lane line may include multiple target three-dimensional points. For each target three-dimensional point in the target point cloud, the distance between the target three-dimensional point and each reference three-dimensional point is calculated, and the minimum distance is selected based on the distance between the target three-dimensional point and each reference three-dimensional point.

[0072] For example, the distance between the target 3D point 1 and each reference 3D point (i.e., the distance between two 3D points) is calculated. Based on the distance between the target 3D point 1 and each reference 3D point, the minimum distance can be selected, and the minimum distance is used as the distance between the target 3D point 1 and the curve equation of the historical lane line.

[0073] Step S14: Determine the distance matching degree between the target lane line and the historical lane line based on the minimum distance corresponding to each target 3D point. For example, the average value of the minimum distances corresponding to all target 3D points can be used as the distance matching degree, or a preset algorithm can be used to calculate the minimum distances corresponding to all target 3D points, and the calculation result can be used as the distance matching degree. There is no restriction on this preset algorithm.

[0074] Step 304: Determine the line type matching degree between the target lane line and the historical lane line based on the target line type of the target lane line and the historical line type of the historical lane line. For example, the line type matching degree can be used to reflect whether the target line type of the target lane line is the same as the historical line type of the historical lane line.

[0075] In a possible implementation, if the target line type of the target lane line is the same as the historical line type of the historical lane line, the line type matching degree is determined to be a first value. If the target line type of the target lane line is different from the historical line type of the historical lane line, the line type matching degree is determined to be a second value. The second value may be greater than the first value, for example, the second value may be 1, and the first value may be 0.

[0076] For example, the line type matching degree is defined as follows: if the target line type of the target lane line is the same as the historical line type of the historical lane line, the line type matching degree is 0; if the target line type of the target lane line is different from the historical line type of the historical lane line, the line type matching degree is 1. The line type matching degree can be expressed by the following formula (2):

[0077]

[0078] In formula (2), S3(i,j) represents the line type matching degree between the target lane line i and the historical lane line j, type(i) represents the target line type of the target lane line i, and type(j) represents the historical line type of the historical lane line j.

[0079] Step 305: Determine the order matching degree between the target lane line and the historical lane line based on the target order value of the target lane line and the historical order value of the historical lane line. The order matching degree is used to indicate the order deviation degree centered on the vehicle. The greater the order matching degree, the greater the order deviation degree centered on the vehicle. The smaller the order matching degree, the smaller the order deviation degree centered on the vehicle.

[0080] For example, assuming that the target lane line is the second lane line on the left side of the vehicle, and the historical lane line is the second lane line on the left side of the vehicle, the order deviation between the target lane line and the historical lane line is not deviated. Assuming that the target lane line is the second lane line on the left side of the vehicle, and the historical lane line is the first lane line on the left side of the vehicle, the order deviation between the target lane line and the historical lane line is small. Assuming that the target lane line is the second lane line on the left side of the vehicle, and the historical lane line is the lane line on the right side of the vehicle, the order deviation between the target lane line and the historical lane line is large.

[0081] In a possible implementation, the order matching degree between the target lane line and the historical lane line can be determined based on the absolute value of the difference between the target order value of the target lane line and the historical order value of the historical lane line. For example, the order matching degree can be expressed by the following formula (3):

[0082] S2(i,j)=|rank(i)-rank(j)| Formula (3)

[0083] In formula (3), S2(i,j) represents the order matching degree between the target lane line i and the historical lane line j, rank(i) represents the target ranking value of the target lane line i, and rank(j) represents the historical ranking value of the historical lane line j.

[0084] For example, see Figure 5 As shown in , it is a schematic diagram of the sequential matching degree between the target lane line and the historical lane line. Figure 5 It can be seen that if the target ranking value of the target lane line is -1 and the historical ranking value of the historical lane line is -1, the order matching degree between the target lane line and the historical lane line is 0, that is, the probability that the target lane line and the historical lane line are the same lane line is high. Alternatively, if the target ranking value of the target lane line is -1 and the historical ranking value of the historical lane line is -2, the order matching degree between the target lane line and the historical lane line is 1, that is, the probability that the target lane line and the historical lane line are the same lane line is low.

[0085] Step 306: Determine the target matching degree between the target lane line and the historical lane line based on the distance matching degree, the order matching degree and the line type matching degree. For example, the distance matching degree, the order matching degree and the line type matching degree may be weighted to obtain the target matching degree between the target lane line and the historical lane line.

[0086] For example, the target matching degree can be determined by the following formula (4):

[0087] S(i,j)=W1S1(i,j)+W2S2(i,j)+W3S3(i,j) Formula (4)

[0088] In formula (4), S1(i, j) may represent the distance matching degree between the target lane line i and the historical lane line j, W1 may represent the weight of the distance matching degree, S2(i, j) may represent the order matching degree between the target lane line i and the historical lane line j, W2 may represent the weight of the order matching degree, S3(i, j) may represent the line type matching degree between the target lane line i and the historical lane line j, W3 may represent the weight of the line type matching degree, and S(i, j) may represent the target matching degree between the target lane line i and the historical lane line j.

[0089] For example, if lane line detection is performed only based on distance matching, multiple target lane lines may meet the matching conditions with a historical lane line at the same time, which may eventually lead to an erroneous matching result. Based on this, in this embodiment, the order matching degree and line type matching degree of the lane line are introduced, and the target matching degree is obtained by weighted operation of the distance matching degree, the order matching degree and the line type matching degree. The target matching degree is used as the final matching degree, and lane line detection is performed based on the target matching degree, thereby avoiding erroneous matching results.

[0090] Step 307: Based on the target matching degree between the target lane line and each historical lane line, determine the associated historical lane line that matches the target lane line from multiple historical lane lines, and merge the target lane line with the associated historical lane line; or, based on the target matching degree between the target lane line and each historical lane line, determine that the target lane line is a new lane line, and the subsequent process will merge the lane line with the target lane line.

[0091] For example, if the target matching degree is smaller, it means that the probability that the target lane line and the historical lane line are the same lane line is greater. On this basis, based on the target matching degree between the target lane line and each historical lane line, the minimum target matching degree can be selected. If the minimum target matching degree is less than the preset threshold, the historical lane line corresponding to the minimum target matching degree is used as the associated historical lane line for the target lane line to match. Alternatively, if the minimum target matching degree is not less than the preset threshold, the target lane line is used as the new lane line, that is, the target lane line does not have an associated historical lane line, but is a newly detected lane line based on the current frame image.

[0092] In a possible implementation, considering that there are multiple target lane lines, and a historical lane line can only be associated with one target lane line, and cannot be associated with multiple target lane lines at the same time, multiple target lane lines can be jointly solved to obtain the associated historical lane lines that match the multiple target lane lines. For example, formula (5) can be used to determine the associated historical lane line that matches each target lane line:

[0093] min m S=min c∈m ∑(i,j)∈c (W1S1(i,j)+W2S2(i,j)+W3S3(i,j)) Formula (5)

[0094] In formula (5), m represents all matching sets of the current lane line and the historical lane line, c represents a matching of the current lane line and the historical lane line, and (i, j) represents the matching pair of the current lane line i and the historical lane line j under the matching of c. On this basis, by solving formula (5), the minimum S value is found, and the combination under the minimum S value is the pairing relationship between the target lane line and the historical lane line.

[0095] Exemplarily, after determining the associated historical lane line that matches the target lane line, or determining that the target lane line is a new lane line, the lane line detection can be completed. After completing the lane line detection, for each target lane line, the curve equation of the target lane line can also be determined based on the target point cloud of the target lane line. In the lane line detection process based on the next frame image, the curve equation of the target lane line is used as the curve equation of the historical lane line, and the target point cloud of the target lane line is used as the historical point cloud of the historical lane line.

[0096] For example, the curve equation of the target lane line can be determined by the following steps, that is, the target lane line and the associated historical lane line can be merged by the following steps:

[0097] Step 308: Based on the target point cloud of the target lane line, the first grid queue of the associated historical lane lines matched with the target lane line is updated to obtain an updated second grid queue.

[0098] Exemplarily, the first grid queue may include multiple first grids, each first grid corresponds to a key point (i.e., a key three-dimensional point), and the key point corresponding to each first grid is determined based on the historical point cloud. For example, the historical point cloud may include multiple historical three-dimensional points, and for each first grid, the first grid may correspond to at least one historical three-dimensional point, and the key point corresponding to the first grid may be determined based on the historical three-dimensional point corresponding to the first grid. For example, a weighted operation is performed on the historical three-dimensional points corresponding to the first grid to obtain the key point, or the average value of the historical three-dimensional points corresponding to the first grid is used as the key point, or the maximum value of the historical three-dimensional points corresponding to the first grid is used as the key point, or the minimum value of the historical three-dimensional points corresponding to the first grid is used as the key point. Of course, the above are just a few examples of determining the key point corresponding to the first grid, and there is no limitation on the method of determining the key point.

[0099] Exemplarily, the second grid queue may include multiple second grids, each second grid corresponds to a key point (i.e., a key 3D point), and the key point corresponding to each second grid is determined based on the historical point cloud and the target point cloud. For example, the historical point cloud may include multiple historical 3D points, and the target point cloud may include multiple target 3D points. For each second grid, the second grid may correspond to a historical 3D point and / or a target 3D point, and the key point corresponding to the second grid may be determined based on the historical 3D point and / or the target 3D point corresponding to the second grid. For example, a weighted operation is performed on all 3D points (historical 3D points and / or target 3D points) corresponding to the second grid to obtain a key point, or the average value of all 3D points corresponding to the second grid is used as the key point, or the maximum value of all 3D points corresponding to the second grid is used as the key point, or the minimum value of all 3D points corresponding to the second grid is used as the key point.

[0100] In a possible implementation, the grid queue may be updated using the following steps:

[0101] Step S21: Generate a reference curve based on the target point cloud of the target lane line, and rasterize the reference curve according to a preset step size (such as a preset mileage step size) to obtain a plurality of second grids.

[0102] Exemplarily, the target point cloud of the target lane line may include multiple target three-dimensional points, and these target three-dimensional points may be fitted, such as by using a cubic polynomial equation to fit these target three-dimensional points to obtain a fitted reference curve, and there is no restriction on the fitting method of this reference curve.

[0103] For example, after obtaining the reference curve, the reference curve is used to define the mileage reference direction, and the starting point of the reference curve can be defined as 0. The reference curve can be rasterized according to the preset step size, that is, a grid is divided every preset step size. After the reference curve is rasterized, multiple grids can be obtained. For the convenience of distinction, these grids are referred to as second grids. Figure 6 , which is a schematic diagram of the lane grid update process. The reference curve can be rasterized according to a preset step size to obtain multiple second grids, which can also be called local grids.

[0104] Step S22: for each target 3D point in the target point cloud, determine the projection point of the target 3D point on the reference curve, and use the second grid to which the projection point belongs as the second grid of the target 3D point.

[0105] For example, for the projection point of the target 3D point on the reference curve, a perpendicular line can be drawn from the target 3D point to the reference curve, and the point where the foot of the perpendicular is located is the projection point of the target 3D point on the reference curve. Figure 6 As shown, each target 3D point in the target point cloud can be locally grid projected, that is, the projection point of each target 3D point on the reference curve is determined, and the second grid to which the projection point belongs is used as the second grid of the target 3D point, thereby obtaining the second grid corresponding to each target 3D point.

[0106] For example, the second grid to which the projection point of target 3D point 1 belongs is used as the second grid of target 3D point 1, and the second grid to which the projection point of target 3D point 2 belongs is used as the second grid of target 3D point 2, and so on, and the second grid of each target 3D point can be obtained. On this basis, for each second grid on the reference curve, the target 3D point corresponding to the second grid can be determined.

[0107] Step S23: for each second grid on the reference curve, determine the first grid corresponding to the second grid in the first grid queue; determine the key point based on all target three-dimensional points corresponding to the second grid and all historical three-dimensional points corresponding to the first grid, and update the key point to the key point of the second grid.

[0108] Alternatively, if the second grid does not have a corresponding first grid, a key point is determined based on all target three-dimensional points corresponding to the second grid, and the key point is updated to be a key point of the second grid.

[0109] For example, assuming that the second grid c1 on the reference curve corresponds to the first grid d1 in the first grid queue, all target 3D points corresponding to the second grid c1 (see step S22) and all historical 3D points corresponding to the first grid d1 (obtained during the processing of the previous frame of the image) can be obtained, and all historical 3D points corresponding to the first grid d1 can be used as historical 3D points corresponding to the second grid c1. On this basis, key points can be determined based on all target 3D points corresponding to the second grid c1 and all historical 3D points corresponding to the first grid d1, such as performing weighted operations on all target 3D points corresponding to the second grid c1 and all historical 3D points corresponding to the first grid d1 to obtain key points, or taking the average value of all target 3D points and all historical 3D points as the key point, or taking the maximum value of all target 3D points and all historical 3D points as the key point, or taking the minimum value of all target 3D points and all historical 3D points as the key point. After obtaining this key point, this key point is used as the key point of the second grid c1. For each second grid on the reference curve, the key point of the second grid can be obtained in the above manner.

[0110] See also Figure 6As shown, for each second grid on the reference curve, the second grid can be grid matched to obtain the first grid corresponding to the second grid in the first grid queue. On this basis, based on all the target 3D points corresponding to the second grid and all the historical 3D points corresponding to the first grid, these target 3D points and historical 3D points can be grid synthesized to obtain the key points of the second grid. By grid synthesizing these target 3D points and historical 3D points, the multi-frame point set information projected on the same grid can be combined with the surrounding grids for small-scale fitting, noise can be filtered out, and the final grid key points can be obtained.

[0111] In the process of processing the historical frame image, a reference curve can be generated based on the historical point cloud of the historical lane line, and the reference curve includes multiple first grids, and the first grid queue includes multiple first grids, and each first grid corresponds to a key point. Considering that there are multiple first grids in the first grid queue, in the grid matching process, for each second grid on the reference curve, it is necessary to find the first grid corresponding to the second grid from the first grid queue, that is, the second grid is the same grid as the first grid.

[0112] In a possible implementation, during the grid matching process, for each second grid on the reference curve, the first grid corresponding to the second grid in the first grid array may be determined in the following manner:

[0113] For each first grid in the first grid queue, the first key point corresponding to the first grid is determined based on all the historical three-dimensional points corresponding to the first grid. For example, if all the historical three-dimensional points corresponding to the first grid are known (obtained during the processing of the previous frame of the image), a weighted operation can be performed on all the historical three-dimensional points corresponding to the first grid to obtain the first key point, or the average value (or maximum value, or minimum value) of all the historical three-dimensional points corresponding to the first grid can be used as the first key point.

[0114] For the second grid on the reference curve, the second key point corresponding to the second grid is determined based on all the target three-dimensional points corresponding to the second grid. For example, if all the target three-dimensional points corresponding to the second grid are known (the acquisition method can refer to step S22), the second key point can be obtained by performing a weighted operation on all the target three-dimensional points corresponding to the second grid, or the average value (or maximum value, or minimum value) of all the target three-dimensional points corresponding to the second grid can be used as the second key point.

[0115] Based on the distance between the second key point and the first key point corresponding to each first grid in the first grid array, the first grid corresponding to the second grid can be determined from the first grid array.

[0116] For example, if the second key point is a three-dimensional point and the first key point is also a three-dimensional point, the distance between the second key point and the first key point corresponding to each first grid (i.e., the distance between the two three-dimensional points) can be calculated. Based on the distance between the second key point and the first key point corresponding to each first grid, the minimum distance can be selected. If the minimum distance is less than a threshold, the first grid corresponding to the minimum distance is used as the first grid corresponding to the second grid. Alternatively, if the minimum distance is not less than a threshold, there is no first grid corresponding to the second grid, that is, the second grid is a new grid.

[0117] For example, considering that there are multiple second grids on the reference curve, and a first grid in the first grid queue can only correspond to one second grid, and cannot correspond to multiple second grids at the same time, therefore, multiple second grids can be jointly solved to obtain the first grids corresponding to the multiple second grids. For example, formula (6) can be used to determine the first grid corresponding to each second grid.

[0118]

[0119] In formula (6), set S1 represents the set of first key points, that is, set S1 includes all first key points corresponding to the first grid in the first grid array. Set S2 represents the set of second key points, that is, set S2 includes all second key points corresponding to the second grid on the reference curve. CD represents the distance between set S1 and set S2. By solving formula (6), we can find the minimum d CD The value of d CD The combination of the values ​​of is the pairing relationship between the second grid and the first grid. x belongs to set S1, that is, x is the first key point in set S1, and y belongs to set S2, that is, y is the second key point in set S2. In formula (6), the first term represents the sum of the minimum distances from any point x in set S1 to set S2, and the second term represents the sum of the minimum distances from any point y in set S2 to set S1.

[0120] In formula (6), when x is the first first key point in set S1 and y is the first second key point in set S2, for the case where x is the second first key point in set S1, y can be the second, third, fourth second key point in set S2, etc., but cannot be the first second key point in set S2. Similarly, when x is the first first key point in set S1 and y is the second second key point in set S2, for the case where x is the second first key point in set S1, y can be the first, third, fourth second key point in set S2, etc., but cannot be the second second key point in set S2. In other words, different x cannot correspond to the same y. On this basis, by solving formula (6), the minimum d is found. CD The value of d CD The combination under the value of is the pairing relationship between the second grid and the first grid. So far, for each second grid on the reference curve, the first grid corresponding to the second grid in the first grid queue can be determined, that is, the pairing relationship between the second grid and the first grid is obtained.

[0121] For example, there is a relative mileage transformation between the second grid and the first grid. By searching for the optimal relative mileage transformation, the chamfer distance of the points in each grid (i.e., the distance between the second key point corresponding to the second grid and the first key point corresponding to the first grid) is minimized, thereby obtaining the best grid matching and projection relationship, that is, finding the pairing relationship between the second grid and the first grid. The definition of the chamfer distance is shown in formula (6), that is, by solving formula (6), the pairing relationship between the second grid and the first grid is obtained.

[0122] For example, when determining the pairing relationship between the second grid and the first grid, the maximum grid length of each lane line can be limited, and the first grid queue can be updated in a sliding window manner. If the real-time detected grid exceeds the existing first grid queue, a new related grid can be created; if the real-time detected grid falls within the existing first grid queue, the grid can be added to the point set of the first grid queue.

[0123] Step S24: construct a second grid array, where the second grid array includes key points of each second grid.

[0124] For example, assuming that the first grid queue includes the first grid d1, the first grid d2, and the first grid d3, and there are the second grid c1, the second grid c2, and the second grid c3 on the reference curve, the second grid c1 corresponds to the first grid d2, the second grid c2 corresponds to the first grid d3, and the second grid c3 is a new grid, then, the first grid d2 in the first grid queue is updated to the second grid c1, the first grid d3 in the first grid queue is updated to the second grid c2, and the second grid c3 is added to obtain the second grid queue, all the grids in the second grid queue are called second grids, and the second grid queue includes the key points of the second grid d1, the key points of the second grid c1, the key points of the second grid c2, and the key points of the second grid c3.

[0125] In the second grid queue, the key points of the second grid d1 are determined based on all historical three-dimensional points corresponding to the first grid d1, the key points of the second grid c1 are determined based on all target three-dimensional points corresponding to the second grid c1 and all historical three-dimensional points corresponding to the first grid d2, the key points of the second grid c2 are determined based on all target three-dimensional points corresponding to the second grid c2 and all historical three-dimensional points corresponding to the first grid d3, and the key points of the second grid c3 are determined based on all target three-dimensional points corresponding to the second grid c3.

[0126] In summary, the second grid queue is obtained by performing a sliding window update on the first grid queue. In the processing of the next frame of the image, the second grid queue is used as the first grid queue, and the first grid queue is updated again in a sliding window manner. In addition, the maximum grid length of the second grid queue can be limited, that is, the number of grids in the second grid queue does not exceed the maximum grid length. If the number of grids in the second grid queue exceeds the maximum grid length, the second grid is deleted from the front so that the number of grids does not exceed the maximum grid length.

[0127] In step S21-step S24, the example of the associated historical lane line that matches the target lane line is used to introduce how to obtain the second grid queue of the target lane line. If there is no associated historical lane line that matches the target lane line, that is, the target lane line is a new lane line, the second grid queue of the target lane line can be obtained in the following manner: a reference curve is generated based on the target point cloud of the target lane line, and the reference curve is rasterized according to a preset step size to obtain multiple second grids. For each target three-dimensional point in the target point cloud, the projection point of the target three-dimensional point on the reference curve is determined, and the second grid to which the projection point belongs is used as the second grid of the target three-dimensional point. For each second grid on the reference curve, a key point is determined based on all target three-dimensional points corresponding to the second grid, and the key point is updated to the key point of the second grid. Construct a second grid queue, and the second grid queue includes the key point of each second grid.

[0128] Step 309: obtain a high confidence point set based on the second grid queue. The second grid queue may include key points of multiple second grids. The high confidence point set may include key points corresponding to each second grid in the second grid queue. For example, the second grid queue may be used as a high confidence point set.

[0129] Step 310: determine an obstacle point cloud around the target lane line based on the current frame image, determine a road trend point based on the obstacle point cloud, and obtain a low confidence point set based on the road trend point.

[0130] Exemplarily, the current frame image can be input into a deep learning model, and the deep learning model determines the category of each pixel of the current frame image. The category of the pixel indicates that the pixel belongs to an obstacle (such as a guardrail and a wall, etc.) or the pixel does not belong to an obstacle. Regarding the network structure and training process of the deep learning model, there is no limitation in this embodiment, and the deep learning model can determine the category of each pixel. For example, the deep learning model can be the same as or different from the above-mentioned neural network model.

[0131] After knowing the category of each pixel point of the current frame image, all the pixels points belonging to the obstacle can be determined, and the area formed by these pixels points is the obstacle. For each pixel point of the obstacle (i.e., the pixel point of the current frame image), the depth prediction of the pixel point can be performed to obtain the three-dimensional point (i.e., three-dimensional coordinates) of the pixel point in the physical space. After performing depth prediction on each pixel point of the obstacle, the three-dimensional point of each pixel point in the physical space can be obtained, and all three-dimensional points constitute the obstacle point cloud.

[0132] After obtaining the obstacle point cloud, the obstacle point cloud may include multiple three-dimensional points, and edge three-dimensional points (such as edge three-dimensional points close to the side of the target lane line, and these edge three-dimensional points need to be located at the edge of the obstacle) can be selected from these three-dimensional points. Then, these edge three-dimensional points can be fitted, such as fitting these edge three-dimensional points using a cubic polynomial equation (such as using a cubic polynomial for least squares fitting) to obtain a fitted edge curve, and there is no restriction on this fitting method.

[0133] After obtaining the edge curve, the edge curve can be translated until the edge curve intersects with the target lane line, that is, the edge curve is spliced ​​with the target lane line. After the edge curve intersects with the target lane line, the translation operation on the edge curve is stopped, and the edge curve after the translation operation is used as the road trend curve. After obtaining the road trend curve, multiple three-dimensional points can be sampled from the road trend curve as road trend points (that is, the three-dimensional points on the road trend curve are used as road trend points).

[0134] After obtaining a plurality of road trend points (a plurality of three-dimensional points), a low-confidence point set may be acquired based on the road trend points, that is, the low-confidence point set may include a plurality of road trend points.

[0135] For example, due to the perspective transformation characteristics of the camera, the length of the lane line that can be detected in real time usually does not meet the requirements, and the trend prediction results based on historical lane lines are not effective at places where the curvature of the lane line changes. Based on this, a low-confidence point set can be constructed, and the low-confidence point set is obtained based on the obstacle point cloud. When constructing the curve equation based on the low-confidence point set, the lane line length can be increased.

[0136] For example, see Figure 7 As shown in the figure, it is a schematic diagram of the splicing of near and far key points. There are near key points A and far key points B in front of the vehicle. For the road trend curve on the wall side, multiple three-dimensional points can be sampled from the road trend curve (i.e. Figure 7 The intersection points of the midline segment and the curve) form a low confidence point set.

[0137] Step 311: Determine curve equation parameters based on the high confidence point set and the low confidence point set.

[0138] In a possible implementation, the loss value can be calculated based on the three-dimensional points in the high-confidence point set, the three-dimensional points in the low-confidence point set, the weight of the high-confidence point set, the weight of the low-confidence point set, the curve equation parameters, the mileage of the three-dimensional points in the high-confidence point set, and the mileage of the three-dimensional points in the low-confidence point set, and the curve equation parameters are calculated by solving the minimum loss value. For example, since the high-confidence point set contains three-dimensional points for lane lines and the low-confidence point set contains three-dimensional points for obstacles, the confidence of the high-confidence point set is higher than that of the low-confidence point set, that is, the weight of the high-confidence point set is greater than the weight of the low-confidence point set. The mileage of the three-dimensional points in the high-confidence point set can be arbitrarily configured according to actual needs, and the mileage of the three-dimensional points in the low-confidence point set can be arbitrarily configured according to actual needs.

[0139] For example, the following formula can be used to determine the curve equation parameters. The following formula is only an example and is not limited to this, as long as the curve equation parameters can be calculated based on the three-dimensional points in the high-confidence point set, the three-dimensional points in the low-confidence point set, the weight of the high-confidence point set, and the weight of the low-confidence point set.

[0140]

[0141] In the above formula, x i ,y i , z i represents the i-th 3D point in the high confidence point set (i.e., high confidence point set A), x j ,y j , zj represents the jth 3D point in the low confidence point set (i.e., low confidence point set B), W A Represents the weight of the high confidence point set, W B represents the weight of the low confidence point set, and the weight of the high confidence point set is W A Can be greater than the weight W of the low confidence point set B .a k , b k and c k Represents the curve equation parameter, the value of k is 0-n, that is, 3*(n+1) curve equation parameters need to be solved. i represents the mileage of the i-th 3D point in the high confidence point set. For the i-th 3D point, (n+1) mileages s can be pre-selected. i , that is, the value of k is 0-n in sequence, and a mileage s is selected for each value of k i .s j represents the mileage of the j-th 3D point in the low confidence point set. For the j-th 3D point, (n+1) mileages s can be pre-selected. j , that is, the value of k is 0-n in sequence, and a mileage s is selected for each value of k j .

[0142] In the above formula, Lx, Ly, Lz represent loss values. After substituting each 3D point in the high confidence point set A and each 3D point in the low confidence point set B into the above formula, the loss values ​​Lx, Ly, Lz are solved by trying different curve equation parameters until the minimum loss value Lx, Ly, Lz is found. The curve equation parameters under the minimum loss value Lx, Ly, Lz are the final solved curve equation parameters.

[0143] In summary, it can be seen that by performing least squares fitting on the three-dimensional points in the high-confidence point set and the mileage s, and performing least squares fitting on the three-dimensional points in the low-confidence point set and the mileage s, the fitting objective function is the above formula, and the curve equation parameters under the minimum loss value Lx, Ly, Lz can be solved.

[0144] Step 312: If there is an associated historical lane line that matches the target lane line, the curve equation of the associated historical lane line is updated based on the curve equation parameters of the target lane line, and the curve equation of the associated historical lane line is used to determine the distance matching degree corresponding to the next frame image of the current frame image. Or,

[0145] If there is no associated historical lane line that matches the target lane line, that is, the target lane line is a new lane line, the curve equation of the target lane line is determined based on the curve equation parameters of the target lane line, and the curve equation of the target lane line is used to determine the distance matching degree corresponding to the next frame image of the current frame image.

[0146] In a possible implementation, for the curve equation of the associated historical lane line or the curve equation of the target lane line, the curve equation can be shown in formula (7).

[0147]

[0148] In formula (7), a i , b i and c i It represents the curve equation parameters of the target lane line, that is, there are 3*(n+1) curve equation parameters in total, s represents the mileage of the three-dimensional point, that is, (n+1) mileages s can be selected in advance, that is, the value of i is 0-n in sequence, and a mileage s is selected for each value of i.

[0149] At this point, the curve equation of the target lane line is obtained. In the lane line detection process based on the next frame image, the curve equation of the target lane line is used as the curve equation of the historical lane line, and the above steps are repeated.

[0150] In one possible implementation, see Figure 8 As shown, it is a schematic diagram of the process of online lane mapping. First, the lane key point segmentation can be performed, and the lane-key point matching can be performed to obtain the high confidence point set A. The process can be referred to in steps 308-309. Then, the far-end trend detection can be performed to obtain the low confidence point set B. The process can be referred to in step 310. Then, the lane line rasterization update can be performed based on the high confidence point set A and the low confidence point set B, and the parameter equation fitting can be performed to obtain the curve equation parameters of the target lane line. The process can be referred to in steps 311-312.

[0151] In a possible implementation manner, all data (image data, three-dimensional point data, etc.) involved in this embodiment are acquired and used only with the knowledge and authorization of the relevant users.

[0152] It can be seen from the above technical solutions that in the embodiments of the present application, the stability and accuracy of lane line detection can be improved, and the continuous and complete output of lane lines around the vehicle can be ensured. Even if the lane line detection performance is affected by factors such as lighting, environment, ground slope, obstacle occlusion or decreased contrast of lane lines, errors in lane line detection results can be avoided, and lane lines can be accurately detected, and the front and back detection results of lane lines are consistent. A lane line detection fusion method based on sliding grids is proposed, which first smoothes the point cloud itself to improve the point cloud quality and then fits it. A near-far fusion scheme is proposed, which divides the original fitting points into high and low confidence levels, introduces constraints on the fitting by far-end trends, and improves the stability of the final curve equation.

[0153] Based on the same application concept as the above method, a lane line detection device is proposed in the embodiment of the present application, see Fig.9A FIG. 1 is a schematic diagram of the structure of a lane line detection device, which may include:

[0154] A detection module 911 is used to determine a target point cloud of a target lane line, a target line type of the target lane line, and a target ranking value of the target lane line based on a current frame image;

[0155] The processing module 912 is used to determine the distance matching degree between the target lane line and the historical lane line based on the target point cloud and the historical point cloud of the historical lane line; determine the order matching degree between the target lane line and the historical lane line based on the target ranking value and the historical ranking value of the historical lane line; determine the line type matching degree between the target lane line and the historical lane line based on the target line type and the historical line type of the historical lane line; determine the target matching degree between the target lane line and the historical lane line based on the distance matching degree, the order matching degree and the line type matching degree; wherein the historical point cloud, the historical line type and the historical ranking value are determined based on the historical frame image before the current frame image;

[0156] The matching fusion module 913 is used to determine the associated historical lane line that matches the target lane line from multiple historical lane lines based on the target matching degree between the target lane line and each historical lane line, and fuse the target lane line with the associated historical lane line; or, based on the target matching degree between the target lane line and each historical lane line, determine that the target lane line is a new lane line.

[0157] Exemplarily, if the target point cloud includes multiple target three-dimensional points, the processing module 912 determines the distance matching degree between the target lane line and the historical lane line by: determining the curve equation of the historical lane line based on the historical point cloud of the historical lane line; sampling multiple reference three-dimensional points based on the curve equation of the historical lane line; for each target three-dimensional point, calculating the distance between the target three-dimensional point and each reference three-dimensional point, and selecting the minimum distance based on the distance between the target three-dimensional point and each reference three-dimensional point; and determining the distance matching degree based on the minimum distance corresponding to each target three-dimensional point.

[0158] Exemplarily, when the processing module 912 determines the line type matching degree between the target lane line and the historical lane line, it is specifically used to: if the target line type is the same as the historical line type, determine the line type matching degree as a first value; if the target line type is different from the historical line type, determine the line type matching degree as a second value, and the second value is greater than the first value; when the processing module 912 determines the sequence matching degree between the target lane line and the historical lane line, it is specifically used to: determine the sequence matching degree based on the absolute value of the difference between the target sorting value and the historical sorting value; wherein, the sorting value of each lane line is decreased from left to right to obtain the target sorting value; or, the sorting value of each lane line is decreased from right to left to obtain the target sorting value.

[0159] Exemplarily, when the matching fusion module 913 fuses the target lane line with the associated historical lane line, it is specifically used to: update the first grid queue of the associated historical lane line based on the target point cloud to obtain an updated second grid queue; wherein the first grid queue includes multiple first grids, and the key points corresponding to each first grid are determined based on the historical point cloud, and the second grid queue includes multiple second grids, and the key points corresponding to each second grid are determined based on the historical point cloud and the target point cloud; obtain a high-confidence point set based on the second grid queue; wherein the high-confidence point set includes the key points corresponding to each second grid in the second grid queue; determine curve equation parameters based on the high-confidence point set; update the curve equation of the associated historical lane line based on the curve equation parameters, and the curve equation of the associated historical lane line is used to determine the distance matching degree corresponding to the next frame image of the current frame image.

[0160] Exemplarily, the matching fusion module 913 updates the first grid queue of the associated historical lane line based on the target point cloud, and when obtaining the updated second grid queue, it is specifically used to: generate a reference curve based on the target point cloud, and rasterize the reference curve according to a preset step size to obtain multiple second grids; for each target three-dimensional point in the target point cloud, determine the projection point of the target three-dimensional point on the reference curve, and use the second grid to which the projection point belongs as the second grid of the target three-dimensional point; for each second grid of the reference curve, determine the first grid corresponding to the second grid in the first grid queue; determine the key point based on all the target three-dimensional points corresponding to the second grid and all the historical three-dimensional points corresponding to the first grid, and update the key point to the key point of the second grid; construct a second grid queue, and the second grid queue may include the key point of each second grid.

[0161] Exemplarily, when the matching fusion module 913 determines the first grid corresponding to the second grid in the first grid queue, it is specifically used to: for each first grid in the first grid queue, determine the first key point corresponding to the first grid based on all historical three-dimensional points corresponding to the first grid; determine the second key point corresponding to the second grid based on all target three-dimensional points corresponding to the second grid; and determine the first grid corresponding to the second grid from the first grid queue based on the distance between the second key point and the first key point corresponding to each first grid in the first grid queue.

[0162] Exemplarily, when the matching fusion module 913 determines the curve equation parameters based on the high confidence point set, it is specifically used to: determine the obstacle point cloud around the target lane line based on the current frame image, determine the road trend points based on the obstacle point cloud, and obtain the low confidence point set based on the road trend points; determine the curve equation parameters based on the high confidence point set and the low confidence point set.

[0163] Exemplarily, when the matching fusion module 913 determines the curve equation parameters based on the high confidence point set and the low confidence point set, it is specifically used to: determine the curve equation parameters using the following formula:

[0164]

[0165] Among them, x i ,y i , z i represents the i-th 3D point in the high confidence point set, x j ,y j , z j represents the j-th 3D point in the low confidence point set, Lx, Ly, Lz represent the loss value, W A Represents the weight of the high confidence point set, W B represents the weight of the low confidence point set;

[0166] a k , b k and c k represents the curve equation parameter, s i represents the mileage of the i-th 3D point in the high confidence point set, s j represents the mileage of the j-th 3D point in the low confidence point set.

[0167] Based on the same application concept as the above method, an electronic device is proposed in the embodiment of the present application, see Fig. 9BAs shown, the electronic device includes: a processor 921 and a machine-readable storage medium 922, the machine-readable storage medium 922 stores machine-executable instructions that can be executed by the processor 921; the processor 921 is used to execute the machine-executable instructions to implement the lane line detection method disclosed in the above example of this application.

[0168] Based on the same application concept as the above method, an embodiment of the present application also provides a machine-readable storage medium, on which a number of computer instructions are stored. When the computer instructions are executed by a processor, the lane line detection method disclosed in the above example of the present application can be implemented.

[0169] The above-mentioned machine-readable storage medium may be any electronic, magnetic, optical or other physical storage device, which may contain or store information, such as executable instructions, data, etc. For example, the machine-readable storage medium may be: RAM (Radom Access Memory), volatile memory, non-volatile memory, flash memory, storage drive (such as hard disk drive), solid state drive, any type of storage disk (such as CD, DVD, etc.), or similar storage medium, or a combination thereof.

[0170] Based on the same application concept as the above method, an embodiment of the present application also provides a computer program product, which may include a computer program. When the computer program is executed by a processor, it implements the lane line detection method disclosed in the above example of the present application.

[0171] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the embodiments of the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0172] The above is only an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of the claims of the present application.

Claims

1. A lane line detection method, characterized in that: The method comprises: Determine the target point cloud, target line type and target sort value of the target lane line based on the current frame image; Based on the target point cloud and the historical point cloud of the historical lane line, determine the distance matching degree between the target lane line and the historical lane line; based on the target sorting value and the historical sorting value of the historical lane line, determine the order matching degree between the target lane line and the historical lane line; based on the target line type and the historical line type of the historical lane line, determine the line type matching degree between the target lane line and the historical lane line; based on the distance matching degree, the order matching degree and the line type matching degree, determine the target matching degree between the target lane line and the historical lane line; wherein the historical point cloud, the historical line type and the historical sorting value are determined based on the historical frame image before the current frame image; Based on the target matching degree between the target lane line and each historical lane line, an associated historical lane line that matches the target lane line is determined from multiple historical lane lines, and the target lane line is merged with the associated historical lane line; or, based on the target matching degree between the target lane line and each historical lane line, the target lane line is determined to be a new lane line.

2. The method according to claim 1, characterized in that If the target point cloud includes a plurality of target three-dimensional points, determining the distance matching degree between the target lane line and the historical lane line based on the target point cloud and the historical point cloud of the historical lane line includes: Determine a curve equation of the historical lane line based on the historical point cloud of the historical lane line; Sampling a plurality of reference three-dimensional points based on a curve equation of the historical lane line; For each target 3D point, calculate the distance between the target 3D point and each reference 3D point, and select the minimum distance based on the distance between the target 3D point and each reference 3D point; The distance matching degree is determined based on the minimum distance corresponding to each target three-dimensional point.

3. The method according to claim 1, characterized in that The determining, based on the target line type and the historical line type of the historical lane type, a line type matching degree between the target lane type and the historical lane type, comprises: if the target line type is the same as the historical line type, determining the line type matching degree to be a first value; if the target line type is different from the historical line type, determining the line type matching degree to be a second value, the second value being greater than the first value; The determining of the order matching degree between the target lane line and the historical lane line based on the target ranking value and the historical ranking value of the historical lane line includes: determining the order matching degree based on the absolute value of the difference between the target ranking value and the historical ranking value; wherein, the target ranking value is obtained by decreasing the ranking value of each lane line from left to right; or, the target ranking value is obtained by decreasing the ranking value of each lane line from right to left.

4. The method according to claim 1, characterized in that: The fusing the target lane line with the associated historical lane line includes: The first grid array associated with the historical lane line is updated based on the target point cloud to obtain an updated second grid array; wherein the first grid array includes a plurality of first grids, and the key point corresponding to each first grid is determined based on the historical point cloud, and the second grid array includes a plurality of second grids, and the key point corresponding to each second grid is determined based on the historical point cloud and the target point cloud; Acquire a high confidence point set based on the second grid array; wherein the high confidence point set includes key points corresponding to each second grid in the second grid array; Determining curve equation parameters based on the high confidence point set; The curve equation of the associated historical lane line is updated based on the curve equation parameter, and the curve equation of the associated historical lane line is used to determine the distance matching degree corresponding to the next frame image of the current frame image.

5. The method according to claim 4, characterized in that The first grid array associated with the historical lane line is updated based on the target point cloud to obtain an updated second grid array, including: Generate a reference curve based on the target point cloud, and rasterize the reference curve according to a preset step size to obtain a plurality of second grids; for each target three-dimensional point in the target point cloud, determine the projection point of the target three-dimensional point on the reference curve, and use the second grid to which the projection point belongs as the second grid of the target three-dimensional point; For each second grid of the reference curve, determine the first grid corresponding to the second grid in the first grid queue; determine a key point based on all target three-dimensional points corresponding to the second grid and all historical three-dimensional points corresponding to the first grid, and update the key point to the key point of the second grid; A second grid array is constructed, wherein the second grid array includes key points of each second grid.

6. The method according to claim 5, characterized in that The determining the first grid corresponding to the second grid in the first grid queue includes: For each first grid in the first grid queue, determining a first key point corresponding to the first grid based on all historical three-dimensional points corresponding to the first grid; Determine a second key point corresponding to the second grid based on all target three-dimensional points corresponding to the second grid; Based on the distance between the second key point and the first key point corresponding to each first grid in the first grid array, the first grid corresponding to the second grid is determined from the first grid array.

7. The method according to claim 4, characterized in that The determining of curve equation parameters based on the high confidence point set comprises: Determine an obstacle point cloud around the target lane line based on the current frame image, determine a road trend point based on the obstacle point cloud, and acquire a low confidence point set based on the road trend point; Curve equation parameters are determined based on the high confidence point set and the low confidence point set.

8. The method according to claim 7, characterized in that Determining the curve equation parameters based on the high confidence point set and the low confidence point set includes: determining the curve equation parameters using the following formula: Among them, x i ,y i , z i represents the i-th 3D point in the high confidence point set, x j ,y j , z j represents the j-th 3D point in the low confidence point set, Lx, Ly, Lz represent the loss value, W A represents the weight of the high confidence point set, W B represents the weight of the low confidence point set; a k , b k and c k represents the curve equation parameter, s i represents the mileage of the i-th 3D point in the high confidence point set, s j represents the mileage of the j-th 3D point in the low confidence point set.

9. A lane line detection device, characterized in that: The device comprises: A detection module, used to determine a target point cloud of a target lane line, a target line type of the target lane line, and a target ranking value of the target lane line based on a current frame image; A processing module, for determining a distance matching degree between the target lane line and the historical lane line based on the target point cloud and the historical point cloud of the historical lane line; determining a sequence matching degree between the target lane line and the historical lane line based on the target sorting value and the historical sorting value of the historical lane line; determining a line type matching degree between the target lane line and the historical lane line based on the target line type and the historical line type of the historical lane line; determining a target matching degree between the target lane line and the historical lane line based on the distance matching degree, the sequence matching degree and the line type matching degree; wherein the historical point cloud, the historical line type and the historical sorting value are determined based on the historical frame image preceding the current frame image; A matching and fusion module is used to determine, from multiple historical lane lines, an associated historical lane line that matches the target lane line based on the target matching degree between the target lane line and each historical lane line, and fuse the target lane line with the associated historical lane line; or, based on the target matching degree between the target lane line and each historical lane line, determine that the target lane line is a new lane line.

10. An electronic device, characterized in that: include: a processor and a machine-readable storage medium storing machine-executable instructions executable by the processor; The processor is used to execute machine executable instructions to implement the method described in any one of claims 1-8.

Images