A 3D lane line prediction method

Abstract

The present invention relates to the field of autonomous driving technology, and discloses a 3D lane line prediction method, the method comprising: extracting the vehicle coordinates and lane centerline coordinates of the vehicles contained in the current image based on the current image acquired by the vehicle; dividing the vehicle coordinates and lane centerline coordinates based on preset rules to generate a lane centerline fitting coordinate set; and fitting and generating 3D lane lines based on the lane centerline fitting coordinate set. The present invention extracts the vehicle coordinates and lane centerline coordinates of the vehicles contained in the current image, and divides the vehicle coordinates and the lane centerline coordinates, that is, adds the vehicle coordinates of all vehicles contained in the image to the corresponding lane centerline coordinate set, and then fits and generates 3D lane lines, so that the predicted 3D lane lines are more accurate.

Description

Technical Field

[0001] The present invention relates to the field of autonomous driving technology, and in particular to a 3D lane line prediction method. Background Art

[0002] In the prior art, a detection model is mainly obtained by training based on a neural network. A picture of a lane line is input into the detection model, and the predicted 3D lane line is directly output through the detection model.

[0003] However, in the existing 3D lane detection task, lanes are distributed over a wide range on the entire feature map. Compared with target detection, lane detection pays more attention to global information, and its feature distribution is relatively sparse (because each lane is relatively narrow). If you want the model to pay attention to global information, you need to calculate a global attention, but this introduces a large amount of calculation, resulting in the model not being able to be well applied in real-time reasoning scenarios; if you use sparse points for learning, it will take a long time for the model to pay attention to sparsely distributed lanes, and the accuracy will also decrease. Summary of the invention

[0004] The technical problem to be solved by the present invention is to provide a 3D lane line prediction method in view of the problems existing in the prior art.

[0005] To solve the above technical problems, an embodiment of the present invention provides a 3D lane line prediction method, including: based on a current image acquired by the vehicle, extracting the vehicle coordinates and lane centerline coordinates of the vehicle contained in the current image; based on preset rules, dividing the vehicle coordinates and the lane centerline coordinates to generate a lane centerline fitting coordinate set; based on the lane centerline fitting coordinate set, fitting and generating a 3D lane line.

[0006] Based on the above technical solution, the present invention can also be improved as follows.

[0007] Furthermore, based on preset rules, the vehicle coordinates and the lane centerline coordinates are divided to generate a lane centerline fitting coordinate set, including: determining the relative position relationship between the vehicle and the lane centerline contained in the current image, and determining the belonging relationship between the vehicle and the lane centerline according to the relative position relationship; and dividing the vehicle coordinates and the corresponding lane centerline coordinates into a lane centerline fitting coordinate set according to the belonging relationship.

[0008] Further, determining the relative position relationship between the vehicle contained in the current image and the lane centerline, and determining the relationship between the vehicle and the lane centerline according to the relative position relationship, including:

[0009] Determine the shortest distance between the center point of the vehicle and the center line of each lane, as well as the angle between the vehicle's driving direction and the center line of each lane; determine the relationship between the vehicle and the center line of the lane based on whether the shortest distance and / or angle meet preset conditions.

[0010] Furthermore, based on preset rules, the vehicle coordinates and the lane centerline coordinates are divided to generate a lane centerline fitting coordinate set, which also includes: determining the driving state of the vehicle contained in the current image; filtering out valid vehicle coordinates in the vehicle coordinates according to the driving state; and dividing the valid vehicle coordinates and the corresponding lane centerline coordinates into a lane centerline fitting coordinate set according to the belonging relationship.

[0011] Further, determining the driving state of the vehicle contained in the current image includes: constructing the state information of the vehicle; comparing the state information of the vehicle with the state cluster center of the vehicle in a preset driving state; and determining the driving state of the vehicle according to the comparison result.

[0012] Furthermore, the status information includes: the angle between the center line of the vehicle and the center line of the lane to which it belongs, and the relative distance between at least two coordinate points of the vehicle and the lane line; when comparing the status information of the vehicle with the state cluster center of the vehicle under a preset driving state, the relative distance is the first priority and the angle is the second priority.

[0013] Furthermore, the vehicle coordinates and the lane centerline coordinates include: three-dimensional coordinates of the coordinate point and corresponding auxiliary correction information, and the auxiliary correction information represents the degree of deviation between the coordinate point predicted by the model and the actual coordinate point.

[0014] Furthermore, the auxiliary correction information formula is as follows:

[0015]

[0016] Among them, f(x) represents the auxiliary correction information of the current coordinate point, the value is in the range of [0,1], x represents the Euclidean distance between the coordinate point predicted by the model and the actual coordinate point, and c represents the preset threshold.

[0017] Furthermore, when the vehicle coordinates and the lane centerline coordinates are divided, when it is determined that the vehicle belongs to multiple lane centerlines at the same time, the three-dimensional coordinates in the vehicle coordinates are simultaneously divided into the lane centerline coordinate sets corresponding to the corresponding multiple lane centerlines, and the auxiliary correction information in the vehicle coordinates is evenly distributed to the lane centerline coordinate sets corresponding to the corresponding multiple lane centerlines.

[0018] Furthermore, based on the lane centerline fitting coordinate set, 3D lane lines are fitted and generated, including: extracting the horizontal coordinate, vertical coordinate and auxiliary correction information of each coordinate point in the lane centerline fitting coordinate set, and constructing a lane centerline coordinate point set to be generated; using the lane centerline coordinate point set to be generated to follow the normal distribution based on the latitude coordinate, 3D lane lines are fitted and generated.

[0019] The beneficial effect of the present invention is: by extracting the vehicle coordinates and lane centerline coordinates of the vehicles contained in the current image, and dividing the vehicle coordinates and lane centerline coordinates, that is, adding the vehicle coordinates of all vehicles contained in the picture to the corresponding lane centerline coordinate set, and then fitting and generating 3D lane lines, the predicted 3D lane lines are made more accurate.

[0020] Additional aspects and advantages of the present invention will be given in part in the following description and in part will be obvious from the following description or learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 A flowchart of a 3D lane line prediction method shown as an exemplary embodiment of the present application;

[0022] Figure 2 A framework diagram of a 3D lane line prediction method shown as an exemplary embodiment of the present application;

[0023] Figure 3 This is a 3D lane detection framework diagram used in existing technologies;

[0024] Figure 4 for Figure 1 The 3D lane detection framework diagram used in step S101 of the illustrated embodiment in an exemplary embodiment;

[0025] Figure 5 Schematic diagram of the object frame representing the vehicle output by the 3D lane detection framework;

[0026] Figure 6 for Figure 1 A flowchart of step S102 in the illustrated embodiment in an exemplary embodiment;

[0027] Figure 7 for Figure 1 A flowchart of step S102 in the illustrated embodiment in an exemplary embodiment;

[0028] Figure 8 for Figure 1 The step S103 in the illustrated embodiment is a flow chart in an exemplary embodiment. DETAILED DESCRIPTION

[0029] The following describes the embodiments of the present disclosure through specific examples, and those skilled in the art can easily understand other advantages and effects of the present disclosure from the contents disclosed in this specification. Obviously, the described embodiments are only part of the embodiments of the present disclosure, rather than all of the embodiments. The present disclosure can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present disclosure. It should be noted that the following embodiments and features in the embodiments can be combined with each other without conflict. Based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present disclosure.

[0030] It should be noted that various aspects of the embodiments within the scope of the appended claims are described below. It should be apparent that the aspects described herein may be embodied in a wide variety of forms, and any specific structure and / or function described herein is merely illustrative. Based on the present disclosure, it should be understood by those skilled in the art that an aspect described herein may be implemented independently of any other aspect, and two or more of these aspects may be combined in various ways. For example, any number of aspects described herein may be used to implement the device and / or practice the method. In addition, other structures and / or functionalities other than one or more of the aspects described herein may be used to implement this device and / or practice this method.

[0031] The method for detecting 3D lane lines in the prior art is:

[0032] The lane lines are structured by defining a series of predefined lane lines in 3D space and projecting them onto the 2D image features to sample the features. The sampled features are used to predict the offset of the real lane lines relative to the predefined lane lines. The offset and the coordinates of the predefined lane lines are added to obtain the final 3D lane lines.

[0033] The problems existing in the above prior art are:

[0034] The reason why Deformable DETR can achieve good results in the field of target detection is that the distribution of objects in target detection is relatively concentrated, so the model can learn the area of ​​real interest through learning. However, in the 3D lane line detection task, the lane lines are distributed over a wide range on the entire feature map. Compared with target detection, lane line detection pays more attention to global information, and its feature distribution is relatively sparse (because each lane line is relatively narrow). If you want the model to pay attention to global information, you need to calculate a global attention, but this introduces a large amount of calculation, resulting in the model not being able to be well applied in real-time reasoning scenarios; if sparse points are used for learning, it will take a long time for the model to pay attention to sparsely distributed lane lines, and the accuracy will also decrease. Therefore, it is difficult to find a compromise between real-time reasoning and accuracy.

[0035] The reasons for the above problems are:

[0036] In an image, according to the principle of camera imaging, lanes in farther areas have fewer pixels in the image than lanes in closer areas. This will result in the model being able to use more nearby information and less distant information during the reasoning process. Therefore, the model's prediction results at farther distances are not so accurate, which is also a problem that cannot be solved by existing technologies.

[0037] Based on the above-described problems, the following solution is proposed. The solution provided by the embodiment of the present invention can be applied to any application scenario where the detected lane line needs to be corrected. It should be noted that the lane line appearing in this solution refers to the center line of the lane, that is, the center line of the lane line drawn on the road surface.

[0038] The solution provided in the embodiments of the present invention can be executed by any electronic device, for example, a terminal device, or jointly executed by a terminal device and a server. The server can be an independent server, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms, etc., and is not limited here.

[0039] Embodiment 1

[0040] Figure 1 FIG. 1 is a flow chart of a 3D lane line prediction method shown in an exemplary embodiment of the present application. Figure 1 As shown, in an exemplary embodiment, the 3D lane line prediction may include steps S101 to S104, which are described in detail as follows:

[0041] S101, based on the current image acquired by the vehicle, extracting the vehicle coordinates and lane centerline coordinates of the vehicle contained in the current image;

[0042] S102, dividing the vehicle coordinates and the lane centerline coordinates based on a preset rule to generate a lane centerline fitting coordinate set;

[0043] S103, generating a 3D lane line by fitting based on the lane centerline fitting coordinate set.

[0044] It should be noted that the 3D lane line generated by fitting in the embodiment of the present invention may be a lane center line generated by fitting, or may be two lane lines generated by fitting. When two lane lines are generated by fitting, the two lane lines may be generated by fitting based on the lane center line generated by fitting, according to a preset lane width, or according to the distance between the two lane center lines.

[0045] Taking the lane centerline generated by fitting as an example, this embodiment is described. The vehicle contained in the current image extracted in this embodiment is a vehicle within the detection range of the image acquisition device in front of the vehicle, wherein the image acquisition device can be a monocular camera / camera, a binocular camera / camera, a multi-camera / camera, and a surround view camera / camera.

[0046] In this embodiment, the following coordinate system is established: the vehicle is taken as the origin, the x-axis points to the right side of the vehicle, the y-axis points to the front / forward direction of the vehicle, and the z-axis points upward.

[0047] like Figure 2 As shown, the embodiment of the present invention adds the result of 3D target detection (i.e., the vehicle coordinates of the vehicle contained in the current image) to the result of 3D lane centerline prediction, which is equivalent to adding several restriction conditions to the already obtained lane centerline. These restriction conditions can further improve the accuracy of 3D lane line curve fitting and reduce the error range of the fitting curve. In this way, the variance of each parameter of the final fitted lane centerline can be reduced, that is, the distance between the predicted lane line and the real lane line is closer, and a more accurate result is obtained.

[0048] Figure 3 This is the 3D lane detection framework diagram used in the existing technology. Figure 4 for Figure 1 The 3D lane line detection framework diagram used in step S101 of the illustrated embodiment in an exemplary embodiment is shown in FIG. Figure 5 Schematic diagram of the object frame representing the vehicle output by the 3D lane detection framework.

[0049] In an exemplary embodiment, a surround view camera is used as an example for explanation. The embodiment of the present invention first inputs the surround view photo taken by the vehicle at the current moment into the 3D lane line detection model, and extracts the vehicle coordinates and lane centerline coordinates of the vehicle contained in the current image. Specifically, the three-dimensional coordinates of the eight vertices of the object frame representing the vehicle in the current image and the coordinates of the center points of the front and rear surfaces of the object frame are extracted, where the coordinates of the center points of the front and rear surfaces can be calculated based on the three-dimensional coordinates of the eight vertices of the object frame; and the three-dimensional coordinates of each coordinate point representing the lane centerline in the current image are extracted.

[0050] Compared to Figure 3 The 3D lane detection framework shown, Figure 4 The 3D lane detection framework shown in the figure adds a 3D object detection branch to the 3D lane detection task to assist in the prediction of lane lines. Figure 4 The input and output of the model shown are as follows:

[0051] Model input: Surround view images around the vehicle;

[0052] Model output: all vehicles (where each vehicle can be represented by eight three-dimensional coordinate points representing the vehicle object box and the coordinates of the center points of the front and rear of the vehicle); Figure 5 As shown;

[0053] All lane lines (where each lane line is represented as a set of coordinate points, each of which contains 3D coordinates).

[0054] After extracting the vehicle coordinates and lane centerline coordinates of the vehicles contained in the current image, based on the preset rules, the vehicle coordinates are divided into corresponding lane centerline coordinates to generate a lane centerline fitting coordinate set; based on the lane centerline fitting coordinate set, 3D lane lines are fitted and generated. The embodiment of the present invention adds the vehicle coordinates of all vehicles contained in the image to the corresponding lane centerline coordinate set, and then fits and generates 3D lane lines, so that the predicted 3D lane lines are more accurate. Among them, each lane centerline fitting coordinate set corresponds to a fitted 3D lane line (lane centerline), and the coordinate set consists of multiple coordinate points.

[0055] It should be noted that in the process of dividing the vehicle coordinates, the preset rules can be set according to actual conditions. For example, the vehicle coordinates can be directly divided according to the distance between the lane centerline coordinates, and the vehicle coordinates can be combined with the nearest lane centerline coordinates to form / generate a lane centerline fitting coordinate set.

[0056] Embodiment 2

[0057] During the development process, the researchers found that different driving conditions of the vehicle would affect the fitted 3D lane lines. Therefore, in order to improve the accuracy of the vehicle coordinate division, after having the several vehicle coordinates and lane centerline coordinate points detected by the model in step 101, this embodiment also shows a method for generating a lane centerline fitting coordinate set, that is, determining which lane centerline each vehicle is matched to.

[0058] It should be noted that the second embodiment optimizes step S102 based on the first embodiment, divides the vehicle coordinates and the lane centerline coordinates based on preset rules, and generates relevant content of the lane centerline fitting coordinate set.

[0059] See also Figure 6 , Figure 6 for Figure 1 The step S102 in the illustrated embodiment is a flow chart in an exemplary embodiment. Figure 6 As shown, step S102 may specifically include steps S201 to S202. The lane centerline fitting coordinate set is generated through the above steps, which is described in detail as follows:

[0060] S201, determining the relative position relationship between the vehicle and the lane centerline contained in the current image, and determining the relationship between the vehicle and the lane centerline according to the relative position relationship.

[0061] S2011, determining the relative position relationship between the vehicle and the lane centerline contained in the current image may include: determining the shortest distance between the center point of the vehicle and each lane centerline, and the angle between the vehicle's driving direction and each lane centerline.

[0062] 1) Calculation of distance

[0063] Determining the shortest distance between the center point of the vehicle and the center line of each lane can specifically include: determining the shortest distance between the front and rear center points of the object frame representing the vehicle and the center line of each lane respectively; and taking the average of the two shortest distances as the shortest distance between the center point of the vehicle and the center line of the lane.

[0064] That is, in the embodiment of the present invention, the shortest distance between the predicted coordinates of the front and rear center points of the object frame representing the vehicle and the coordinate point set corresponding to the center line of each lane is first calculated as:

[0065]

[0066] Where M is the number of all lane center lines predicted, (dx front ,dy front ), (dx back ,dy back) is the shortest distance between the front and rear center points of the predicted object frame representing the vehicle and the i-th predicted lane centerline, where the z coordinate of the vehicle has no reference value and is therefore not included in the calculation.

[0067] The Euclidean distances between the front and rear center points of the object frame representing the vehicle and the corresponding predicted lane center line are calculated respectively; the average of the two front and rear Euclidean distances is taken as the distance between the center point of the vehicle and the lane center line.

[0068] 2) Calculation of angles

[0069] Determining the angle between the vehicle's driving direction and the center line of each lane may specifically include: determining the center point of the vehicle and the coordinate point on the lane center line that is closest to the center point of the vehicle, recorded as the angle reference point; determining the tangent direction vector of the lane center line at the angle reference point; determining the angle between the line connecting the center points of the front and rear doors of the vehicle and the adjacent tangent direction at the angle reference point, to obtain the angle between the vehicle's driving direction and the center line of each lane.

[0070] That is to say, in the embodiment of the present invention, Figure 5 The front and rear center points of the object frame representing the vehicle are connected to obtain a line segment, and then the midpoint of this line segment is calculated as the center point of the vehicle; the coordinate point on the lane centerline that is closest to the vehicle center point is determined, and the tangent of the lane centerline at this coordinate point is calculated to obtain a tangent direction vector. The angle between the vehicle and the lane centerline is calculated with the vector corresponding to the line connecting the front and rear center points of the object frame representing the vehicle, and the angle is unified to [0,90°].

[0071] S2012, determining the relationship between the vehicle and the lane centerline based on the relative position relationship may include: determining the relationship between the vehicle and the lane centerline based on whether the closest distance and / or angle meets a preset condition.

[0072] The embodiment of the present invention mainly determines the relationship between the vehicle and the lane centerline based on the distance between the center point of the vehicle and the lane centerline and / or the angle between the center line of the vehicle and the lane centerline. For example, the vehicle can be divided into the lane centerline corresponding to the minimum distance only based on the distance between the two; the vehicle can also be divided into the lane centerline corresponding to the minimum angle only based on the angle between the two; the judgment can also be made based on the distance and the angle between the two. The specific preset conditions can be as follows:

[0073] The lane centerline corresponding to the minimum value of the closest distance is taken as the lane centerline to which the current vehicle belongs; if there are multiple minimum values ​​of the closest distance, the lane centerline corresponding to the minimum distance and the minimum angle is taken as the lane centerline to which the current vehicle belongs; if there are multiple minimum values ​​of the closest distance and multiple minimum angles, the lane centerlines corresponding to the multiple minimum values ​​of the closest distance and the minimum angle are all taken as the lane centerline to which the current vehicle belongs.

[0074] That is to say, in the embodiment of the present invention, for each vehicle, all lane center lines are traversed and classified as the lane center line closest to it; if such lane center line is not unique, the angle between such vehicle and each lane center line is calculated and classified as the lane center line with the smallest angle; if it is still not unique, the current vehicle is assigned to all lane center lines with the smallest angle and the shortest distance.

[0075] This step outputs a set of several vehicles and lane centerlines, where each set contains only one lane centerline, but may include multiple vehicles that are determined to belong to the current lane centerline according to the rules.

[0076] S202: Group the vehicle coordinates and the corresponding lane centerline coordinates into a lane centerline fitting coordinate set according to the belonging relationship.

[0077] It should be noted that when dividing the vehicle coordinates and the lane centerline coordinates, when it is determined that the vehicle belongs to multiple lane centerlines at the same time, the three-dimensional coordinates in the vehicle coordinates are simultaneously divided into the lane centerline coordinate sets corresponding to the corresponding multiple lane centerlines.

[0078] The embodiment of the present invention assigns each detected vehicle to the corresponding lane centerline through the matching rule between the vehicle and the lane centerline, so that when fitting the lane centerline later, not only the existing lane centerline coordinates but also the added vehicle coordinates can be used, thereby improving the accuracy of lane centerline fitting.

[0079] After determining the relationship between the vehicle and the lane centerline, the available information of the vehicle belonging to each lane centerline is screened, because different vehicles may be in different states: going straight, changing lanes, turning around, and the vehicle coordinates in different states will have different effects on the fitting results. Next, for each vehicle and lane centerline set obtained in step 102, the vehicle state is judged using the vehicle and lane centerline coordinate points obtained in step 101, and useful information is further screened.

[0080] See also Figure 7 , Figure 7 for Figure 1 The step S102 in the illustrated embodiment is a flow chart in an exemplary embodiment. Figure 7 As shown, step S102 may specifically include steps S201, S203 and S204. The lane centerline fitting coordinate set is generated through the above steps, which is described in detail as follows:

[0081] S201, determining the relative position relationship between the vehicle and the lane centerline contained in the current image, and determining the relationship between the vehicle and the lane centerline according to the relative position relationship.

[0082] S203, determining the driving state of the vehicle contained in the current image, and filtering out valid vehicle coordinates from the vehicle coordinates according to the driving state.

[0083] Wherein, determining the driving state of the vehicle contained in the current image includes: constructing the state information of the vehicle; comparing the state information of the vehicle with the state cluster center of the vehicle in a preset driving state; and determining the driving state of the vehicle according to the comparison result.

[0084] It should be noted that the status information of the vehicle in the embodiment of the present invention includes: the angle between the center line of the vehicle and the center line of the lane to which it belongs, and the relative distance between at least two coordinate points of the vehicle and the lane line; that is, a preset three-dimensional coordinate, three-dimensional coordinate (angle, front distance, rear distance).

[0085] The embodiment of the present invention also pre-generates the state cluster center of the vehicle in a preset driving state, and the cluster center is calculated as follows:

[0086] Calculate the angle between the center line of each vehicle in the training data and the center line of the lane to which it belongs, and the Euclidean distance between the center point of the front and rear of the object frame representing the vehicle and the center line of the lane to form a three-dimensional coordinate (angle, front distance, rear distance). With such coordinate points, cluster these coordinate points into three categories by clustering, and obtain the cluster centers of three categories. Since the distance between the vehicle and the center line of the lane is the closest and the angle is the smallest in the straight-ahead scenario, followed by lane change and finally U-turn, the three cluster centers can be labeled according to clustering and angle. The sorting method of the three cluster centers is the same as that in the vehicle matching rule, that is, distance is the first priority and angle is the second priority. In this way, the state cluster centers of the vehicles in the three scenarios are obtained. In the prediction process, it is only necessary to calculate the distance between the current vehicle and each cluster center and classify it into the nearest category. If multiple categories are the same, it can be considered that the vehicle may have multiple behaviors, which are assigned to each category.

[0087] Filter out valid vehicle coordinates from the vehicle coordinates based on the driving status, which may include the following situations:

[0088] 1) If the vehicle is moving straight, the front and rear coordinate points of the vehicle can be used directly as valid vehicle coordinates to assist in correcting the lane centerline;

[0089] 2) If the vehicle is changing lanes (crossing the lane line drawn on the road), the coordinate points behind the vehicle are used as valid vehicle coordinates to assist in correcting the lane centerline, while the coordinate points in front need to be screened out due to large deviations;

[0090] 3) If the vehicle is making a U-turn, the U-turn is mainly divided into two stages: adjusting the vehicle angle to about 90 degrees and changing the vehicle to another lane. In this scenario, the vehicle information cannot be used, so such data is directly filtered out.

[0091] S204: grouping the valid vehicle coordinates and the corresponding lane centerline coordinates into a lane centerline fitting coordinate set according to the belonging relationship.

[0092] It should be noted that Figure 7 The embodiment shown is in Figure 6 On the basis of the illustrated embodiment, a step of filtering valid information of vehicle coordinates is added, that is, valid coordinates in the vehicle coordinates are filtered out according to the vehicle state, thereby improving the accuracy of the lane centerline generated by fitting.

[0093] After having the lane centerline fitting coordinate set, the corresponding lane centerline can be generated by fitting the coordinate set. In most scenarios, the shape of the lane centerline is relatively regular, so the embodiment of the present invention can use a quadratic function to simulate the shape of the lane centerline:

[0094] y=t 1 x 2 +t 2 x+t 3 ;

[0095] Among them, t 1 ,t 2 ,t 3 are the lane centerline parameters to be estimated, (x, y) are the horizontal and vertical coordinates of the lane centerline coordinate point, and the lane centerline fitting coordinate set is used to estimate the three parameters of the lane centerline.

[0096] Embodiment 3

[0097] In order to improve the accuracy of fitting and generating 3D lane lines, this embodiment also adds auxiliary correction information to the vehicle coordinates and lane centerline coordinates during the fitting process. The auxiliary correction information represents the degree of offset between the coordinate points predicted by the model and the actual coordinate points. The greater the offset, the smaller the auxiliary correction information; and vice versa. The offset in the embodiment of the present invention can refer to the Euclidean distance between the actual coordinate point and the predicted coordinate point, or it can refer to the difference between the two coordinates (e.g., it can be only the difference in the x-axis coordinates).

[0098] Embodiment 3 optimizes step S103 based on embodiment 1 or embodiment 2 to fit and generate relevant content of 3D lane lines based on the lane centerline fitting coordinate set.

[0099] It should be noted that before fitting and generating 3D lane lines based on the lane centerline fitting coordinate set, Figure 1 In an exemplary embodiment, step S101 in the illustrated embodiment, extracting the vehicle coordinates and lane centerline coordinates of the vehicle contained in the current image may include: three-dimensional coordinates of the coordinate points and corresponding auxiliary correction information.

[0100] In the embodiment of the present invention, the auxiliary correction information is calculated by substituting the calculated Euclidean distance (degree of deviation) between the real coordinate point and the predicted coordinate point into the following function to obtain an auxiliary correction information in the range of [0,1]:

[0101]

[0102] Among them, x is the Euclidean distance between the real coordinate point and the predicted coordinate point, c is a threshold set manually, and f(x) is the output of the function, with a value in the range of [0,1].

[0103] Take the lane centerline as an example:

[0104] The 3D lane centerline output by the model is in the form of an ordered set of 3D coordinate points Where N′ is a fixed value, which is the number of coordinate points in each predicted 3D lane centerline, (x i ′,y i ′,z i ′) is the three-dimensional coordinate of the predicted coordinate point, c′ i is the auxiliary correction information of the predicted coordinate point. The true value coordinates of the 3D lane line are also an ordered set of 3D coordinate points Here, N is also a fixed value.

[0105] The Euclidean distance d between two coordinate points i The calculation formula is:

[0106]

[0107]

[0108] Among them, (dx i ,dy i ,dz i ) is the shortest distance between the i-th predicted coordinate point and the true coordinate point set.

[0109] In the model training stage, the predicted auxiliary correction information can be made close to the calculated auxiliary correction information through neural network training, so that in the inference stage, a relatively accurate predicted auxiliary correction information can be directly obtained.

[0110] For the detection frame of the vehicle, the coordinates of the center points of the front and rear faces can be calculated based on the vertex coordinates of the corresponding faces. Similarly, the corresponding auxiliary correction information can be obtained through the above method.

[0111] It should be noted that, based on the second embodiment, when the vehicle coordinates and the lane centerline coordinates are divided, when it is determined that the vehicle belongs to multiple lane centerlines at the same time, the three-dimensional coordinates in the vehicle coordinates are simultaneously divided into the lane centerline coordinate sets corresponding to the corresponding multiple lane centerlines, and the auxiliary correction information in the vehicle coordinates is evenly distributed to the lane centerline coordinate sets corresponding to the corresponding multiple lane centerlines (that is, the auxiliary correction information is divided by n, where n is the number of lane centerlines that meet the requirements).

[0112] Similarly, after filtering out the valid vehicle coordinates in the vehicle coordinates according to the driving status of the vehicle, when it is determined that the vehicle belongs to multiple lane centerlines at the same time, the three-dimensional coordinates in the valid vehicle coordinates are simultaneously divided into the lane centerline coordinate sets corresponding to the corresponding multiple lane centerlines, and the auxiliary correction information in the valid vehicle coordinates is evenly distributed to the lane centerline coordinate sets corresponding to the corresponding multiple lane centerlines (that is, the auxiliary correction information is divided by n, where n is the number of lane centerlines that meet the requirements).

[0113] See also Figure 8 , Figure 8 for Figure 1 The step S103 in the embodiment shown is a flow chart in an exemplary embodiment. Figure 8 As shown, step S103 may specifically include steps S301 to S302. The lane centerline fitting coordinate set is generated through the above steps, which are described in detail as follows:

[0114] S301, extracting the horizontal coordinate, vertical coordinate and auxiliary correction information of each coordinate point in the lane centerline fitting coordinate set, and constructing a lane centerline coordinate point set to be generated.

[0115] Among them, the lane line coordinate point set to be generated is:

[0116]

[0117] Among them, (x i ,y i , c i ) is the coordinates and auxiliary correction information of the i-th coordinate point, N 1 、N 2They are the number of originally predicted lane centerline coordinate points and the number of valid vehicle coordinate points divided into the current lane centerline coordinate set.

[0118] S302, using the lane centerline coordinate point set to be generated, based on the latitude coordinates obeying the normal distribution, to fit and generate a 3D lane line.

[0119] Specifically, we use f(x, θ) to represent the lane function, where x is the input horizontal coordinate and θ is the three parameters to be estimated. θ can be any real number. Assuming that each coordinate point follows a normal distribution, for a certain coordinate point (x i ,y i , c i ), whose ordinate follows the normal distribution:

[0120]

[0121] The mean is:

[0122] u i =f(x i ,θ),

[0123] The variance is:

[0124]

[0125] In the formula, inf is a large constant set in advance Its specific value can be determined according to the actual situation, that is, when the auxiliary correction information of the coordinate point is too small, it is considered that the prediction of this point is very inaccurate, and a larger variance is set. Here t is an artificially set threshold and is also a constant.

[0126] With the assumption of normal distribution, we can estimate the coordinate point i =(x i ,y i , c i ) occurs, that is, the variable θ takes the value θ^ at the point point i Probability of occurrence:

[0127]

[0128] Here we may as well assume that the predictions of each coordinate point are independent of each other, then we have:

[0129]

[0130] That is, the probability of this set of coordinate points appearing when the variable θ = θ^, and what needs to be calculated in this scheme is That is, under the premise of predicting the generation of such a set of coordinate points, the probability of the parameter θ=θ^ in the lane line function.

[0131] Finally, the parameter with the maximum probability is taken as the final estimated result. It is not convenient. This solution is based on the following Bayesian formula:

[0132]

[0133] You can get:

[0134]

[0135] The lane line parameter θ can be any set of values ​​in the parameter space. Here, we assume that it obeys a uniform distribution. It only depends on the predicted coordinate points and has nothing to do with the parameters θ. In the end, we only need to maximize The parameter θ that maximizes the probability of the current predicted coordinate point can be obtained.

[0136] That is, the optimization goal is to maximize the following formula:

[0137]

[0138] In order to simplify the calculation, we can take the logarithm of this probability for calculation. Since the logarithmic function is monotonically increasing, the logarithm of the maximized probability is equivalent to the maximized probability:

[0139]

[0140] Based on the above formula, the three parameters t 1 ,t 2 ,t 3 Taking partial derivatives, we can get:

[0141]

[0142] Let the partial derivative be 0, and we get three equations, which can be solved for t 1 ,t 2 ,t 3 These three unknown quantities. The output lane line can be expressed as y = t 1 x 2 +t 2 x+t 3 , based on the obtained quadratic function expression of the lane line, the target three-dimensional coordinate point of the lane center line can be determined.

[0143] It should be noted that after obtaining the fitted lane centerline, the embodiment of the present invention can also obtain the lane line drawn on the road surface according to the lane centerline. Specifically, it can be determined according to the lane centerline and the lane width, or it can be determined according to two adjacent lane centerlines.

[0144] From the above, it can be seen that in the embodiment of the present invention, the result of 3D target detection is added to the result of 3D lane line detection, which is equivalent to adding several restriction conditions to the already obtained lane center line. These restriction conditions can further limit the fitting of the curve and narrow the range of the fitting curve. In this way, the variance of each parameter of the final fitted lane line can be reduced, that is, the predicted lane center line is more likely to be closer to the actual lane center line, and a relatively more accurate result is obtained.

[0145] Moreover, after introducing 3D target detection to detect road vehicles in the embodiment of the present invention, the trajectory of the lane centerline in the local area can be corrected according to the position of the vehicle. At the same time, the continuity of the lane centerline trajectory can be utilized to connect the correction results of multiple local areas to obtain a relatively accurate detection result.

[0146] Although there is less information about the center line of the lane in the distance, according to the principle of camera imaging, there are relatively more vehicles farther away. If the 3D object detection framework can be used to detect distant vehicles and use them to correct the distant lane lines, the problem of insufficient information in the distance can be solved.

[0147] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A 3D lane line prediction method, characterized in that: include: Based on the current image acquired by the vehicle, extracting the vehicle coordinates and lane centerline coordinates of the vehicle contained in the current image; Based on a preset rule, the vehicle coordinates and the lane centerline coordinates are divided, and the vehicle coordinates of all vehicles included in the image are added to the corresponding lane centerline coordinate set to generate a lane centerline fitting coordinate set; The method of dividing the vehicle coordinates and the lane centerline coordinates based on a preset rule, adding the vehicle coordinates of all vehicles contained in the image to a corresponding lane centerline coordinate set, and generating a lane centerline fitting coordinate set includes: determining a relative position relationship between the vehicle contained in the current image and the lane centerline, determining a belonging relationship between the vehicle and the lane centerline according to the relative position relationship; and dividing the vehicle coordinates and the corresponding lane centerline coordinates into a lane centerline fitting coordinate set according to the belonging relationship; Based on the lane centerline fitting coordinate set, a 3D lane line is fitted and generated.

2. The method according to claim 1, characterized in that The determining of the relative position relationship between the vehicle and the lane centerline contained in the current image, and determining the relationship between the vehicle and the lane centerline according to the relative position relationship, comprises: Determine the shortest distance between the center point of the vehicle and the center line of each lane, and the angle between the vehicle's driving direction and the center line of each lane; The relationship between the vehicle and the lane centerline is determined based on whether the closest distance and / or the angle meets a preset condition.

3. The method according to claim 1, characterized in that The dividing the vehicle coordinates and the lane centerline coordinates based on a preset rule to generate a lane centerline fitting coordinate set also includes: Determining a driving state of a vehicle contained in the current image; Filter out valid vehicle coordinates from the vehicle coordinates according to the driving state; According to the belonging relationship, the valid vehicle coordinates and the corresponding lane centerline coordinates are divided into a lane centerline fitting coordinate set.

4. The method according to claim 3, characterized in that The determining the driving state of the vehicle contained in the current image includes: Constructing status information of the vehicle; Comparing the state information of the vehicle with the state cluster center of the vehicle in a preset driving state; The driving state of the vehicle is determined according to the comparison result.

5. The method according to claim 4, characterized in that The state information includes: the angle between the center line of the vehicle and the center line of the lane to which it belongs, and the relative distance between at least two coordinate points of the vehicle and the lane line; When comparing the state information of the vehicle with the state cluster center of the vehicle in a preset driving state, the relative distance is given a first priority and the angle is given a second priority.

6. The method according to any one of claims 1 to 5, characterized in that: The vehicle coordinates and lane centerline coordinates include: three-dimensional coordinates of coordinate points and corresponding auxiliary correction information, wherein the auxiliary correction information represents the degree of deviation between the coordinate points predicted by the model and the actual coordinate points.

7. The method according to claim 6, characterized in that The auxiliary correction information formula is as follows: Among them, f(x) represents the auxiliary correction information of the current coordinate point, the value is in the range of [0,1], x represents the Euclidean distance between the coordinate point predicted by the model and the actual coordinate point, and c represents the preset threshold.

8. The method according to claim 6, characterized in that When the vehicle coordinates and the lane centerline coordinates are divided, when it is determined that the vehicle belongs to multiple lane centerlines at the same time, the three-dimensional coordinates in the vehicle coordinates are simultaneously divided into lane centerline coordinate sets corresponding to the corresponding multiple lane centerlines, and the auxiliary correction information in the vehicle coordinates is evenly distributed to the lane centerline coordinate sets corresponding to the corresponding multiple lane centerlines.

9. The method according to claim 8, characterized in that The step of fitting and generating a 3D lane line based on the lane centerline fitting coordinate set includes: Extracting the horizontal coordinate, the vertical coordinate and the auxiliary correction information of each coordinate point in the lane centerline fitting coordinate set, and constructing a lane centerline coordinate point set to be generated; The lane centerline coordinate point set to be generated is used to fit and generate the 3D lane line based on the latitude coordinates following a normal distribution.

Images