3D lane line detection method and system and electronic equipment

Abstract

The invention provides a 3D lane line detection method, a 3D lane line detection system and electronic equipment, and the method comprises the steps: setting a 3D anchor line according to collection device information, standard lane information and a marked data set in combination with spatial information; acquiring an image acquired by the acquisition device, extracting 2D image features from the image, and generating a first feature image; projecting the 3D anchor line to the first feature image according to parameter information of the acquisition device to generate a second feature image; anchor line features are extracted from the second feature image, and a third feature image is generated; and 3D lane line information is calculated according to the third feature image. Through the method, the 3D lane line information can be predicted more accurately, and the calculation amount is reduced, so that the lane line detection efficiency and quality are improved.

Description

Technical Field

[0001] The present application relates to the field of computer vision, and in particular to a 3D lane line detection method, a 3D lane line detection system and an electronic device. Background Art

[0002] Existing 3D lane line detection methods usually convert the front view image or features into features of the bird's-eye view space through inverse perspective transformation, and detect lane lines from the bird's-eye view features. On the one hand, the inverse perspective transformation needs to assume that the ground is flat. When encountering a ground with height differences, it will lead to inaccurate transformation, resulting in inaccurate 3D lane line detection. If the 3D lane line is predicted directly from the front view image feature representation, there will be a lack of structured representation of the 3D lane line; on the other hand, the anchor line design used in existing lane line detection is usually based on equidistant sampling of 2D images, which occupies a large amount of computing power, and the anchor line cannot completely cover all lane lines, resulting in low detection accuracy. Summary of the invention

[0003] In view of this, the present application proposes a 3D lane line detection method, a 3D lane line detection system and an electronic device to solve the problems reflected in the above-mentioned background technology.

[0004] The technical solution of one aspect of the present application provides a 3D lane line detection method, comprising the following steps:

[0005] According to the acquisition device information, the standard lane information and the annotated data set, the 3D anchor line is set in combination with the spatial information; wherein the acquisition device information at least includes parameter information of the acquisition device;

[0006] Acquire an image acquired by an acquisition device, extract 2D image features from the image, and generate a first feature image;

[0007] Projecting the 3D anchor line onto the first feature image according to the parameter information of the acquisition device to generate a second feature image;

[0008] Extracting anchor line features from the second feature image to generate a third feature image;

[0009] The 3D lane line information is calculated based on the third feature image.

[0010] Furthermore, the collection device information includes at least: the field of view of the collection device and the height of the collection device;

[0011] According to the acquisition device information, standard lane information and annotated data set, the 3D anchor line is set in combination with the spatial information, including:

[0012] Construct the vehicle coordinate system;

[0013] According to the standard lane information, the sampling points of the initial x-axis of the anchor line are set in the ego-vehicle coordinate system to form an array of sampling points of the initial x-axis of the anchor line;

[0014] According to the field of view of the acquisition device, the sampling points of the initial y-axis of the anchor line are set in the vehicle coordinate system to form an array of sampling points of the initial y-axis of the anchor line;

[0015] According to the height of the acquisition device, the sampling point of the initial z-axis of the anchor line is set in the vehicle coordinate system;

[0016] According to the spatial information, the 3D anchor line is generated by combining the sampling point arrays of the initial x and y axes of the anchor line and the sampling points of the initial z axis of the anchor line.

[0017] Furthermore, according to the standard lane information, after setting the sampling points of the initial x-axis of the anchor line in the vehicle coordinate system and forming the sampling point array of the initial x-axis of the anchor line, it also includes:

[0018] Statistically calculate the coordinate distribution of the lane line x-axis in the data set;

[0019] According to the coordinate distribution of the lane line x-axis, the sampling points in the sampling point array of the initial x-axis of the anchor line are optimized to form an optimized sampling point array of the initial x-axis of the anchor line;

[0020] The optimized sampling point array of the initial x-axis of the anchor line is used to generate the 3D anchor line.

[0021] Furthermore, the spatial information includes at least yaw angle information and pitch angle information;

[0022] According to the spatial information, the 3D anchor line is generated by combining the sampling point arrays of the initial x and y axes of the anchor line and the sampling points of the initial z axis of the anchor line, including:

[0023] Statistically calculate the offset distribution of the lane lines on the x-axis and z-axis in the data set to obtain the yaw angle information and pitch angle information;

[0024] Calculate the yaw angle range according to the yaw angle information and construct a yaw angle array;

[0025] Calculate the pitch angle range according to the pitch angle information and construct a pitch angle array;

[0026] Generate a 3D anchor line according to the anchor line's initial x- and y-axis sampling point arrays, the anchor line's initial z-axis sampling point arrays, the yaw angle array, and the pitch angle array.

[0027] Furthermore, the 3D lane line information is calculated based on the third feature image, including:

[0028] Input the third feature image;

[0029] Predicting 3D lane line information from the third feature image using the output head of the neural network model to obtain first 3D lane line information;

[0030] Based on the first 3D lane line information, a neural network model is trained using a loss function;

[0031] Use the trained optimal model to detect and output the second 3D lane line information.

[0032] Furthermore, the neural network model is trained using a loss function, including:

[0033] Constructing a classification loss function, where the classification loss function is used to calculate the loss of the classification part in the first 3D lane line information;

[0034] Construct a regression loss function, which is used to calculate the loss of the regression part in the first 3D lane line information;

[0035] Construct a total loss function, which is used to calculate the weighted sum of the classification loss function and the regression loss function;

[0036] By minimizing the classification loss function and the regression loss function, the total loss function is minimized, and the neural network model is trained according to the total loss function. The neural network model is used to perform all steps from extracting 2D image features to calculating 3D lane line information;

[0037] When the total loss function meets the set conditions, the training ends and the optimal model is saved.

[0038] Furthermore, the process of constructing the classification loss function includes:

[0039] Obtain the category prediction probability of the anchor line from the first 3D lane line information;

[0040] Compare the predicted probability of each possible category of each anchor line with the category of the annotated label and calculate the category loss;

[0041] Use hyperparameters to balance the loss contribution of each class;

[0042] Based on the balanced loss, a classification loss function is constructed.

[0043] Furthermore, the process of constructing the regression loss function includes:

[0044] Obtain predicted coordinates and predicted visible attributes of the anchor line sampling point from the first 3D lane line information;

[0045] Compare the predicted coordinates of each predicted visible sampling point in each anchor line with the coordinates of the annotated label and calculate the coordinate loss;

[0046] Compare the predicted visible attributes of each sampling point in each anchor line with the visible attributes of the annotation label and calculate the visible loss;

[0047] Based on coordinate loss and visible loss, a regression loss function is constructed.

[0048] Another technical solution of the present application provides a 3D lane line detection system, including:

[0049] Anchor line setting module, used to set 3D anchor line according to acquisition device information, standard lane information and annotated data set combined with spatial information;

[0050] An image feature extraction module is used to obtain the image collected by the collection device, extract 2D image features from the image, and generate a first feature image;

[0051] An anchor line projection module, used to project the 3D anchor line onto the first feature image according to the parameter information of the acquisition device to generate a second feature image;

[0052] An anchor line feature extraction module, used to extract anchor line features from the second feature image to generate a third feature image;

[0053] The lane line information calculation module is used to calculate the 3D lane line information according to the third feature image.

[0054] A technical solution according to another aspect of the present application provides an electronic device, comprising:

[0055] processor;

[0056] a memory for storing processor-executable instructions;

[0057] Wherein, the processor is configured to implement the above-mentioned 3D lane line detection method when executing executable instructions.

[0058] Beneficial effects of this application:

[0059] By setting 3D anchor lines in combination with spatial information, this application can take into account the conditions of lane turns and uphill and downhill conditions, better cover the real lane lines, and improve the detection rate; the 3D anchor lines are projected into the 2D feature image and sampled, which can reduce the dimension of the feature mapping while retaining the key information of the image, reduce the amount of calculation and memory consumption; using the loss function to train the model and detect 3D lane line information can further improve the detection quality. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] Figure 1 A schematic diagram of a flow chart of a 3D lane line detection method according to an embodiment of the present application;

[0061] Figure 2A schematic diagram of the flow of an anchor line feature sampling process according to an embodiment of the present application;

[0062] Figure 3 A schematic diagram of a process of setting a 3D anchor line according to an embodiment of the present application;

[0063] Figure 4 A schematic diagram of a vehicle coordinate system according to an embodiment of the present application;

[0064] Figure 5 A schematic diagram of x-axis sampling points according to an embodiment of the present application;

[0065] Figure 6 A schematic diagram of the vis visible property of an embodiment of the present application;

[0066] Figure 7 A schematic diagram of a process of calculating 3D lane line information from a third feature image according to an embodiment of the present application;

[0067] Figure 8 This is a schematic diagram of lane line detection results in the front view of an embodiment of the present application;

[0068] Fig. 9 This is a schematic diagram of lane line detection results in 3D space according to an embodiment of the present application;

[0069] Fig.10 A system block diagram of a 3D lane detection system according to an embodiment of the present application. DETAILED DESCRIPTION

[0070] Various exemplary embodiments, features and aspects of the present application will be described in detail below with reference to the accompanying drawings. The same reference numerals in the accompanying drawings represent elements with the same or similar functions. Although various aspects of the embodiments are shown in the accompanying drawings, the drawings are not necessarily drawn to scale unless otherwise specified.

[0071] Among them, it should be understood that the terms "center", "longitudinal", "lateral", "length", "width", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside", "clockwise", "counterclockwise", "axial", "radial", "circumferential" and the like indicate orientations or positional relationships based on the orientations or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present application or simplifying the description, and do not indicate or imply that the referred device or element must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be understood as a limitation on the present application.

[0072] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of this application, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.

[0073] The word “exemplary” is used exclusively herein to mean “serving as an example, example, or illustration.” Any embodiment described herein as “exemplary” is not necessarily to be construed as preferred or advantageous over other embodiments.

[0074] In addition, in order to better illustrate the present application, numerous specific details are given in the following specific embodiments. It should be understood by those skilled in the art that the present application can also be implemented without certain specific details. In some examples, methods, means, components and circuits well known to those skilled in the art are not described in detail in order to highlight the subject matter of the present application.

[0075] The solution of the present application is applicable to the process of vehicle driving. It relies on 3D lane line detection technology to process the images collected by the acquisition device to obtain 3D lane line information, guide the vehicle to drive in the correct lane, and provide a reliable basis for the automatic cruising, lane keeping, lane changing and other functions of the autonomous driving car.

[0076] Before using the embodiments of this application, the following preparations need to be made:

[0077] Install a collection device, which can be a camera or video camera installed on the front of the vehicle as a vehicle-mounted front-view camera for image collection, or installed on a mobile device such as a drone for image collection; it can also be a sensing device such as a laser radar, installed on the vehicle body or other mobile devices for information collection. The embodiment of the present application mainly uses a vehicle-mounted front-view camera as a collection device to perform 3D lane line detection.

[0078] Collecting and annotating a data set to generate an annotated data set includes the following steps:

[0079] Step D1, data collection: Use vehicles equipped with perception devices such as cameras and lidar to drive in various road environments (such as cities, suburbs, highways, etc.) to collect a large amount of road data, which includes the actual conditions of the road, the location of lane lines, traffic signs, obstacles, etc.

[0080] Step D2, data preprocessing: cleaning and preprocessing the collected raw data, including but not limited to steps such as denoising, filtering, and registration of point cloud data, so as to remove noise and improve data quality.

[0081] Step D3, lane marking: accurately mark the lanes in the preprocessed data. The labeler needs to locate and describe the shape of the lanes in three-dimensional space based on the point cloud information of the road conditions, and accurately mark the location, attributes, direction and other information of the lanes.

[0082] Step D4, generate a data set: After verification and review, the reviewed annotation data is integrated into a unified data format to form the final 3D lane line data set. This data set can contain multiple types of data, such as RGB images, 3D point cloud annotation results, etc., and needs to be stored and organized in a certain format to facilitate subsequent model training and algorithm evaluation.

[0083] Example 1

[0084] Figure 1 The flowchart of the 3D lane line detection method according to an embodiment of the present application is shown. This method can reduce the amount of calculation, better predict the 3D lane line information, and improve the accuracy of lane line detection. Figure 1 , the flow chart includes:

[0085] Step S100, setting a 3D anchor line according to the acquisition device information, the standard lane information and the annotated data set in combination with the spatial information.

[0086] Specifically, the 3D anchor line is a series of predefined 3D spatial points in the ego-vehicle coordinate system, which are used to represent all possible lane line positions. The setting of the 3D anchor line takes into account the lane line shape and possible range of variation of the actual road, providing a basis for subsequent projection and feature extraction, as well as accurate lane line detection.

[0087] Among them, the collection device information includes the parameter information of the collection device; the standard lane information includes lane width, lane line width and other information, and the national road-related standards and specifications can be referred to; the annotated data set can be obtained through the aforementioned preparation work; the spatial information is the change information of the lane line in space, including yaw angle information and pitch angle information.

[0088] Optionally, the collection device information also includes the field of view of the collection device and the height of the collection device.

[0089] Step S200: acquiring an image acquired by an acquisition device, extracting 2D image features from the image, and generating a first feature image.

[0090] Specifically, an image of a vehicle-mounted front-view camera is obtained, and the image is an RGB image; 2D image features are extracted from the image using an existing backbone network model; and a first feature image is generated based on the 2D image features to facilitate subsequent feature recognition and processing.

[0091] Optionally, the backbone network model can be a network such as ResNet or EfficientNet.

[0092] Step S300: projecting the 3D anchor line onto the first feature image according to the parameter information of the acquisition device to generate a second feature image.

[0093] Specifically, the internal and external parameter information of the vehicle-mounted forward-looking camera is obtained; according to the parameter information, the 3D anchor line is projected onto the first feature image through a projection algorithm based on the pinhole camera model to generate a second feature image, thereby realizing the dimensionality reduction processing of the anchor line, which is convenient for the subsequent processing of the anchor line features in the 2D image features.

[0094] Among them, the camera's internal and external parameters describe the physical properties of the camera and its position and orientation in space: the internal parameters include focal length, principal point, etc., which are used to describe the internal optical characteristics of the camera; while the external parameters provide the relative relationship between the camera and the world coordinate system, such as rotation matrix and translation matrix.

[0095] Optionally, the point (X w ,Y w ,Z w ) is transformed to the 2D image pixel point (u, v) through projection. It is necessary to first transform the 3D lane anchor point (X w ,Y w ,Z w ) is converted to the camera coordinate system (X c ,Y c ,Z c ) and then set the camera coordinate system (X c ,Y c ,Z c ) point is converted to the point (u,v) in the image coordinate system. The formula is as follows:

[0096]

[0097] Among them, K is the camera intrinsic parameter, which is a 3*3 matrix, R is the rotation matrix, which is a 3*3 matrix, and T is the translation vector, which is a 3*1 matrix.

[0098] Step S400: extracting anchor line features from the second feature image to generate a third feature image.

[0099] Specifically, the 2D anchor line feature extraction is to sample the second feature image using the 2D lane line anchor line after the 3D lane line anchor line is projected, obtain the sampled anchor line feature, and generate the third feature image. Figure 2, assuming that the features of the second feature image are [16, 64, 45, 60], and the 2D anchor line projected by the 3D anchor line in the front view space is [16, 3165, 10, 2], the sampling process is as follows:

[0100] Step S410, taking the second feature image as the input feature image, the features of the input feature image are [16, 64, 45, 60], where 16 is the batch size, 64 is the number of feature channels, and the feature size is 45*60.

[0101] Step S420, set the sampling grid. The sampling network is a four-dimensional tensor [16, 3165, 10, 2], where 3165 means that a total of 3165 points are sampled along the height direction. Each point corresponds to a 10-dimensional feature vector, which is used to represent the local features near the point. In addition, each point also contains two coordinate dimensions, u and v.

[0102] Step S430, grid generation, based on the sampling grid, generates a series of grid points on the input feature map. These grid points can be regarded as reference points of the sampling positions. Since the size of the sampling grid is [16, 3165, 10, 2], and the features of the input feature image are [16, 64, 45, 60], the input feature image needs to be interpolated.

[0103] Step S440, interpolation operation, for each sampling point, obtain its corresponding pixel value from the input feature map. This step is usually completed through an interpolation operation, such as bilinear interpolation or bicubic interpolation. Bilinear interpolation is used here, that is, according to the floating-point coordinates corresponding to each point in the grid, four adjacent integer coordinate points are found in the input feature image, and the pixel values ​​of the corresponding points in the output feature map are calculated through bilinear interpolation based on the pixel values ​​of these four points and the relative position relationship between the floating-point coordinates and them.

[0104] Step S450, output generation, after sampling and interpolation, we get a new feature map, this new feature map will be a [16,3165,2] feature map, because 3165 points are selected on the second feature image for sampling, each point represents the coordinates u and v of a pixel value, and the third feature image is obtained.

[0105] Through the above steps, the dimension of the feature map is further reduced, the amount of calculation and memory consumption are reduced, while retaining the key information in the image features.

[0106] Step S500, calculating and obtaining 3D lane line information according to the third feature image.

[0107] Specifically, an output head is used to predict 3D lane line information, and the output head includes a classification branch and a regression branch, wherein the classification branch is used to classify the input features and determine the lane line category; the regression branch is used to regress the input features and predict the lane line coordinates; then based on the first 3D lane line information, a loss function is used to measure the difference between the model prediction result and the actual result, and by minimizing the loss function, steps S200-S500 are repeated to perform model training until the loss function converges to the set value, ending the model training and saving the optimal model; the optimal model is used to detect and output the 3D lane line information, thereby further improving the detection quality.

[0108] Example 2

[0109] In this embodiment, an alternative solution for setting the 3D anchor line in step S100 is provided, specifically a strategy for designing 3D anchor lines in different ranges, which can better cover the real lane lines and improve the detection rate by 3.7 points. Figure 3 ,include:

[0110] Step S110, constructing the vehicle coordinate system.

[0111] Specifically, see Figure 4 , with the center point of the vehicle as the origin of the coordinate system; the right direction of the vehicle is the positive direction of the x-axis; the forward direction of the vehicle is the positive direction of the y-axis; and the upward direction of the vehicle is the positive direction of the z-axis.

[0112] Step S120, according to the standard lane information, the sampling points of the initial x-axis of the anchor line are set in the vehicle coordinate system to form an array of sampling points of the initial x-axis of the anchor line.

[0113] Specifically, according to the "Technical Standards for Highway Engineering" and "Design Specifications for Urban Road Engineering", we can know that: the width of each lane on urban roads is 3.5m, the lane line width is 17cm, the width of each diversion lane at the intersection is 2.3m-2.5m, the width of each lane on the trunk road is 3.75m, the shoulder is 1.5m-2.5m, the width of each lane at the highway toll station is 2.5m, and the extra-wide toll lane is over 3.5m.

[0114] See also Figure 5 On the road where the vehicle is traveling, assuming that the width of a single lane is w1 = 3.5 m, and the width of the car is w c =1.8m.

[0115] When the vehicle is driving close to the right lane line, the distance between the center point of the vehicle and the nearest lane line on the right, right 1, is:

[0116]

[0117] The distance from the nearest lane line on the left is:

[0118]

[0119] At this time, if the width of the lane line is added, d1 = 17 cm = 0.17 m, the distance range on the x-axis from the center point of the vehicle to the nearest lane line on the left and right is:

[0120]

[0121] Similarly, the distance range on the x-axis from the center point of the vehicle to the right 2 lane line is:

[0122]

[0123] The distance range on the x-axis from the center point of the vehicle to the left 2 lane line is:

[0124]

[0125] Then the distance range on the x-axis of the two lane lines right 2 and left 2 that are closest to the center point of the vehicle is:

[0126] Range2=±[d 2_min ,d 2_max ]

[0127] Similarly, the x-axis distance range of the lane lines 3 to the right and 3 to the left of the vehicle is calculated as:

[0128]

[0129] Then the distance range on the x-axis of the two lane lines closest to the center point of the vehicle, right 3 and left 3, is:

[0130] Range3=±[d 3_min ,d 3_max ]

[0131] Similarly, the x-axis distance range of the 4 lanes on the right and 4 lanes on the left of the vehicle is calculated as:

[0132]

[0133] Then the distance range on the x-axis of the two lane lines right 4 and left 4 that are closest to the center point of the vehicle is:

[0134] Range4=±[d 4_min ,d 4_max ]

[0135] The number of lanes in the same direction is at least 1 lane and at most 4 lanes, so the above 4 ranges can basically be covered.

[0136] To set the initial x-axis value of the anchor line, k1 points can be sampled at equal intervals in the above four ranges, so there are 4*k1 points.

[0137] In addition, considering that the vehicle may change lanes by crossing the dashed line, the lane line may have a value of ±w relative to the x-axis direction of the vehicle. c / 2, can be between w c / 2, and add the zero point, so there are 2*k2+1 points. If k2=3, there are 7 points.

[0138] The above sampling points form an x_start_init_sample array, which is the sampling point array of the initial x-axis of the anchor line. The number and value of the sampling points of the initial x-axis of the anchor line can be dynamically adjusted according to the actual label distribution in the data set.

[0139] Optionally, after setting the sampling points of the initial x-axis of the anchor line in the vehicle coordinate system according to the standard lane information to form the sampling point array of the initial x-axis of the anchor line, the method further includes:

[0140] Step A: Statistically calculate the coordinate distribution of the x-axis of the lane line in the data set.

[0141] Specifically, the steps include:

[0142] Step A1, collecting the three-dimensional coordinate information of each lane line in each lane line label annotated in the statistical data set.

[0143] Step A2: for a lane line in one of the lane line labels, count the maximum and minimum values ​​of the x-coordinate, recorded as max and min.

[0144] Step A3: for one of the lane line labels, count the maximum and minimum x-coordinates of all lane lines according to step A2 and store them in the max_min sequence.

[0145] Step A4: In order to remove noise points or incorrectly marked lane lines, the values ​​in the max_min sequence whose absolute value of the difference between the max and min values ​​is greater than 2 times the lane width are removed to obtain a filtered max_min_filter sequence.

[0146] Step A5, select the next lane line label, repeat steps A2 and A3, if there is a new maximum value or minimum value of x that is repeated in the sequence max_min, filter out the latest maximum value or minimum value and do not proceed to the subsequent steps; if there is no repetition, proceed to step A4, add the max_min_filter sequence obtained from the previous lane line label, and obtain the sequence max_min_filter_merge after integrating the two lane line labels.

[0147] Step A6, traverse all image labels in the data set, repeat the above steps A1-A5, obtain the max_min_filter_merge sequence after all image labels are integrated, and obtain the maximum and minimum value sequence max_min_x of the x-coordinate of the three-dimensional lane line of the data set according to the max_min_filter_merge sequence.

[0148] Step B: Optimize the sampling points in the sampling point array of the initial x-axis of the anchor line according to the coordinate distribution of the x-axis of the lane line to form an optimized sampling point array of the initial x-axis of the anchor line.

[0149] Specifically, the steps include:

[0150] Step B1, according to the above steps, the maximum and minimum value sequence of x-coordinates max_min_x is counted, and it is determined whether the maximum and minimum values ​​of all x-coordinates in the sequence are within the above four ranges. If there is a point x_peak that is not within the range, the frequency of occurrence of x_peak is counted. If the frequency of occurrence is more than 10% among all lane line points, the range closest to this x_peak extreme value is expanded to include this x_peak extreme value, and the sampling points of the expanded range are appropriately increased according to the distance difference.

[0151] Step B2, optimizing the x_start_init_sample array to obtain an optimized anchor line initial x-axis sampling point x_start array.

[0152] Through the above steps, the sampling points of the initial x-axis of the anchor line are optimized, making the anchor line setting closer to the actual situation.

[0153] Step S130, according to the field of view of the acquisition device, the sampling points of the initial y-axis of the anchor line are set in the vehicle coordinate system to form an array of sampling points of the initial y-axis of the anchor line.

[0154] Specifically, starting from the origin of the vehicle coordinate system, the farthest range of the imaging field of view of the vehicle-mounted forward-view camera, end_y, is detected, which is about 100m or 150m in front. Considering that the forward-view camera of the vehicle has a blind spot in the near visible range and cannot really capture the image of 0m, the blind spot of the forward-view camera of the vehicle is calculated here, which is 2 meters in front of the vehicle coordinate system. It will be slightly different according to different camera parameters and camera installation positions.

[0155] The starting point of the initial y-axis sampling point of the anchor line is designed as M, the end point is end_y, and 20 points are sampled at equal intervals to form an array y_sample, which is the sampling point array of the initial y-axis of the anchor line. Among them, the y-axis sampling interval step is:

[0156]

[0157] Step S140, setting the initial z-axis sampling point of the anchor line in the vehicle coordinate system according to the height of the acquisition device.

[0158] Specifically, according to the installation heights of multiple collection devices of the sampling samples in the data set, it is determined that the average installation height of the collection devices in the data set is h meters.

[0159] Set the initial z-axis sampling point of the anchor line to:

[0160] z_start=-h

[0161] Step S150: Generate a 3D anchor line based on the spatial information by combining the sampling point arrays of the initial x and y axes of the anchor line and the sampling points of the initial z axis of the anchor line.

[0162] The spatial information at least includes yaw angle information and pitch angle information.

[0163] When the lane line has a certain curvature and this curvature forms an angle with the positive direction of the y-axis in the x, y plane, this angle is usually called the curve yaw angle, or yaw angle for short. This angle describes the curvature of the lane line in the horizontal plane, which is manifested as the change in the x-axis of the lane line. The yaw angle information can be obtained by calculating the offset distribution of the x-axis of the lane line.

[0164] When a point on the lane line has a certain height, and this height forms an angle with the positive direction of the y-axis in the y, z plane, this angle is usually called the pitch angle. This angle describes the angle between the lane line or the road surface and the horizontal plane, and is expressed as a change in the z-axis of the lane line. The pitch angle information can be obtained by calculating the offset distribution of the z-axis of the lane line.

[0165] Specifically, the steps include:

[0166] Step S151, statistically calculate the offset distribution of the lane lines in the data set on the x-axis and z-axis to obtain yaw angle information and pitch angle information.

[0167] Specifically, the steps include:

[0168] Step a: Calculate the three-dimensional coordinate information of each lane line in each lane line label annotated in the statistical data set.

[0169] Step b: for a lane line in one of the lane line labels, count the maximum and minimum values ​​of the x-coordinate, and the absolute value of the maximum value minus the minimum value is the range of the x-coordinate, that is, range_x = |maximum value - minimum value|.

[0170] Step c: for one of the lane line labels, count the maximum value, minimum value, and range of x-coordinates of all lane lines according to step b, and record them as range_x sequence.

[0171] Step d: In order to remove noise points or incorrectly marked lane lines, remove the values ​​greater than (2*lane width) in the range_x sequence to obtain the filtered range_x_filter sequence and max_min_filter sequence.

[0172] Step e: Select the maximum value of the filtered sequence range_x_filter, which is x_shift_max.

[0173] Step f: Select the next lane line label and repeat the above steps to obtain the filtered sorted sequence range_x_filter_i.

[0174] Step g, compare the maximum value of the sequence range_x_filter_i with x_shift_max. If the maximum value of the sequence range_x_filter_i is greater than x_shift_max, update x_shift_max to be equal to the maximum value of the sequence range_x_filter_i. Use the final result as the offset distribution of the x-axis to obtain the yaw angle information.

[0175] Step h, repeat the above steps, replace the x coordinate with the z coordinate, calculate z_shift_max as the offset distribution of the z axis, and obtain the pitch angle information.

[0176] Step S152, calculating the yaw angle range according to the yaw angle information, and constructing a yaw angle array.

[0177] Specifically, considering that the lane lines on the actual driving road may turn and curve, the x value of the anchor line is adjusted by introducing a yaw angle that varies within a certain range. According to the yaw angle information x_shift_max calculated above, the yaw angle range of the 3D lane line is determined, and the maximum yaw angle is calculated as follows:

[0178]

[0179] The yaw angle is sampled at equal intervals between [0, yaw_max] to form the yaw angle array yaw_list.

[0180] Step S153, calculating the pitch angle range according to the pitch angle information, and constructing a pitch angle array.

[0181] Specifically, considering the ups and downs in the actual driving road, the pitch angle is introduced to adjust the z value of the anchor line. According to the pitch angle information z_shift_max calculated above, the pitch angle range of the 3D lane line is determined. The maximum pitch angle is calculated as follows:

[0182]

[0183] The pitch angle is sampled at k_pitch points at equal intervals between [0, pitch_max], forming the pitch angle array pitch_list.

[0184] Step S154, generating a 3D anchor line according to the anchor line initial x-axis and y-axis sampling point arrays, the anchor line initial z-axis sampling point arrays and the yaw angle and pitch angle arrays.

[0185] Specifically, see Figure 6 Considering that some of the calculated anchor lines may exceed the x-axis sampling range, a vis attribute is added. When the x-coordinate value of the generated anchor line is within the x-axis sampling range [x_min, x_max], the visible attribute of this point on the anchor line is True; when it exceeds this range, the point on the anchor line is False.

[0186] Among them, x_min is the minimum value of the sampling point array x_start of the initial x-axis of the optimized anchor line, and x_max is the maximum value of the sampling point array x_start of the initial x-axis of the optimized anchor line.

[0187] According to the above statistics of the maximum and minimum extreme values ​​of x, the minimum value x_min and the maximum value x_max of the x-axis range are obtained; the 3D anchor line is generated according to the optimized anchor line initial x-axis sampling point array x_start, anchor line initial y-axis sampling point array y_sample, anchor line initial z-axis sampling point z_start, yaw angle array yaw_list, and pitch angle array pitch_list.

[0188] For an initial point x_start, a yaw angle pitch, and a pitch angle yaw, the length of each anchor line, that is, the number of points contained in each anchor line is len_anchor, which is equal to the number of sampling points on the y-axis, which is 20 here.

[0189] Anchor line x coordinate array = x_start + y_sample array * tan(yaw).

[0190] Anchor line z coordinate array = z_start + y_sample array * tan(pitch).

[0191] Whether the anchor point is visible vis array = {True|x∈[x_min,x_max]}.

[0192]

[0193] Traverse each value in the sampling point array x_start of the initial x-axis of the optimized anchor line, traverse each value in the yaw angle array pitch_list, traverse each value in the pitch angle array yaw_list, and calculate multiple lane anchor lines through the above steps.

[0194] Finally, the number of 3D anchor lines is: x_start array length * pitch_list array length * pitch angle array yaw_list. The number of points in each anchor line is equal to the number of y-axis sampling points, which is 20 here.

[0195] Through the above steps, spatial information is introduced and 3D anchor lines are generated, thereby optimizing the detection problems caused by a series of non-straight roads such as bends, uphill and downhill roads, making the anchor line setting closer to the actual situation.

[0196] Example 3

[0197] In this embodiment, an alternative solution for calculating the 3D lane line according to the third feature image in step S500 is provided. Figure 7 ,include:

[0198] Step S510: input a third feature image.

[0199] Specifically, according to the above steps S200-S400, the third feature image is calculated using the neural network model, and the third feature image is input for subsequent 3D lane line detection.

[0200] Step S520: Use the output head of the neural network model to predict 3D lane line information from the third feature image to obtain first 3D lane line information.

[0201] Specifically, the output head contains multiple branches for processing different tasks. One branch is the classification branch, which classifies the input features and determines which type of lane line they belong to, including solid white line, dashed white line, dashed yellow line, guide line, etc. The other branch is the point regression branch, which regresses the input features and predicts the x, z coordinates and vis visible attributes of the lane line.

[0202] Among them, in the classification branch, the classification head is used to generate classification probability. The classification head is a convolutional neural network layer, which converts the input features into the probability distribution of each category. By comparing these probability values, the lane line category to which each feature point belongs can be determined.

[0203] In the regression branch, a regression head is used to predict the 3D offset of the anchor line. The regression head is also a convolutional neural network layer, which learns the mapping relationship from input features to target coordinates. Through the output of the regression head, the precise position of the lane line in 3D space (x-axis and z-axis offset information) and whether each position is visible can be obtained.

[0204] Through the classification branch and the regression branch, the first 3D lane line information can be predicted, which is convenient for subsequent model training and result output based on the predicted information.

[0205] Step S530: Based on the first 3D lane line information, a loss function is used to train a neural network model.

[0206] Specifically, the loss function is used to measure the difference between the model prediction results and the actual labels. During the training process, the loss function is minimized based on the image label data in the sample image database and the predicted lane line category, x-axis and z-axis offset information, and whether each position is visible, and the network is continuously back-propagated to gradually approach the actual situation.

[0207] The following steps are involved:

[0208] Step S531, constructing a classification loss function, where the classification loss function is used to calculate the loss of the classification part in the first 3D lane line information.

[0209] Specifically, the category prediction probability of the anchor line is obtained from the first 3D lane line information; the predicted probability of each possible category of each anchor line is compared with the category of the annotation label, and the category loss is calculated; the loss contribution of each category is balanced using hyperparameters; and a classification loss function is constructed based on the balanced loss.

[0210] The classification loss function is:

[0211]

[0212] in,

[0213] α c =α*y l +(1-α)*(1-y l )

[0214] q lm =p m *y l +(1-p m )*(1-y l )

[0215] L represents the number of lane line types;

[0216] M represents the number of 3D anchor lines;

[0217] q lm is the probability that the model predicts that the anchor line m belongs to the true category l;

[0218] p m is the probability of anchor line m predicted by the model;

[0219] y l When the true category l is the true category 1, y l =1, not category 1, y l =0;

[0220] α c and r represent hyperparameters, which are used to balance the loss contribution of each lane line category and adjust the model's attention to lane lines of different difficulty levels. c is a parameter used to control the balance between positive and negative lane lines. c The value of can reduce the weight of easy-to-classify lane lines or increase the weight of difficult-to-classify lane lines, thereby balancing the contribution of positive and negative lane lines in the loss function. r is a parameter used to adjust the model's attention to difficult-to-classify lane lines. By adjusting the value of r, you can control which lane line samples the model pays more attention to during training. Here α c =0.5, r=2.

[0221] Step S532: construct a regression loss function, where the regression loss function is used to calculate the loss of the regression part in the first 3D lane line information.

[0222] Specifically, the predicted coordinates and predicted visible attributes of the anchor line sampling points are obtained from the first 3D lane line information; the predicted coordinates of each predicted visible sampling point in each anchor line are compared with the coordinates of the annotation label, and the coordinate loss is calculated; the predicted visible attribute of each sampling point in each anchor line is compared with the visible attribute of the annotation label, and the visible loss is calculated; based on the coordinate loss and the visible loss, a regression loss function is constructed.

[0223] The regression loss function is:

[0224]

[0225] Among them, M p Represents the total number of 3D anchor lines, i represents one of the anchor lines, N represents the number of points contained in each anchor line, and j represents one of the points. Indicates the x-coordinate value of the j-th point of the i-th anchor line in the annotated label. Indicates the z-coordinate value of the j-th point of the i-th anchor line in the annotated label. Represents the visible property of the jth point of the i-th anchor line in the annotated label, Represents the x-coordinate value of the j-th point of the corresponding i-th anchor line predicted by the model, It represents the z-coordinate value of the j-th point of the corresponding i-th anchor line predicted by the model. Represents the visible attribute of the jth point of the corresponding i-th anchor line predicted by the model.

[0226] Step S533: construct a total loss function, which is used to calculate the weighted sum of the classification loss function and the regression loss function.

[0227] Total loss function:

[0228] Loss = α cls L cls +α reg L reg

[0229] in,

[0230] α_cls=0.5,α_reg=0.5

[0231] Step S534, by minimizing the classification loss function and the regression loss function, the total loss function is minimized, and the neural network model is trained according to the total loss function. The neural network model is used to execute all steps from extracting 2D image features to calculating 3D lane line information.

[0232] Through the above steps, the loss function is defined for each branch, and the regression loss function is adjusted so that it can introduce spatial information to calculate the loss.

[0233] Step S540: Use the trained optimal model to detect and output the second 3D lane line information, which is used as the final 3D lane line information, that is, the 3D lane line detection result.

[0234] Specifically, see Figure 8 and Fig. 9 , Figure 8 is the lane line detection result on the front view, Fig. 9 is the lane line detection result in 3D space. Among them: red represents the marked lane line, and other colors are the lane lines detected by the model. Different colors represent different lane line categories. The selected colors can be set as needed, such as yellow represents the curb, sky blue represents the white solid line, and purple represents the white dotted line. The model can be used to detect and output the line type and spatial position information of the lane line in the current image, making the detection result closer to the actual situation.

[0235] Example 4

[0236] Based on the same principle as the above method, a 3D lane line detection system is also proposed, see Fig.10 , Fig.10 A 3D lane line detection system according to an embodiment of the present application is shown, comprising:

[0237] The anchor line setting module 100 is used to set the 3D anchor line according to the acquisition device information, the standard lane information and the annotated data set in combination with the spatial information.

[0238] The image feature extraction module 200 is used to obtain the image collected by the collection device, extract 2D image features from the image, and generate a first feature image.

[0239] The anchor line projection module 300 is used to project the 3D anchor line onto the first feature image according to the parameter information of the acquisition device to generate a second feature image.

[0240] The anchor line feature extraction module 400 is used to extract the anchor line feature from the second feature image to generate a third feature image.

[0241] The lane line information calculation module 500 is used to calculate the 3D lane line information according to the third feature image.

[0242] Obviously, those skilled in the art should understand that all or part of the processes in the above-mentioned embodiment method can be implemented by instructing the relevant hardware through a computer program, and the program can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned control methods. The modules or steps of the above-mentioned application can be implemented by a general-purpose computing device. They can be concentrated on a single computing device or distributed on a network composed of multiple computing devices. Optionally, they can be implemented with program codes executable by computing devices, so that they can be stored in a storage device and executed by the computing device, or they can be made into individual integrated circuit modules respectively, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. In this way, the present application is not limited to any specific combination of hardware and software.

[0243] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment method can be implemented by instructing the relevant hardware through a computer program, and the program can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned control methods. Among them, the storage medium can be a disk, an optical disk, a read-only memory (ROM), a random access memory (RAM), a flash memory (Flash Memory), a hard disk (Hard Disk Drive, abbreviated as: HDD) or a solid-state drive (SSD), etc.; the storage medium can also include a combination of the above-mentioned types of memory.

[0244] Example 5

[0245] Furthermore, an electronic device is proposed, comprising:

[0246] processor;

[0247] Memory used to store processor-executable instructions.

[0248] Among them, the processor is configured to implement the 3D lane line detection method of Examples 1-3 when executing executable instructions.

[0249] The electronic device of the embodiment of the present application includes a processor and a memory for storing processor executable instructions, wherein the processor is configured to implement any of the above 3D lane line detection methods when executing the executable instructions.

[0250] It should be noted that the number of processors may be one or more. At the same time, the electronic device in the embodiment of the present application may also include an input device and an output device. The processor, memory, input device, and output device may be connected via a bus or in other ways, which are not specifically limited here.

[0251] The memory is a computer-readable storage medium for the 3D lane line detection method, and can be used to store software programs, computer executable programs, and various modules, such as the program or module corresponding to the 3D lane line detection method of the embodiment of the present application. The processor executes various functional applications and data processing of the electronic device by running the software program or module stored in the memory.

[0252] The input device can be used to receive input numbers or signals. The signal can be a key signal related to user settings and function control of the device / terminal / server. The output device can include a display device such as a display screen.

[0253] The embodiments of the present application have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and changes will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The selection of terms used herein is intended to best explain the principles of the embodiments, practical applications, or technical improvements in the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.

Claims

1. A 3D lane line detection method, characterized in that: The following steps are involved: According to the acquisition device information, the standard lane information and the annotated data set, the 3D anchor line is set in combination with the spatial information; wherein the acquisition device information at least includes the parameter information of the acquisition device; Acquire the image acquired by the acquisition device, extract 2D image features from the image, and generate a first feature image; Projecting the 3D anchor line onto the first feature image according to the parameter information of the acquisition device to generate a second feature image; Extracting anchor line features from the second feature image to generate a third feature image; The 3D lane line information is calculated based on the third feature image.

2. The 3D lane line detection method according to claim 1, characterized in that: The acquisition device information at least includes: the field of view of the acquisition device and the height of the acquisition device; The step of setting the 3D anchor line according to the acquisition device information, the standard lane information and the annotated data set in combination with the spatial information includes: Construct the vehicle coordinate system; According to the standard lane information, sampling points of the initial x-axis of the anchor line are set in the vehicle coordinate system to form an array of sampling points of the initial x-axis of the anchor line; According to the field of view of the acquisition device, sampling points of the initial y-axis of the anchor line are set in the vehicle coordinate system to form an array of sampling points of the initial y-axis of the anchor line; According to the height of the acquisition device, setting a sampling point of the initial z-axis of the anchor line in the vehicle coordinate system; The 3D anchor line is generated according to the spatial information and in combination with the sampling point arrays of the initial x and y axes of the anchor line and the sampling points of the initial z axis of the anchor line.

3. The 3D lane line detection method according to claim 2, characterized in that: After setting the sampling points of the initial x-axis of the anchor line in the vehicle coordinate system according to the standard lane information to form an array of sampling points of the initial x-axis of the anchor line, the method further includes: Statistically calculating the coordinate distribution of the lane line x-axis in the data set; According to the coordinate distribution of the lane line x-axis, the sampling points in the sampling point array of the initial x-axis of the anchor line are optimized to form an optimized sampling point array of the initial x-axis of the anchor line; The optimized sampling point array of the initial x-axis of the anchor line is used to generate the 3D anchor line.

4. The 3D lane line detection method according to claim 2, characterized in that: The spatial information includes at least yaw angle information and pitch angle information; The generating the 3D anchor line according to the spatial information and combining the sampling point arrays of the initial x and y axes of the anchor line and the sampling point of the initial z axis of the anchor line comprises: Statistically calculating the offset distribution of the lane line x-axis and z-axis in the data set to obtain the yaw angle information and the pitch angle information; Calculate the yaw angle range according to the yaw angle information and construct a yaw angle array; Calculate the pitch angle range according to the pitch angle information and construct a pitch angle array; The 3D anchor line is generated according to the sampling point arrays of the initial x and y axes of the anchor line, the sampling point arrays of the initial z axis of the anchor line, the yaw angle array, and the pitch angle array.

5. The 3D lane detection method according to any one of claims 1 to 4, characterized in that: The calculating and obtaining 3D lane line information according to the third feature image includes: inputting the third feature image; Predicting 3D lane line information from the third feature image using an output head of the neural network model to obtain first 3D lane line information; Based on the first 3D lane line information, training the neural network model using a loss function; Use the trained optimal model to detect and output the second 3D lane line information.

6. The 3D lane line detection method according to claim 5, characterized in that: The using a loss function to train the neural network model comprises: Constructing a classification loss function, where the classification loss function is used to calculate the loss of the classification part in the first 3D lane line information; Constructing a regression loss function, where the regression loss function is used to calculate the loss of the regression part in the first 3D lane line information; Constructing a total loss function, wherein the total loss function is used to calculate the weighted sum of the classification loss function and the regression loss function; The total loss function is minimized by minimizing the classification loss function and the regression loss function, and the neural network model is trained according to the total loss function, wherein the neural network model is used to perform all steps from extracting 2D image features to calculating 3D lane line information; When the total loss function meets the set conditions, the training is terminated and the optimal model is saved.

7. The 3D lane line detection method according to claim 6, characterized in that: The process of constructing the classification loss function includes: Obtaining a category prediction probability of an anchor line from the first 3D lane line information; Compare the predicted probability of each possible category of each anchor line with the category of the annotated label and calculate the category loss; Use hyperparameters to balance the loss contribution of each class; Based on the balanced loss, the classification loss function is constructed.

8. The 3D lane line detection method according to claim 6, characterized in that: The process of constructing the regression loss function includes: Acquire predicted coordinates and predicted visible attributes of anchor line sampling points from the first 3D lane line information; Compare the predicted coordinates of each predicted visible sampling point in each anchor line with the coordinates of the annotated label and calculate the coordinate loss; Compare the predicted visible attributes of each sampling point in each anchor line with the visible attributes of the annotation label and calculate the visible loss; The regression loss function is constructed based on the coordinate loss and the visible loss.

9. A 3D lane detection system, characterized in that: include: Anchor line setting module, used to set 3D anchor line according to acquisition device information, standard lane information and annotated data set combined with spatial information; An image feature extraction module, used to obtain the image collected by the collection device, extract 2D image features from the image, and generate a first feature image; an anchor line projection module, configured to project the 3D anchor line onto the first feature image according to the parameter information of the acquisition device to generate a second feature image; An anchor line feature extraction module, used to extract anchor line features from the second feature image to generate a third feature image; A lane line information calculation module is used to calculate 3D lane line information according to the third feature image.

10. An electronic device, characterized in that: include: processor; a memory for storing processor-executable instructions; Wherein, the processor is configured to implement the 3D lane line detection method described in any one of claims 1-8 when executing the executable instructions.

Images