Large-curvature lane line monocular 3D detection method, device, equipment, medium and product

Abstract

The invention relates to a large-curvature lane line monocular 3D detection method and device, equipment, a medium and a product, and the method comprises the steps: obtaining the sampling data of a target vehicle, and determining the truth value data of a lane line where the target vehicle is located in a three-dimensional space according to the sampling data; fitting the lane lines according to the truth value data to obtain fitted lane lines and sampling point coordinates, and determining the offset of each sampling point according to a preset offset determination method; camera parameters on the target vehicle are obtained, camera parameter feature vectors are generated according to the camera parameters, and a three-dimensional lane line detection network model is constructed according to the camera parameter feature vectors and the offset of the sampling points; and iteratively training the three-dimensional lane line detection network model until a preset model convergence condition is reached, and obtaining a target lane line detection network model. By adopting the method, the 3D detection result of the large-curvature lane line can be more accurate and reliable, so that the driving safety is improved.

Description

Technical Field

[0001] The present application relates to the field of intelligent driving technology, and in particular to a method, device, equipment, medium and product for monocular 3D detection of lane lines with large curvature. Background Art

[0002] In recent years, intelligent driving technology has become a hot topic in the automotive industry, with advances in artificial intelligence providing strong support for its development. Vehicle environmental perception and traffic scene understanding are crucial components of autonomous driving. Lane perception provides data support for downstream tasks such as vehicle planning and control, guiding and assisting vehicles in normal driving. It is also a key means of ensuring vehicles stay in the correct and safe areas.

[0003] There are various monocular 3D lane marking detection schemes in the related art. The differences primarily focus on the conversion of perspective view (PV) image features into a bird's-eye view (BEV) feature space, such as those based on MLP (Multilayer Perceptron) and IPM (Inverse Perspective Mapping); or on detection head optimization, such as the use of the NMS (Non-Maximum Suppression)-free method based on the detr (Detection Transformer) architecture. However, when modeling lane markings, the equidistant resampling along the y-axis employed in related art cannot effectively represent lane markings, especially when the x-axis values vary dramatically within a unit spacing on the y-axis (i.e., lane markings with large curvature or horizontal lines). This results in these methods' limited ability to detect lane markings with large curvature, resulting in low reliability of the resulting detection results. Summary of the Invention

[0004] Based on this, it is necessary to provide a monocular 3D detection method, device, equipment, medium and product for large curvature lane lines to address the technical problem that the above-mentioned related technologies lack the ability to detect large curvature lane lines, resulting in low reliability of detection results.

[0005] In a first aspect, the present application provides a monocular 3D detection method for lane markings with large curvature, the method comprising:

[0006] Obtain sampling data of the target vehicle under different types of roads, different weather conditions, and different traffic densities, and determine the true value data of the lane line where the target vehicle is located in three-dimensional space based on the sampling data;

[0007] Fit the lane lines according to the true value data to obtain the fitted lane lines and sampling point coordinates, and determine the offset of each sampling point according to the preset offset determination method;

[0008] Obtain the camera parameters on the target vehicle, generate a camera parameter feature vector based on the camera parameters, and build a 3D lane detection network model based on the camera parameter feature vector and the offset of the sampling point;

[0009] The three-dimensional lane line detection network model is iteratively trained until the preset model convergence conditions are reached to obtain the target lane line detection network model. The target lane line detection network model is deployed on the target vehicle to obtain the lane detection data of the lane line where the target vehicle is located. The lane detection data includes the three-dimensional coordinates and curvature of the lane line.

[0010] In one embodiment, the sampling data includes front view image data and lidar point cloud data of the target vehicle, and the true value data includes position data and shape data of the lane line in three-dimensional space.

[0011] In one embodiment, lane lines are fitted according to true value data to obtain fitted lane lines and sampling point coordinates, including:

[0012] Based on the true value data and the preset spline interpolation strategy, polynomials of the lane lines in the X, Y, and Z directions are determined to obtain the fitted lane lines. The polynomials include unique fitting parameters.

[0013] In the preset fitting parameter value range, multiple fitting parameter sampling values are randomly selected, and multiple three-dimensional sampling point coordinates are obtained according to the fitting parameter sampling values and the polynomial.

[0014] In one embodiment, determining the offset of each sampling point according to a preset offset determination method includes:

[0015] Divide the three-dimensional space to which the lane lines belong into a plurality of three-dimensional subspaces with side lengths of preset length values;

[0016] A first offset in the X direction and a second offset in the Y direction of the coordinates of each sampling point relative to the center point of the three-dimensional subspace to which it belongs are calculated.

[0017] In one embodiment, the camera parameters include the focal length, principal point coordinates, translation vector, and rotation matrix of the camera.

[0018] In one embodiment, generating a camera parameter feature vector according to the camera parameters includes:

[0019] Fill all camera parameters into an array in order, input the array into the preset multi-layer perceptron, and obtain the corresponding camera parameter feature vector.

[0020] In a second aspect, the present application further provides a monocular 3D detection device for lane lines with large curvature, the device comprising:

[0021] The sampling data acquisition module is used to obtain sampling data of the target vehicle under different types of roads, different weather conditions and different traffic densities, and determine the true value data of the lane line where the target vehicle is located in three-dimensional space based on the sampling data;

[0022] An offset determination module is used to fit the lane lines according to the true value data, obtain the fitted lane lines and sampling point coordinates, and determine the offset of each sampling point according to a preset offset determination method;

[0023] The detection model construction module is used to obtain the camera parameters on the target vehicle, generate a camera parameter feature vector based on the camera parameters, and build a 3D lane detection network model based on the camera parameter feature vector and the offset of the sampling point;

[0024] The detection data generation module is used to iteratively train the three-dimensional lane line detection network model until the preset model convergence conditions are reached, obtain the target lane line detection network model, and deploy the target lane line detection network model on the target vehicle to obtain the lane line detection data of the target vehicle. The lane detection data includes the three-dimensional coordinates and curvature of the lane line.

[0025] In a third aspect, the present application further provides a computer device, comprising a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0026] Obtain sampling data of the target vehicle under different types of roads, different weather conditions, and different traffic densities, and determine the true value data of the lane line where the target vehicle is located in three-dimensional space based on the sampling data;

[0027] Fit the lane lines according to the true value data to obtain the fitted lane lines and sampling point coordinates, and determine the offset of each sampling point according to the preset offset determination method;

[0028] Obtain the camera parameters on the target vehicle, generate a camera parameter feature vector based on the camera parameters, and build a 3D lane detection network model based on the camera parameter feature vector and the offset of the sampling point;

[0029] The three-dimensional lane line detection network model is iteratively trained until the preset model convergence conditions are reached to obtain the target lane line detection network model. The target lane line detection network model is deployed on the target vehicle to obtain the lane detection data of the lane line where the target vehicle is located. The lane detection data includes the three-dimensional coordinates and curvature of the lane line.

[0030] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the following steps:

[0031] Obtain sampling data of the target vehicle under different types of roads, different weather conditions, and different traffic densities, and determine the true value data of the lane line where the target vehicle is located in three-dimensional space based on the sampling data;

[0032] Fit the lane lines according to the true value data to obtain the fitted lane lines and sampling point coordinates, and determine the offset of each sampling point according to the preset offset determination method;

[0033] Obtain the camera parameters on the target vehicle, generate a camera parameter feature vector based on the camera parameters, and build a 3D lane detection network model based on the camera parameter feature vector and the offset of the sampling point;

[0034] The three-dimensional lane line detection network model is iteratively trained until the preset model convergence conditions are reached to obtain the target lane line detection network model. The target lane line detection network model is deployed on the target vehicle to obtain the lane detection data of the lane line where the target vehicle is located. The lane detection data includes the three-dimensional coordinates and curvature of the lane line.

[0035] In a fifth aspect, the present application further provides a computer program product, the computer program product comprising a computer program, which, when executed by a processor, implements the following steps:

[0036] Obtain sampling data of the target vehicle under different types of roads, different weather conditions, and different traffic densities, and determine the true value data of the lane line where the target vehicle is located in three-dimensional space based on the sampling data;

[0037] Fit the lane lines according to the true value data to obtain the fitted lane lines and sampling point coordinates, and determine the offset of each sampling point according to the preset offset determination method;

[0038] Obtain the camera parameters on the target vehicle, generate a camera parameter feature vector based on the camera parameters, and build a 3D lane detection network model based on the camera parameter feature vector and the offset of the sampling point;

[0039] The three-dimensional lane line detection network model is iteratively trained until the preset model convergence conditions are reached to obtain the target lane line detection network model. The target lane line detection network model is deployed on the target vehicle to obtain the lane detection data of the lane line where the target vehicle is located. The lane detection data includes the three-dimensional coordinates and curvature of the lane line.

[0040] The above-mentioned monocular 3D detection method for lane lines with large curvature first obtains sampling data of the target vehicle under different types of roads, different weather conditions and different traffic densities, and determines the true value data of the lane line where the target vehicle is located in three-dimensional space based on the sampling data; then, the lane line is fitted according to the true value data to obtain the fitted lane line and sampling point coordinates, and the offset of each sampling point is determined according to a preset offset determination method; then, the camera parameters on the target vehicle are obtained, a camera parameter feature vector is generated based on the camera parameters, and a three-dimensional lane line detection network model is constructed based on the camera parameter feature vector and the offset of the sampling point; finally, the three-dimensional lane line detection network model is iteratively trained until the preset model convergence condition is reached, and the target lane line detection network model is obtained. The target lane line detection network model is deployed on the target vehicle to obtain the lane detection data of the lane line where the target vehicle is located, and the lane detection data includes the three-dimensional coordinates and curvature of the lane line. The present application adopts the above-mentioned method to make the 3D detection results of lane lines with large curvature more accurate and reliable, thereby providing more accurate lane line coordinate information, which in turn helps to enhance the processing capability of autonomous driving perception for complex road scenes, improve driving safety, and reduce accidents caused by lane departure. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 This is a diagram showing the application environment of a monocular 3D detection method for large-curvature lane lines in one embodiment;

[0042] Figure 2 A flowchart of obtaining a target lane line detection network model in one embodiment;

[0043] Figure 3 A flowchart of obtaining three-dimensional sampling point coordinates in one embodiment;

[0044] Figure 4 A flowchart of calculating the offset of each sampling point coordinate in one embodiment;

[0045] Figure 5 Flowchart of a method for monocular 3D detection of lane markings with large curvature according to one embodiment;

[0046] Figure 6 This is a block diagram of a monocular 3D detection device for large curvature lane markings according to an embodiment;

[0047] Figure 7 is a diagram of the internal structure of a computer device in one embodiment;

[0048] Figure 8 FIG. 4 is a diagram showing the internal structure of a computer device in another embodiment. DETAILED DESCRIPTION

[0049] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0050] The monocular 3D detection method for large curvature lane lines provided in the embodiment of the present application can be applied to Figure 1 In the application environment shown. Among them, the terminal 102 communicates with the server 104 through the network. The data storage system can store data that the server 104 needs to process. The data storage system can be integrated on the server 104, or it can be placed on the cloud or other network servers. Among them, the terminal 102 can be, but is not limited to, various personal computers, laptops, smart phones, tablets and Internet of Things devices. The Internet of Things devices can be smart speakers, smart air conditioners, smart car-mounted devices, etc. The terminal 102 is used to obtain the status data of the target vehicle and the lane line data, etc. The server 104 can be implemented as an independent server or a server cluster consisting of multiple servers.

[0051] Lane markings with large curvature refer to those with significant curvature, such as those found on curves on highways, sharp turns on urban roads, and winding curves on mountain roads. Compared to straight or shallowly curvatured lane markings, lane markings with large curvature exhibit more complex shape variations, placing higher demands on detection technology. "Monocular" refers to the use of a single camera as an image acquisition device. In a vision system, a monocular camera acts like a human eye, capturing scene information by capturing two-dimensional images. Compared to binocular or multi-camera systems, monocular cameras are lower cost, easier to install, and occupy less space, making them more widely applicable in practical applications. 3D detection aims to determine the position, shape, and pose of a target object (in this case, a lane marking) in three-dimensional space. Unlike traditional 2D detection, which only provides the position and size of an object on the image plane, 3D detection provides autonomous driving systems with more comprehensive and accurate environmental information, helping the vehicle better understand the surrounding road scene and make more informed decisions.

[0052] In one embodiment, Figure 2 As shown, this method is applied to Figure 1 The terminal in the example is used for illustration. It is understood that the method can also be applied to a server, and can also be applied to a system including a terminal and a server, and is implemented through the interaction between the terminal and the server. In this embodiment, the method includes the following steps:

[0053] Step 202 : acquiring sampling data of the target vehicle under different types of roads, different weather conditions, and different traffic densities, and determining the true value data of the lane line where the target vehicle is located in three-dimensional space based on the sampling data.

[0054] This step aims to comprehensively collect the driving data of the target vehicle in various complex scenarios, and then accurately determine the real position information of the lane line where the target vehicle is located in three-dimensional space based on this data.

[0055] The target vehicle is the vehicle selected for monocular 3D detection of lane lines with large curvature. Different road types include urban roads, highways, and rural roads. Different weather conditions include sunny, rainy, foggy, and snowy days. Different traffic densities refer to traffic flow density. By sampling driving data from the target vehicle in these diverse scenarios, the subsequent model can learn from a variety of possible real-world situations, thereby achieving better generalization capabilities.

[0056] Sampled data can be information collected about the target vehicle during a specific time period and scenario. This information may include the vehicle's position, speed, acceleration, camera images, and LiDAR point cloud data. Ground truth data is accurate reference data. In this example, it can be the actual position and shape of the lane in which the target vehicle is located in three-dimensional space. This data is typically acquired using high-precision measurement equipment (such as LiDAR and GPS) and serves as a benchmark for subsequent model training and evaluation, measuring the accuracy of model predictions.

[0057] Step 204 : Fit the lane lines according to the true value data to obtain the fitted lane lines and sampling point coordinates, and determine the offset of each sampling point according to a preset offset determination method.

[0058] After obtaining the true lane line data, these discrete true data points need to be fitted to produce a continuous, smooth lane line curve. Common fitting methods include polynomial fitting and spline fitting. The fitted lane line more accurately represents the actual lane shape. Then, using a preset offset determination method, the offset of each sampling point on the lane line relative to a reference position (such as the grid center) is calculated. This offset calculation provides more detailed lane line position information, helping to improve the model's detection accuracy.

[0059] Fitting involves mathematically finding a curve or function that approximates a given set of discrete data points as closely as possible. In lane fitting, the goal is to find a curve that accurately describes the shape of the lane for subsequent analysis and processing. The sampling point coordinates are the coordinates of a series of representative points selected on the fitted lane line. These points describe the shape and position of the lane and serve as the basis for subsequent offset calculations and model construction. The offset is the positional deviation of the sampling point relative to a reference position. In lane detection, this typically refers to the x and y offset of the sampling point relative to the center of the grid within a given grid. The introduction of the offset can help the model more accurately locate the position of the lane line.

[0060] Step 206 , obtaining camera parameters on the target vehicle, generating a camera parameter feature vector based on the camera parameters, and constructing a three-dimensional lane detection network model based on the camera parameter feature vector and the offset of the sampling point.

[0061] Camera parameters are a set of parameters that describe the camera's imaging characteristics and installation location. Intrinsic parameters are primarily related to the camera's optical properties, such as focal length, which determines the camera's field of view, and principal point coordinates, which represent the image center. Extrinsic parameters, on the other hand, are related to the camera's position and orientation in the world coordinate system and are used to convert the camera coordinate system to the world coordinate system.

[0062] The camera parameter feature vector is a vector obtained by processing and transforming the camera parameters. This vector encodes various camera parameter information and can be more easily integrated with other features for subsequent model training and inference.

[0063] A 3D lane detection network model is a neural network model specifically designed to detect the position and shape of lane lines in 3D space. This model typically consists of multiple layers of neural networks. By learning input information such as image features, camera parameter feature vectors, and sampling point offsets, it outputs information such as the 3D coordinates and curvature of lane lines.

[0064] Specifically, various parameters of the target vehicle's camera must first be acquired, including intrinsic parameters (such as focal length and principal point coordinates) and extrinsic parameters (such as camera position and pose). These parameters describe the camera's imaging characteristics and its installation position on the vehicle. These camera parameters are then input into a specific model (such as a multi-layer perceptron (MLP)) to generate a camera parameter feature vector. This feature vector contains key information about the camera and provides useful context for subsequent lane detection. Finally, the offsets of the previously obtained sampling points and the camera parameter feature vector are combined to construct a 3D lane detection network model. This model uses this information to learn how to accurately detect the 3D position and shape of lane lines from camera images.

[0065] In step 208, the three-dimensional lane line detection network model is iteratively trained until the preset model convergence condition is reached, and a target lane line detection network model is obtained. The target lane line detection network model is deployed on the target vehicle to obtain lane detection data of the lane line where the target vehicle is located. The lane detection data includes the three-dimensional coordinates and curvature of the lane line.

[0066] The preset model convergence conditions are pre-set criteria for determining whether model training is complete. Common convergence conditions include the loss function reaching a certain minimum value, the loss function decreasing less than a certain threshold, and the number of training rounds reaching a preset maximum value. When the convergence conditions are met, the model is considered to have learned enough information and training can be stopped.

[0067] The target lane detection network model is a 3D lane detection network model that has been trained and meets convergence requirements. This model has high accuracy and reliability and can accurately detect the 3D information of lane lines in practical applications.

[0068] Lane detection data is information about lane lines output by the target lane detection network model. It primarily includes the 3D coordinates and curvature of the lane lines. The 3D coordinates describe the position of the lane lines in three-dimensional space, while the curvature reflects the degree of curvature. This information is crucial for autonomous driving and navigation.

[0069] Specifically, the constructed three-dimensional lane line detection network model is iteratively trained using the collected sample data and the corresponding true value data. During the training process, the model will continuously adjust its own parameters to minimize the difference between the predicted results and the true value data. Training will continue until the preset model convergence conditions are met, such as the value of the loss function no longer decreases or the decrease is less than a certain threshold. When the model converges, the target lane line detection network model is obtained. Finally, this trained model is deployed on the target vehicle, enabling it to process the images captured by the camera in real time and output the lane detection data of the lane line in which the target vehicle is located, including information such as the three-dimensional coordinates and curvature of the lane line, to support the vehicle's autonomous driving decision-making.

[0070] The above-mentioned monocular 3D detection method for large-curvature lane lines can make the 3D detection results of large-curvature lane lines more accurate and reliable, thereby providing more accurate lane line coordinate information, which in turn helps to enhance the autonomous driving perception's ability to handle complex road scenes, improve driving safety, and reduce accidents caused by lane departure.

[0071] In one embodiment, the sampled data includes front view image data and lidar point cloud data of the target vehicle, and the true value data includes position data and shape data of the lane line in three-dimensional space.

[0072] The forward-view image data is a sequence of images captured by a camera mounted in front of the target vehicle. These images capture visual information about the road scene ahead, including lane markings, other vehicles, pedestrians, traffic signs, and the surrounding environment. Various features can be extracted from these images, such as lane marking color, texture, and edges. These features are crucial for identifying and detecting lane markings. By processing and analyzing these images, areas within the image that may represent lane markings can be identified.

[0073] LiDAR (LiDAR) uses laser beams to emit and measure the time it takes for reflected light to acquire the three-dimensional coordinates of surrounding objects. These three-dimensional coordinates form a point cloud. For lane detection, this point cloud contains the precise position of lane lines and surrounding objects in three-dimensional space.

[0074] Lane line position data in three-dimensional space refers to the specific coordinates of lane lines in a three-dimensional coordinate system, typically expressed in a world coordinate system or a vehicle coordinate system. These coordinates precisely describe the position of lane lines in space, including the starting and ending points of the lane lines, as well as the 3D coordinates of each point along the line. When training the subsequent lane line detection model, this position data serves as the ground truth to monitor the model's learning process. The model continuously adjusts its parameters to ensure that the predicted lane line positions are as close as possible to the ground truth data, thereby improving the model's detection accuracy. After model training is complete, the position data is used to evaluate the model's performance. By comparing the differences between the model's predicted lane line positions and the ground truth data, various evaluation metrics, such as mean squared error and mean absolute error, can be calculated to objectively assess the model's accuracy and reliability.

[0075] Lane shape data describes the lane's geometric shape, such as straight lines, curves, and arcs, as well as information such as the curvature and direction of the curve. This data more comprehensively characterizes lane features and reflects their true form. Combining position and shape data allows for a more accurate mathematical model of lanes. For example, for curved lanes, the curvature information in the shape data can help the model better fit the curve, improving the model's detection capabilities for complex lanes.

[0076] In one embodiment, Figure 3 As shown, the lane line is fitted according to the true value data to obtain the fitted lane line and sampling point coordinates, including:

[0077] Step 302 : Based on the true value data and a preset spline interpolation strategy, polynomials of the lane lines in the X, Y, and Z directions are determined to obtain fitted lane lines. The polynomials include unique fitting parameters.

[0078] Using known ground-truth lane data, a pre-set spline interpolation strategy is used to construct a polynomial describing the lane's shape in three-dimensional space. Specifically, the polynomials for the lane's shape in the X, Y, and Z directions are processed separately. Because the lane's position in three-dimensional space must be accurately represented by coordinates in three dimensions, a polynomial is determined for each direction. Together, these polynomials can fully describe the lane's shape in three dimensions.

[0079] Among them, the spline interpolation strategy is a mathematical method used to construct smooth curves between known discrete data points. In lane line fitting, spline interpolation can generate a continuous, smooth curve based on the discrete points in the true value data to approximate the lane line. Common spline interpolation methods include cubic spline interpolation. It constructs polynomial functions between each data point, so that these polynomial functions have a certain degree of continuity and smoothness at the connection points. A polynomial is an algebraic expression consisting of variables and coefficients. In lane line fitting, polynomials are determined separately for the X direction, Y direction, and Z direction, and each polynomial contains unique fitting parameters.

[0080] Specifically, this embodiment considers resampling from the dimensions of the X direction and the Y direction. n ,y n , z n ), firstly, the curve is parameterized, and the corresponding parameter t is calculated for each point. Then, cubic spline interpolation is used to obtain the polynomials x(t), y(t), and z(t) of x, y, and z with respect to t, respectively. In order to sample uniformly on the xy plane, the arc length s(t) of the projection of the lane curve on the xy plane is calculated. Then, the arc length s is divided into N parts, s i is the arc length s of the i-th sampling point i = s / N; for each s i , use Newton iteration method to calculate the corresponding s(t i )=s i ; get t i After that, the coordinates of the final sampling point (x i ,y i , z i ). The 3D space to which the lane lines belong is then gridded, and the offset of the sampling point in the grid relative to the grid center in the X and Y directions is calculated.

[0081] Step 304 : within a preset fitting parameter value range, select a plurality of fitting parameter sampling values, and obtain a plurality of three-dimensional sampling point coordinates according to the fitting parameter sampling values and the polynomial.

[0082] Specifically, after determining the polynomials describing the lane lines, each polynomial has corresponding fitting parameters. To obtain multiple sampling points on the lane lines, it is necessary to select multiple different fitting parameter sampling values within a preset fitting parameter value range. This value range is pre-set based on actual conditions and experience to ensure that the selected parameter values can reasonably reflect the changes in the lane lines. In other words, the fitting parameter value range is the pre-set value range for each fitting parameter, and the fitting parameter sampling values are specific values selected from the fitting parameter value range.

[0083] After obtaining multiple fitting parameter sample values, these values are substituted into the previously determined polynomials for the X, Y, and Z directions to calculate the coordinate values in each direction. This way, for each combination of fitting parameter sample values, a set of three-dimensional coordinates (X, Y, Z) is obtained. These coordinates represent a sampling point on the lane line. By selecting multiple different fitting parameter sample values, multiple three-dimensional sampling point coordinates can be obtained, which can approximately describe the shape and position of the lane line.

[0084] In this embodiment, by determining the polynomials of the lane lines in the X, Y, and Z directions respectively, the position and changes of the lane lines in three-dimensional space can be described in more detail. Compared with simple two-dimensional fitting, three-dimensional polynomial fitting takes into account the height information of the lane lines and can more accurately reflect the actual situation of the lane lines, thereby reducing fitting errors.

[0085] In one embodiment, Figure 4 As shown, according to the preset offset determination method, the offset of each sampling point is determined, including:

[0086] Step 402: Divide the three-dimensional space to which the lane lines belong into a plurality of three-dimensional subspaces each having a side length of a preset length value.

[0087] The three-dimensional space to which lane lines belong is the entire three-dimensional physical space containing the lane lines. Within this space, lane lines exist as three-dimensional coordinates, with information in the three dimensions of length, width, and height. A three-dimensional subspace is a small region of space obtained by dividing the three-dimensional space containing the lane lines according to certain rules. Each three-dimensional subspace is a cube with a preset length on each side. Together, these subspaces constitute the entire three-dimensional space. The preset length is a fixed value that determines the size of the three-dimensional subspace. The choice of this value affects the accuracy and computational complexity of subsequent offset calculations.

[0088] For example, the 3D space to which the lane lines belong is gridded, for example, the space is divided into small grids with a side length of 0.5 meters.

[0089] Step 404 : Calculate a first offset in the X direction and a second offset in the Y direction of the coordinates of each sampling point relative to the center point of the three-dimensional subspace to which it belongs.

[0090] The first offset refers to the difference in the X-direction between the sampling point coordinates and the center of the 3D subspace to which they belong, reflecting the degree to which the sampling point deviates from the center of the 3D subspace in the X-direction. The second offset refers to the difference in the Y-direction between the sampling point coordinates and the center of the 3D subspace to which they belong, reflecting the degree to which the sampling point deviates from the center of the 3D subspace in the Y-direction.

[0091] Specifically, for each sampling point (X i , Y i , Z i ), determine the grid to which it belongs, and calculate the offset of the sampling point in the X and Y directions relative to the grid center within the grid to which it belongs.

[0092] In this embodiment, after dividing the three-dimensional space into multiple subspaces, the calculation of sampling point offsets only needs to be performed within the subspace to which they belong, avoiding traversal of the entire three-dimensional space. This significantly reduces the computational effort and improves computational efficiency. Furthermore, by calculating the offsets of sampling points relative to the subspace center, the characteristics and variations of lane markings within a local area can be highlighted. The distribution of offsets within different subspaces can reveal local characteristics of lane markings, such as curvature and tilt, at different locations, facilitating a more detailed analysis of lane marking morphology and structure.

[0093] In one embodiment, the camera parameters include the focal length, principal point coordinates, translation vector, and rotation matrix of the camera.

[0094] Specifically, for each frame of data collected, the camera internal and external parameters corresponding to this frame are recorded, including focal length, principal point coordinates, translation vector, and rotation matrix. The camera internal and external parameters describe the imaging characteristics of the camera and its position and posture in the world coordinate system. These parameters are very important for converting image features into three-dimensional space. For example, the focal length determines the scaling of the image, the principal point coordinates indicate the center position of the image, and the translation vector and rotation matrix describe the position and direction of the camera relative to the world coordinate system. For example, the focal length f in a certain camera parameter is x =2500,f y =2500, principal point coordinates u0=960, v0=540, translation vector x p =0.5,y p =0,z p =1.5.

[0095] In one embodiment, generating a camera parameter feature vector according to camera parameters includes: filling all camera parameters into an array in order, inputting the array into a preset multi-layer perceptron, and obtaining a corresponding camera parameter feature vector.

[0096] Specifically, the multilayer perceptron in this example has two layers, the first layer has 64 neurons, and the second layer has 32 neurons. After the MLP calculation, the camera parameter feature vector is output. MLP can perform nonlinear transformation and feature extraction on the camera parameters to obtain a more representative feature vector. For example, assuming the camera parameter array is {f x , f y ,u0,v0,x p ,y p , z p}, input it into the MLP, and after calculation by two layers of neurons, output a 32-dimensional camera parameter feature vector.

[0097] In a detailed embodiment, referring to Figure 5 , install a high-definition camera with a resolution of 1920*1080 on the roof of the target vehicle, and set up a lidar device. First, synchronously collect the front view image of the high-definition camera and the lidar point cloud data. Use the point cloud data as a reference to accurately mark the 3D lane lines in the image. When collecting data, it is necessary to collect data under different road types, different weather conditions and different traffic densities. Then, for the marked lane lines, the corresponding parameter t is calculated for each point, and the cubic spline interpolation method is used to obtain the polynomials x(t), y(t), and z(t) of x, y, and z with respect to t, respectively. Cubic spline interpolation can fit a smooth curve so that the x, y, and z coordinates can be uniquely determined by the parameter t. For example, suppose there are three marked points on a lane line with coordinates of (1, 2, 0), (2, 4, 0.5), and (3, 6, 1), respectively. The cubic spline interpolation algorithm can be used to obtain three polynomials x(t), y(t), and z(t). When t is within a certain range, the coordinates of any point on the lane line can be calculated. Calculate the arc length s(t) of the projection of the lane curve on the xy plane. Assuming that the arc length s is a fixed value, divide it into N=100 parts (N can be adjusted according to actual conditions). Then the arc length s of the i-th sampling point is i =s / 100, for each S i , use Newton iteration method to calculate the corresponding s (t i ) =s i , constantly adjust the value of t so that s(t) gradually approaches s i , and finally get t that meets the conditions i . We can get t iThen, substitute it into x(t), y(t), z(t) to calculate the coordinates of the final sampling point (x i ,y i , z i ), for example, for the calculated t1=0.1, substitute it into x(t), y(t), z(t) to obtain x1=x(0.1), y1=y(0.1), z1=z(0.1), that is, the coordinates of the first sampling point are obtained.

[0098] Secondly, we generate the ground truth for training. Specifically, we grid the 3D space to which the lane lines belong, for example, we divide the space into small grids with a side length of 0.5 meters. For each sampling point (x i ,y i , z i ), determine the grid to which it belongs, and calculate the x and y offsets of the sampling point relative to the grid center within the grid to which it belongs. For example, if the coordinates of a sampling point are (1.2, 2.3, 0) and the center coordinates of the grid are (1, 2, 0), then the first offset of the sampling point in the x direction is 0.2m, and the second offset in the y direction is 0.3m.

[0099] Next, obtain the camera parameters. Specifically, for each frame of captured data, record the camera's intrinsic and extrinsic parameters corresponding to that frame, including focal length, principal point coordinates, translation vector, and rotation matrix. These parameters describe the camera's imaging characteristics and its position and pose in the three-dimensional coordinate system. All camera parameters are sequentially entered into an array and fed into a two-layer multilayer perceptron (MLP). The first layer of the MLP has 64 neurons, and the second layer has 32 neurons. After calculation, the MLP outputs a camera parameter feature vector.

[0100] Next, a 3D lane detection network model is built. Specifically, ResNet-50 is used as the backbone network for image feature extraction. This network performs multi-level downsampling on the input front-view image to generate multi-scale image features. ResNet-50 is a deep convolutional neural network with strong feature extraction capabilities. Through different convolutional and pooling layers, it can extract features at different levels of the image, such as edges, texture, and shape. For example, after the first layer of convolution and pooling, a 1 / 4-scale feature map is generated, which retains some macroscopic features of the image. Subsequent convolutional and pooling layers produce feature maps at scales of 1 / 8, 1 / 16, and so on, which contain more refined image features. The multi-scale image features are then fused with the camera parameter features obtained in step 3. First, the two features are combined through feature concatenation. Then, through a series of convolution and upsampling operations, fused image features from the perspective (PV) view are generated. Next, methods such as inverse perspective transformation (IPM) are used to convert the PV-view fused image features into BEV spatial features. For example, after concatenating image features at 1 / 4, 1 / 8, and 1 / 16 scales with camera parameter features, convolution and upsampling are performed to obtain a higher-resolution PV-perspective fused image feature. This feature is then converted to a bird's-eye-view BEV spatial feature using an inverse perspective transformation. This more intuitively represents the position and layout of lane lines on the horizontal plane. A lane detection head is then constructed based on the Transformer architecture. This head receives the BEV spatial features as input and, through a multi-head attention mechanism and a feedforward neural network, regresses and predicts lane-related information. This includes lane confidence, which indicates the reliability of the detected lane line; lane line category, such as solid or dashed; lane line coordinate z-axis height, which is the height of the lane line relative to the ground; and lane line offsets in the x and y directions. Finally, the actual lane line coordinates are parsed based on the predicted offsets and grid information. For example, the multi-head attention mechanism allows the detection head to focus on important lane-related information in the BEV spatial features. After calculations in the feedforward neural network, the head outputs information such as lane confidence, category, height, and offset. Based on the offset and grid information, the actual coordinates of the lane line in three-dimensional space can be calculated.

[0101] Then, model training is performed. Specifically, FocalLoss is used as the loss function for lane confidence and lane classification to address sample imbalance. FocalLoss reduces the weight of easily classified samples and increases the weight of difficult-to-classify samples, thereby more effectively handling sample imbalance. For example, in lane detection, there may be a large number of background samples and a small number of lane samples. Using FocalLoss allows the model to focus more on classifying lane samples. For coordinate offset, the mean squared error (MSE) is used as the loss function to measure the difference between the predicted value and the true value. The weights of the two are set to 2 and 7, respectively: FocalLoss has a weight of 2 and MSE has a weight of 7. This allows the contribution of the loss function to be adjusted based on the importance of different tasks. The model is then trained using the Adam optimizer. The Adam optimizer combines the advantages of AdaGrad and RMSProp, adaptively adjusting the learning rate of each parameter and achieving good convergence performance. The number of training rounds was set to 100, with an initial learning rate of 0.001, and the learning rate decayed to 0.8 of the previous round's value with each round. During training, the collected annotated data was divided into a training set, a validation set, and a test set in an 8:1:1 ratio. The training set was used for model training, the validation set was used to adjust model parameters, and the test set was used to evaluate model performance. For example, during training, the model parameters were continuously adjusted based on the data from the training set. The validation set was used to evaluate the model's performance on unseen data, and the model's hyperparameters, such as the learning rate, were adjusted based on the results of the validation set. Finally, the test set was used to evaluate the model's final performance.

[0102] Finally, the application is deployed on a real vehicle. Specifically, the trained network model is quantized to INT8 using the TensorRT tool, reducing the model's storage space and computational complexity while increasing its inference speed. TensorRT is a high-performance deep learning inference optimizer from NVIDIA. It converts model parameters from floating-point numbers to 8-bit integers, thereby reducing the model's storage space and computational complexity. For example, a model that originally occupied 100MB of storage space may require only 25MB of storage after INT8 quantization, significantly improving inference speed. The quantized model is then deployed on an edge computing device. This device runs Ubuntu and is equipped with an Nvidia graphics card, boasting a computing power of 100 Tops per second or higher, meeting the requirements for real-time processing of the vehicle's forward-view imagery and outputting lane detection results. As the vehicle drives, forward-view images are captured in real time, processed by the model, and output with lane position and related information to assist the vehicle in making autonomous driving decisions. For example, when a vehicle is driving, the edge computing device will receive the forward-looking images captured by the high-definition camera in real time and input them into the quantized model for processing. The model will quickly output information such as the location, category, and confidence level of the lane line. The vehicle's autonomous driving system can make decisions such as lane keeping and lane changing based on this information.

[0103] It should be understood that, although the various steps in the flowcharts involved in the various embodiments described above are displayed in sequence according to the instructions of the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and these steps can be executed in other orders. Moreover, at least a portion of the steps in the flowcharts involved in the various embodiments described above can include multiple steps or multiple stages, and these steps or stages are not necessarily executed and completed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a portion of steps or stages in other steps.

[0104] Based on the same inventive concept, embodiments of the present application also provide a detection device for implementing the aforementioned method for monocular 3D detection of lane markings with large curvature. The solution provided by this device is similar to the solution described in the aforementioned method. Therefore, the specific limitations of one or more embodiments of the monocular 3D detection device for lane markings with large curvature provided below can be found in the limitations of the monocular 3D detection method for lane markings with large curvature described above and will not be repeated here.

[0105] In one embodiment, Figure 6As shown, a monocular 3D detection device for lane lines with large curvature is provided, comprising: a sampling data acquisition module 602, an offset determination module 604, a detection model construction module 606, and a detection data generation module 608, wherein:

[0106] The sampling data acquisition module 602 is used to obtain sampling data of the target vehicle under different types of roads, different weather conditions and different traffic densities, and determine the true value data of the lane line where the target vehicle is located in three-dimensional space based on the sampling data;

[0107] An offset determination module 604 is configured to fit the lane lines according to the true value data, obtain the fitted lane lines and sampling point coordinates, and determine the offset of each sampling point according to a preset offset determination method;

[0108] A detection model construction module 606 is used to obtain camera parameters on the target vehicle, generate a camera parameter feature vector based on the camera parameters, and construct a three-dimensional lane detection network model based on the camera parameter feature vector and the offset of the sampling point;

[0109] The detection data generation module 608 is used to iteratively train the three-dimensional lane line detection network model until the preset model convergence conditions are reached, obtain the target lane line detection network model, and deploy the target lane line detection network model on the target vehicle to obtain the lane line detection data of the lane line where the target vehicle is located. The lane detection data includes the three-dimensional coordinates and curvature of the lane line.

[0110] In one embodiment, the sampling data acquisition module 602 is further used to define that the sampling data includes the front view image data and the laser radar point cloud data of the target vehicle, and the true value data includes the position data and shape data of the lane line in three-dimensional space.

[0111] In one embodiment, the offset determination module 604 is further used to: determine the polynomials of the lane line in the X direction, Y direction, and Z direction respectively based on the true value data and a preset spline interpolation strategy to obtain the fitted lane line, where the polynomial includes a unique fitting parameter; within the preset fitting parameter value range, randomly select multiple fitting parameter sampling values, and obtain multiple three-dimensional sampling point coordinates based on the fitting parameter sampling values and the polynomial.

[0112] In one embodiment, the offset determination module 604 is further configured to: divide the three-dimensional space to which the lane line belongs into a plurality of three-dimensional subspaces with side lengths having preset length values; and calculate a first offset in the X direction and a second offset in the Y direction of the coordinates of each sampling point relative to the center point of the three-dimensional subspace to which it belongs.

[0113] In one embodiment, the detection model building module 606 is further configured to define camera parameters including the focal length, principal point coordinates, translation vector, and rotation matrix of the camera.

[0114] In one embodiment, the detection model building module 606 is further configured to sequentially fill all camera parameters into an array, input the array into a preset multi-layer perceptron, and obtain a corresponding camera parameter feature vector.

[0115] Each module in the aforementioned monocular 3D lane detection device for large curvature lanes can be implemented in whole or in part through software, hardware, or a combination thereof. Each module can be embedded in or independent of a processor within a computer device in hardware form, or stored in a computer device's memory as software, allowing the processor to call and execute the corresponding operations of each module.

[0116] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 7 As shown. The computer device includes a processor, a memory and a network interface connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store sampling data, lane line detection data, etc. The network interface of the computer device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, a monocular 3D detection method for large curvature lane lines is implemented.

[0117] In one embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as follows: Figure 8As shown. The computer device includes a processor, memory, a communication interface, a display screen, and an input device connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and computer program in the non-volatile storage medium. The communication interface of the computer device is used to communicate with an external terminal via wired or wireless communication. The wireless communication can be achieved via Wi-Fi, a mobile cellular network, NFC (near-field communication), or other technologies. When executed by the processor, the computer program implements a monocular 3D detection method for lane lines with large curvature. The display screen of the computer device can be a liquid crystal display or an electronic ink display. The input device of the computer device can be a touch layer covering the display screen, or keys, a trackball, or a touchpad provided on the computer device housing, or an external keyboard, touchpad, or mouse.

[0118] Those skilled in the art will understand that Figure 7 and Figure 8 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0119] In one embodiment, a computer device is provided, including a memory and a processor. The memory stores a computer program, and the processor implements the steps in the above method embodiments when executing the computer program.

[0120] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0121] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0122] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties.

[0123] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the above-mentioned embodiments. In particular, any reference to memory, database, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The databases involved in the various embodiments provided herein may include at least one of a relational database and a non-relational database. Non-relational databases may include, but are not limited to, distributed databases based on blockchains. The processors involved in the various embodiments provided herein may be, but are not limited to, general-purpose processors, central processing units (CPUs), graphics processors (GPUs), digital signal processors (DSPs), programmable logic devices (PLDs), data processing logic devices based on quantum computing, and the like.

[0124] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0125] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present application. It should be noted that a person of ordinary skill in the art may make various modifications and improvements without departing from the spirit of the present application, and these modifications and improvements fall within the scope of protection of the present application. Therefore, the scope of protection of the present application shall be determined by the appended claims.

Claims

1. A monocular 3D detection method for lane markings with large curvature, characterized by: The method comprises: Acquire sampling data of the target vehicle under different types of roads, different weather conditions, and different traffic densities, and determine the true value data of the lane line where the target vehicle is located in three-dimensional space based on the sampling data; Fitting the lane line according to the true value data to obtain the fitted lane line and sampling point coordinates, and determining the offset of each sampling point according to a preset offset determination method; Obtaining camera parameters on the target vehicle, generating a camera parameter feature vector based on the camera parameters, and constructing a three-dimensional lane detection network model based on the camera parameter feature vector and the offset of the sampling point; The three-dimensional lane line detection network model is iteratively trained until a preset model convergence condition is reached to obtain a target lane line detection network model, and the target lane line detection network model is deployed on the target vehicle to obtain lane detection data of the lane line where the target vehicle is located, wherein the lane detection data includes the three-dimensional coordinates and curvature of the lane line.

2. The method according to claim 1, characterized in that The sampling data includes the front view image data and the laser radar point cloud data of the target vehicle, and the true value data includes the position data and shape data of the lane line in three-dimensional space.

3. The method according to claim 1, characterized in that The fitting of the lane line according to the true value data to obtain the fitted lane line and sampling point coordinates includes: Determining polynomials of the lane line in the X direction, the Y direction, and the Z direction, respectively, based on the true value data and a preset spline interpolation strategy, to obtain the fitted lane line, wherein the polynomial includes a unique fitting parameter; In a preset fitting parameter value range, a plurality of fitting parameter sampling values are randomly selected, and a plurality of three-dimensional sampling point coordinates are obtained according to the fitting parameter sampling values and the polynomial.

4. The method according to claim 3, characterized in that Determining the offset of each sampling point according to a preset offset determination method includes: Dividing the three-dimensional space to which the lane line belongs into a plurality of three-dimensional subspaces with side lengths of preset length values; A first offset in the X direction and a second offset in the Y direction of the coordinates of each sampling point relative to the center point of the three-dimensional subspace to which it belongs are calculated.

5. The method according to claim 1, wherein The camera parameters include the focal length, principal point coordinates, translation vector, and rotation matrix of the camera.

6. The method according to claim 1, characterized in that Generating a camera parameter feature vector according to the camera parameters includes: All the camera parameters are sequentially filled into an array, and the array is input into a preset multi-layer perceptron to obtain the corresponding camera parameter feature vector.

7. A monocular 3D detection device for lane lines with large curvature, characterized in that: The device comprises: A sampling data acquisition module is used to obtain sampling data of the target vehicle under different types of roads, different weather conditions and different traffic densities, and determine the true value data of the lane line where the target vehicle is located in three-dimensional space based on the sampling data; an offset determination module, configured to fit the lane line according to the true value data, obtain the fitted lane line and sampling point coordinates, and determine the offset of each sampling point according to a preset offset determination method; a detection model construction module, configured to obtain camera parameters on the target vehicle, generate a camera parameter feature vector based on the camera parameters, and construct a three-dimensional lane detection network model based on the camera parameter feature vector and the offset of the sampling point; A detection data generation module is used to iteratively train the three-dimensional lane line detection network model until a preset model convergence condition is reached, thereby obtaining a target lane line detection network model. The target lane line detection network model is deployed on the target vehicle to obtain lane detection data of the lane line where the target vehicle is located, wherein the lane detection data includes the three-dimensional coordinates and curvature of the lane line.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images