3D lane line detection method and device

Abstract

The present application provides a 3D lane line detection method and device. The method includes: obtaining an image semantic feature matrix and a multidimensional depth feature matrix based on a road scene image and a detection model, converting the multidimensional depth feature matrix into depth probabilities at different depths of the camera's visual cone, and generating lane line BEV features based on the image semantic feature matrix and the depth probabilities at different depths of the camera's visual cone; then, supervising the lane line BEV features based on the road scene image and point cloud ground truth data and performing explicit depth supervision on the multidimensional depth feature matrix to update model parameters until model training is complete; in actual application, performing 3D lane line detection based on the 3D lane line detection model and real-time road scene images to obtain the lane line coordinates in the 3D coordinate system of the vehicle. The method and device can effectively identify 3D lane lines, have low computational complexity, high computational efficiency, and can adapt to complex and changing driving scenarios.

Description

Technical Field

[0001] The present application relates to the field of autonomous driving technology, and more specifically, to a 3D lane line detection method and device. Background Art

[0002] In recent years, lane detection has played a crucial role in autonomous driving technology, particularly in intelligent assisted driving systems. Accurate lane recognition is crucial for the implementation of key driver assistance features such as lane departure warning (LDW), blind spot monitoring (BSM), and adaptive cruise control (ACC). However, traditional 2D lane detection methods, due to a lack of depth information, suffer from reduced accuracy and insufficient robustness when projecting detection results into 3D space. To address this challenge, the research community has gradually shifted its focus to the 3D domain. Existing 3D lane detection methods typically first capture a lane image, then encode the image, map the lane features into a virtual top-down view, and predict lane points represented in this view. Finally, these lane points are converted to the ego vehicle coordinate system through geometric transformations to obtain 3D lane point position information. Despite this, current 3D lane detection methods still face numerous limitations in practice. They often struggle to adapt to increasingly complex and changing driving scenarios and suffer from computational efficiency bottlenecks, resulting in high computational complexity. Summary of the Invention

[0003] The purpose of the embodiments of the present application is to provide a 3D lane line detection method and device that can effectively identify 3D lane lines, has low computational complexity, high computational efficiency, and can adapt to complex and changing driving scenarios.

[0004] In a first aspect, the present application provides a 3D lane line detection method, comprising:

[0005] Build a detection model;

[0006] Obtain road scene images and point cloud ground truth data containing lane lines;

[0007] Obtaining an image semantic feature matrix and a multi-dimensional depth feature matrix according to the road scene image and the detection model;

[0008] Converting the multidimensional depth feature matrix into depth probabilities at different depths of the camera frustum;

[0009] Generate lane line BEV features according to the image semantic feature matrix and the depth probabilities of different depths of the camera cone;

[0010] Supervising the lane line BEV feature according to the road scene image and the point cloud true value data to obtain a first loss value;

[0011] Performing explicit depth supervision on the multi-dimensional depth feature matrix according to the point cloud true value data to obtain a second loss value;

[0012] Updating the model parameters of the detection model according to the first loss value and the second loss value to obtain an updated detection model;

[0013] When the model training termination condition is reached, the final updated detection model is determined as the trained 3D lane line detection model;

[0014] Acquire a real-time road scene image, and perform 3D lane line detection based on the 3D lane line detection model and the real-time road scene image to obtain the lane line coordinates in the 3D coordinate system of the vehicle.

[0015] In the above implementation process, this method can effectively identify 3D lane lines with low computational complexity and high computational efficiency, and can adapt to complex and changing driving scenarios.

[0016] Furthermore, the obtaining of an image semantic feature matrix and a multi-dimensional depth feature matrix based on the road scene image and the detection model includes:

[0017] Extracting features from the road scene image using the MobileNet feature extraction network of the detection model to obtain initial image features;

[0018] Performing depth image coordinate conversion on the point cloud true value data to obtain a depth true value map;

[0019] The FPN feature fusion network of the detection model is used to extract deep features from the initial image features to obtain an image semantic feature matrix and a multi-dimensional deep feature matrix.

[0020] Furthermore, generating lane line BEV features according to the image semantic feature matrix and the depth probabilities at different depths of the camera viewing cone includes:

[0021] Performing an outer product operation on the depth probabilities at different depths of the camera frustum and the image semantic feature matrix to obtain a feature matrix;

[0022] Mapping the feature matrix to position points corresponding to depth probabilities at different depths of the camera viewing cone to obtain a first feature point cloud;

[0023] Convert the first feature point cloud to the vehicle's 3D coordinate system to obtain a second feature point cloud;

[0024] The second feature point cloud is converted from the ego-vehicle 3D coordinate system to the BEV grid coordinate system to obtain the lane line BEV feature.

[0025] Furthermore, the lane line BEV feature is supervised according to the road scene image and the point cloud true value data to obtain a first loss value, including:

[0026] Performing true value annotation in the BEV space according to the road scene image and the point cloud true value data to obtain the BEV true value;

[0027] Encoding the lane line BEV feature through a pre-built ResNet network to obtain an encoded lane line feature;

[0028] Detecting the encoded lane line features using a pre-built BEV feature detection head to obtain lane line features in a BEV space;

[0029] A first loss value is calculated according to a preset FocalLoss loss function, the BEV true value, and the lane line features in the BEV space.

[0030] Furthermore, performing explicit deep supervision on the multi-dimensional depth feature matrix according to the point cloud true value data to obtain a second loss value includes:

[0031] Performing true value annotation on the point cloud true value data to obtain a depth true value map;

[0032] A second loss value is calculated based on the depth truth map, a preset binary cross entropy loss function, and the multidimensional depth feature matrix.

[0033] Furthermore, the method further comprises:

[0034] Determine whether the model training termination condition has been reached;

[0035] If yes, execute the above-mentioned method to determine the final updated detection model as the trained 3D lane line detection model;

[0036] If not, execute the method of obtaining an image semantic feature matrix and a multi-dimensional depth feature matrix based on the road scene picture and the detection model.

[0037] Furthermore, performing 3D lane line detection based on the 3D lane line detection model and the real-time road scene image to obtain lane line coordinates in the 3D coordinate system of the vehicle includes:

[0038] generating a real-time BEV feature vector based on the 3D lane detection model and the real-time road scene image;

[0039] Performing lane line decoding processing on the real-time BEV feature vector to obtain lane line BEV grid coordinates;

[0040] The lane line BEV grid coordinates are converted to the 3D coordinate system of the vehicle to obtain the lane line coordinates in the 3D coordinate system of the vehicle.

[0041] A second aspect of the present application provides a 3D lane line detection device, the 3D lane line detection device comprising:

[0042] A construction unit for constructing a detection model;

[0043] A first acquisition unit is used to acquire a road scene image containing lane lines and point cloud true value data;

[0044] A second acquisition unit is configured to acquire an image semantic feature matrix and a multi-dimensional depth feature matrix based on the road scene image and the detection model;

[0045] A conversion unit, configured to convert the multidimensional depth feature matrix into depth probabilities at different depths of a camera viewing cone;

[0046] A second generating unit is configured to generate a lane line BEV feature according to the image semantic feature matrix and the depth probabilities at different depths of the camera viewing cone;

[0047] A first supervision unit is configured to supervise the lane line BEV feature according to the road scene image and the point cloud true value data to obtain a first loss value;

[0048] A second supervision unit is configured to perform explicit depth supervision on the multi-dimensional depth feature matrix according to the point cloud true value data to obtain a second loss value;

[0049] an updating unit, configured to update model parameters of the detection model according to the first loss value and the second loss value to obtain an updated detection model;

[0050] A determination unit, configured to determine the final updated detection model as the trained 3D lane line detection model when a model training termination condition is reached;

[0051] A third acquisition unit is used to acquire real-time road scene images;

[0052] The lane line detection unit is used to perform 3D lane line detection based on the 3D lane line detection model and the real-time road scene image to obtain the lane line coordinates in the 3D coordinate system of the vehicle.

[0053] Furthermore, the second acquiring unit includes:

[0054] an extraction subunit, configured to extract features from the road scene image using the MobileNet feature extraction network of the detection model to obtain initial image features;

[0055] A first conversion subunit is configured to perform depth image coordinate conversion on the point cloud true value data to obtain a depth true value image;

[0056] The extraction subunit is further used to perform deep feature extraction on the initial image features according to the FPN feature fusion network of the detection model to obtain an image semantic feature matrix and a multi-dimensional deep feature matrix.

[0057] Furthermore, the second generating unit includes:

[0058] an outer product subunit, configured to perform an outer product operation on the depth probabilities at different depths of the camera frustum and the image semantic feature matrix to obtain a feature matrix;

[0059] A mapping subunit, configured to map the feature matrix to position points corresponding to depth probabilities at different depths of the camera viewing cone to obtain a first feature point cloud;

[0060] A second conversion subunit is used to convert the first feature point cloud into a 3D coordinate system of the vehicle to obtain a second feature point cloud;

[0061] The second conversion subunit is further used to convert the second feature point cloud from the vehicle 3D coordinate system to the BEV grid coordinate system to obtain the lane line BEV feature.

[0062] Furthermore, the first supervision unit includes:

[0063] A first annotation subunit is configured to perform true value annotation in a BEV space based on the road scene image and the point cloud true value data to obtain a BEV true value;

[0064] An encoding subunit, configured to encode the lane line BEV feature through a pre-built ResNet network to obtain an encoded lane line feature;

[0065] A detection subunit, configured to detect the encoded lane line features using a pre-built BEV feature detection head to obtain lane line features in a BEV space;

[0066] The first calculation subunit is used to calculate a first loss value according to a preset FocalLoss loss function, the BEV true value, and the lane line feature in the BEV space.

[0067] Furthermore, the second supervision unit includes:

[0068] A second annotation subunit is used to perform true value annotation on the point cloud true value data to obtain a depth true value map;

[0069] The second calculation subunit is used to calculate a second loss value according to the depth truth map, a preset binary cross entropy loss function and the multidimensional depth feature matrix.

[0070] Furthermore, the 3D lane line detection device further includes:

[0071] A judgment unit is used to judge whether the model training termination condition is currently met;

[0072] The determining unit is specifically configured to determine the finally updated detection model as the trained 3D lane line detection model when the model training termination condition is met;

[0073] The second acquisition unit is further used to obtain an image semantic feature matrix and a multi-dimensional depth feature matrix based on the road scene image and the detection model when the model training termination condition is not met.

[0074] Furthermore, the lane line detection unit includes:

[0075] A generating subunit, configured to generate a real-time BEV feature vector based on the 3D lane line detection model and the real-time road scene image;

[0076] A decoding subunit, configured to perform lane line decoding processing on the real-time BEV feature vector to obtain lane line BEV grid coordinates;

[0077] The third conversion subunit is used to convert the lane line BEV grid coordinates into the 3D coordinate system of the vehicle to obtain the lane line coordinates in the 3D coordinate system of the vehicle.

[0078] In a third aspect, the present application provides an electronic device, including a memory and a processor, wherein the memory is used to store a computer program, and the processor runs the computer program to enable the electronic device to perform the 3D lane line detection method described in any one of the first aspects of the present application.

[0079] A fourth aspect of the present application provides a computer-readable storage medium storing computer program instructions. When the computer program instructions are read and executed by a processor, the 3D lane line detection method described in any one of the first aspects of the present application is executed.

[0080] The beneficial effects of the present application are as follows: the method and device can detect 3D lane lines based on BEV feature supervision and deep explicit supervision, thereby achieving effective recognition of 3D lane lines;

[0081] In addition, the method and device can also transform features from image-based view projection to BEV view by utilizing the camera's internal and external parameters, and establish implicit depth supervision by learning BEV view features. At the same time, the true value depth of the point cloud is used to perform explicit joint supervision on the depth features, thereby effectively improving the accuracy of the model's lane line depth prediction, and thus promoting the final model's predicted lane line foreground point distribution to be closer to the true value.

[0082] Finally, the method and device can fuse the lane line predictions of all cameras into a unified representation of a scene, thereby improving the comprehensiveness and accuracy of lane line recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0083] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments of the present application. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other relevant drawings can be obtained based on these drawings without creative work.

[0084] Figure 1 A schematic diagram of a 3D lane detection method provided in an embodiment of the present application;

[0085] Figure 2 A schematic diagram of a flow chart of another 3D lane detection method provided in an embodiment of the present application;

[0086] Figure 3 A schematic diagram illustrating an example flow of a 3D lane detection method provided in an embodiment of the present application;

[0087] Figure 4 A schematic diagram of the structure of a 3D lane detection device provided in an embodiment of the present application;

[0088] Figure 5 A schematic diagram of the structure of another 3D lane detection device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0089] The technical solutions in the embodiments of the present application will be described below in conjunction with the drawings in the embodiments of the present application.

[0090] It should be noted that similar reference numerals and letters represent similar items in the following drawings. Therefore, once an item is defined in one drawing, it does not need to be further defined or explained in subsequent drawings. At the same time, in the description of this application, the terms "first", "second", etc. are only used to distinguish the description and should not be understood as indicating or implying relative importance.

[0091] Example 1

[0092] Please see Figure 1 , Figure 1 The following is a flow chart of a 3D lane line detection method provided in this embodiment. The 3D lane line detection method includes:

[0093] S101. Build a detection model.

[0094] S102: Obtain a road scene image containing lane lines and point cloud true value data.

[0095] S103. Obtain an image semantic feature matrix and a multi-dimensional depth feature matrix based on the road scene image and the detection model.

[0096] S104: Convert the multi-dimensional depth feature matrix into depth probabilities at different depths of the camera frustum.

[0097] S105: Generate lane line BEV features based on the image semantic feature matrix and the depth probabilities at different depths of the camera frustum.

[0098] S106 : Supervise the lane line BEV feature based on the road scene image and the point cloud true value data to obtain a first loss value.

[0099] S107. Perform explicit deep supervision on the multi-dimensional deep feature matrix based on the true value data of the point cloud to obtain a second loss value.

[0100] S108. Update the model parameters of the detection model according to the first loss value and the second loss value to obtain an updated detection model.

[0101] S109: When the model training termination condition is reached, the finally updated detection model is determined as the trained 3D lane line detection model.

[0102] S110: Acquire a real-time road scene image, and perform 3D lane line detection based on the 3D lane line detection model and the real-time road scene image to obtain the lane line coordinates in the 3D coordinate system of the vehicle.

[0103] In this embodiment, the 3D lane line detection method is specifically a 3D lane line detection method based on BEV feature supervision and explicit depth joint supervision scheme.

[0104] In this embodiment, the execution subject of the method may be a computing device such as a computer or a server, and this is not limited in this embodiment.

[0105] In this embodiment, the execution subject of the method may also be a smart device such as a smart phone, a tablet computer, etc., which is not limited in this embodiment.

[0106] It can be seen that the 3D lane line detection method described in this embodiment can effectively identify 3D lane lines, has low computational complexity and high computational efficiency, and can adapt to complex and changing driving scenarios.

[0107] Example 2

[0108] Please see Figure 2 , Figure 2 The following is a flow chart of a 3D lane line detection method provided in this embodiment. The 3D lane line detection method includes:

[0109] S201: Build a detection model.

[0110] S202: Obtain a road scene image containing lane lines and point cloud true value data.

[0111] In this embodiment, the method can use 6 cameras (front, left front, right front, left rear, right rear, and rear) and 1 laser radar installed on the vehicle to collect road scene images containing lane lines and laser point cloud true value data.

[0112] In this embodiment, the method can label lane line points in the point cloud ground truth data to obtain training and test data.

[0113] S203: Extract features from the road scene image using the MobileNet feature extraction network of the detection model to obtain initial image features.

[0114] S204: Perform depth image coordinate conversion on the point cloud true value data to obtain a depth true value map.

[0115] In this embodiment, the method can use the surround view images from the perspectives of the six cameras on the vehicle as input X k ∈R 3 ×H×W , k=0,1,…,5; at the same time, the corresponding camera intrinsic parameter is determined to be E k ∈3×4, external parameter is I k ∈3×3.

[0116] S205. Perform deep feature extraction on the initial image features according to the FPN feature fusion network of the detection model to obtain an image semantic feature matrix and a multi-dimensional deep feature matrix.

[0117] In this embodiment, the method can use MobileNet as the backbone network of the network structure to extract features from the image inputs under the six camera perspectives respectively, and then use the FPN network to fuse the features to obtain an image semantic feature matrix and a D-dimensional depth feature matrix.

[0118] S206: Convert the multi-dimensional depth feature matrix into depth probabilities at different depths of the camera frustum.

[0119] In this embodiment, the method can generate depth probabilities of different depths of the camera's viewing cone based on a 2D image.

[0120] In this embodiment, the method can define D discrete depth value distributions for each pixel in the input image (i.e., in the depth direction of the image, starting from 4 meters from the camera plane, taking a depth value every 1 meter, up to a range of (D+4) meters). Each 2D image can then generate a depth probability for different depths of the camera frustum, where each point can be described by P(u, v, d), i.e., the coordinates and corresponding depth value in the image coordinate system. The depth probability of different depths of the camera frustum is a spatial position matrix obtained based on the camera projection principle and does not contain image feature information.

[0121] In this embodiment, the method can also use the camera internal and external parameters to convert the depth probability of the point P (u, v, d) at different depths of the camera cone to the coordinate P' (x e ,y e , z e ).

[0122] In this embodiment, the method may use a softmax function to convert D-dimensional depth features into depth probability information, that is, each feature map point corresponds to D depth probability values.

[0123] S207 , performing an outer product operation on the depth probabilities at different depths of the camera frustum and the image semantic feature matrix to obtain a feature matrix.

[0124] In this embodiment, the method may perform an outer product operation on the depth probability and the image semantic feature matrix to obtain a feature matrix that is assigned D depth probability information for each image semantic feature point.

[0125] S208 , mapping the feature matrix to position points corresponding to depth probabilities at different depths of the camera's viewing cone to obtain a first feature point cloud.

[0126] In this embodiment, the given BEV grid is a three-dimensional space with a length of 50 meters in front, behind, left, and right of the vehicle and no height limit, and is composed of 200×200 BEV grids.

[0127] In this embodiment, the method maps the image features endowed with depth probability information obtained in the above steps to the corresponding position points of depth probabilities at different depths of the camera viewing cone obtained in the above steps according to the camera projection space relationship to obtain a first feature point cloud.

[0128] S209: Convert the first feature point cloud to the vehicle 3D coordinate system to obtain a second feature point cloud.

[0129] In this embodiment, the method converts the first feature point cloud into the 3D coordinate system of the vehicle to obtain the second feature point cloud.

[0130] S210 : Convert the second feature point cloud from the ego-vehicle 3D coordinate system to the BEV grid coordinate system to obtain the lane line BEV feature.

[0131] In this embodiment, the method converts the ego-vehicle 3D coordinates to BEV grid coordinates and performs a height elimination operation to remove points outside the space represented by the BEV grid. The remaining valid 3D feature point cloud is then assigned to each BEV grid according to its coordinate position. Then, for each grid, the method performs a cumulative summation of all features belonging to the same grid to obtain the lane line BEV feature.

[0132] S211: Supervise the lane line BEV feature based on the road scene image and point cloud true value data to obtain a first loss value.

[0133] As an optional implementation, the lane line BEV feature is supervised based on the road scene image and point cloud ground truth data to obtain a first loss value, including:

[0134] Based on the road scene images and point cloud truth data, perform true value annotation in the BEV space to obtain the BEV true value;

[0135] The lane line BEV features are encoded through the pre-built ResNet network to obtain the encoded lane line features;

[0136] The encoded lane line features are detected by the pre-built BEV feature detection head to obtain the lane line features in the BEV space;

[0137] The first loss value is calculated based on the preset FocalLoss loss function, the BEV true value, and the lane line features in the BEV space.

[0138] In this embodiment, the method can use ResNet to encode and extract BEV features, and then use the BEV feature detection head to detect lane line features in the BEV space.

[0139] In this embodiment, the BEV feature detection head is a convolutional network. The detection head outputs the lane line foreground feature distribution state in the BEV space (i.e., the lane line features under the BEV space), and uses FocalLoss to supervise the lane line foreground and background distribution state of the BEV feature map.

[0140] In this embodiment, the feedback of the feature detection loss corresponding to the detection head prompts the middle layer of the network to learn the depth of the lane line foreground point, thereby establishing an implicit lane line foreground point depth supervision.

[0141] S212. Based on the true value data of the point cloud, perform explicit deep supervision on the multi-dimensional deep feature matrix to obtain a second loss value.

[0142] As an optional implementation, explicit deep supervision is performed on the multi-dimensional deep feature matrix based on the true point cloud data to obtain a second loss value, including:

[0143] Perform true value annotation on the point cloud data to obtain the depth truth map;

[0144] The second loss value is calculated based on the depth truth map, the preset binary cross entropy loss function and the multi-dimensional depth feature matrix.

[0145] In this embodiment, regarding the deep supervision:

[0146] This method uses the true depth of the point cloud to jointly supervise the depth features, which will make the distribution of lane line foreground points predicted by the final model closer to the true value.

[0147] Specifically, this method first uses the lane line point cloud truth value to transform the depth image coordinates to obtain the depth truth map: the lane line point cloud truth value P(x e ,y e , z e ) is converted to the k-th camera view image coordinate system with depth through the internal and external parameter matrix of the k-th camera, and P is obtained k (u, v, d), where u and v represent pixel coordinates and d represents depth. k If (u, v, d) does not fall on the kth view, the coordinate is discarded. For the depth ground truth map, if there is a true depth d conversion result at a certain coordinate (u, v), the probability value at depth d is filled with 1, otherwise the true depth probability is 0.

[0148] Then, based on the depth ground truth map, Binary Cross Entropy Loss is used to perform visual supervision on the depth features extracted from the lane line BEV features. This supervision process requires masking of non-lane line foreground points.

[0149] S213: Update the model parameters of the detection model according to the first loss value and the second loss value to obtain an updated detection model.

[0150] S214: Determine whether the model training termination condition is currently met. If so, execute step S215; if not, execute step S203.

[0151] S215: Determine the finally updated detection model as the trained 3D lane line detection model.

[0152] S216 , obtaining a real-time road scene image, and generating a real-time BEV feature vector based on the 3D lane line detection model and the real-time road scene image.

[0153] S217 , performing lane line decoding processing on the real-time BEV feature vector to obtain lane line BEV grid coordinates.

[0154] S218 : Convert the lane line BEV grid coordinates into the ego vehicle 3D coordinate system to obtain the lane line coordinates in the ego vehicle 3D coordinate system.

[0155] In this embodiment, the 3D lane detection model determines lane points in the feature map output by the BEV feature detection head during the BEV feature supervision and explicit depth joint supervision detection process. These points have eigenvalues ​​greater than a threshold γ, thereby obtaining the corresponding BEV grid coordinates. These coordinates are then converted to the ego-vehicle 3D coordinate system to obtain the lane coordinates in the ego-vehicle 3D coordinate system. This process is known as the BEV lane decoding process.

[0156] Please see Figure 3 , Figure 3 A schematic diagram of an example flow chart of a 3D lane line detection method is shown.

[0157] In this embodiment, the execution subject of the method may be a computing device such as a computer or a server, and this is not limited in this embodiment.

[0158] In this embodiment, the execution subject of the method may also be a smart device such as a smart phone, a tablet computer, etc., which is not limited in this embodiment.

[0159] It can be seen that the 3D lane line detection method described in this embodiment can effectively identify 3D lane lines, has low computational complexity and high computational efficiency, and can adapt to complex and changing driving scenarios.

[0160] Example 3

[0161] Please see Figure 4 , Figure 4 This is a schematic diagram of the structure of a 3D lane line detection device provided in this embodiment. Figure 4 As shown, the 3D lane line detection device includes:

[0162] A construction unit 310 is used to construct a detection model;

[0163] A first acquisition unit 320 is configured to acquire a road scene image including lane lines and point cloud ground truth data;

[0164] A second acquisition unit 330 is configured to acquire an image semantic feature matrix and a multi-dimensional depth feature matrix based on the road scene image and the detection model;

[0165] A conversion unit 340 is used to convert the multi-dimensional depth feature matrix into depth probabilities at different depths of the camera frustum;

[0166] The second generating unit 350 is used to generate lane line BEV features according to the image semantic feature matrix and the depth probabilities of different depths of the camera frustum;

[0167] A first supervision unit 360 is configured to supervise the lane line BEV feature based on the road scene image and the point cloud ground truth data to obtain a first loss value;

[0168] A second supervision unit 370 is configured to perform explicit depth supervision on the multi-dimensional depth feature matrix based on the point cloud ground truth data to obtain a second loss value;

[0169] An updating unit 380, configured to update model parameters of the detection model according to the first loss value and the second loss value to obtain an updated detection model;

[0170] a determination unit 390 for determining the finally updated detection model as the trained 3D lane detection model when a model training termination condition is met;

[0171] The third acquisition unit 400 is used to acquire a real-time road scene image;

[0172] The lane line detection unit 410 is used to perform 3D lane line detection based on the 3D lane line detection model and the real-time road scene image to obtain the lane line coordinates in the 3D coordinate system of the vehicle.

[0173] In this embodiment, the explanation of the 3D lane line detection device can refer to the description in Example 1 or Example 2, and will not be further elaborated in this embodiment.

[0174] It can be seen that the 3D lane line detection device described in this embodiment can effectively identify 3D lane lines, has low computational complexity and high computational efficiency, and can adapt to complex and changing driving scenarios.

[0175] Example 4

[0176] Please see Figure 5 , Figure 5 This is a schematic diagram of the structure of a 3D lane line detection device provided in this embodiment. Figure 5 As shown, the 3D lane line detection device includes:

[0177] A construction unit 310 is used to construct a detection model;

[0178] A first acquisition unit 320 is configured to acquire a road scene image including lane lines and point cloud ground truth data;

[0179] A second acquisition unit 330 is configured to acquire an image semantic feature matrix and a multi-dimensional depth feature matrix based on the road scene image and the detection model;

[0180] A conversion unit 340 is used to convert the multi-dimensional depth feature matrix into depth probabilities at different depths of the camera frustum;

[0181] The second generating unit 350 is used to generate lane line BEV features according to the image semantic feature matrix and the depth probabilities of different depths of the camera frustum;

[0182] A first supervision unit 360 is configured to supervise the lane line BEV feature based on the road scene image and the point cloud ground truth data to obtain a first loss value;

[0183] A second supervision unit 370 is configured to perform explicit depth supervision on the multi-dimensional depth feature matrix based on the point cloud ground truth data to obtain a second loss value;

[0184] An updating unit 380, configured to update model parameters of the detection model according to the first loss value and the second loss value to obtain an updated detection model;

[0185] a determination unit 390 for determining the finally updated detection model as the trained 3D lane detection model when a model training termination condition is met;

[0186] The third acquisition unit 400 is used to acquire a real-time road scene image;

[0187] The lane line detection unit 410 is used to perform 3D lane line detection based on the 3D lane line detection model and the real-time road scene image to obtain the lane line coordinates in the 3D coordinate system of the vehicle.

[0188] As an optional implementation, the second acquiring unit 330 includes:

[0189] An extraction subunit 331 is used to extract features from the road scene image using the MobileNet feature extraction network of the detection model to obtain initial image features;

[0190] The first conversion subunit 332 is used to perform depth image coordinate conversion on the point cloud truth data to obtain a depth truth map;

[0191] The extraction subunit 331 is further used to perform deep feature extraction on the initial image features according to the FPN feature fusion network of the detection model to obtain an image semantic feature matrix and a multi-dimensional deep feature matrix.

[0192] As an optional implementation manner, the second generating unit 350 includes:

[0193] The outer product subunit 351 is used to perform an outer product operation on the depth probabilities at different depths of the camera frustum and the image semantic feature matrix to obtain a feature matrix;

[0194] A mapping subunit 352 is configured to map the feature matrix to position points corresponding to depth probabilities at different depths of the camera's viewing cone to obtain a first feature point cloud;

[0195] The second conversion subunit 353 is used to convert the first feature point cloud into the 3D coordinate system of the vehicle to obtain a second feature point cloud;

[0196] The second conversion subunit 353 is further used to convert the second feature point cloud from the ego-vehicle 3D coordinate system to the BEV grid coordinate system to obtain the lane line BEV feature.

[0197] As an optional implementation, the first supervision unit 360 includes:

[0198] The first annotation subunit 361 is used to perform true value annotation in the BEV space based on the road scene image and the point cloud true value data to obtain the BEV true value;

[0199] The encoding subunit 362 is used to encode the lane line BEV feature through a pre-built ResNet network to obtain the encoded lane line feature;

[0200] The detection subunit 363 is configured to detect the encoded lane features using a pre-built BEV feature detection head to obtain lane features in the BEV space;

[0201] The first calculation subunit 364 is used to calculate a first loss value according to a preset FocalLoss loss function, a BEV true value, and lane line features in the BEV space.

[0202] As an optional implementation, the second supervision unit 370 includes:

[0203] The second annotation subunit 371 is used to perform true value annotation on the point cloud true value data to obtain a depth true value map;

[0204] The second calculation subunit 372 is used to calculate a second loss value according to the depth truth map, a preset binary cross entropy loss function and a multi-dimensional depth feature matrix.

[0205] As an optional implementation, the 3D lane line detection device further includes:

[0206] A judgment unit 420 is used to judge whether the model training termination condition is currently met;

[0207] A determination unit 390 is specifically configured to determine the final updated detection model as the trained 3D lane detection model when a model training termination condition is met;

[0208] The second acquisition unit 330 is further configured to acquire an image semantic feature matrix and a multi-dimensional depth feature matrix based on the road scene image and the detection model when the model training termination condition is not met.

[0209] As an optional implementation, the lane line detection unit 410 includes:

[0210] A generating subunit 411 is configured to generate a real-time BEV feature vector based on the 3D lane detection model and the real-time road scene image;

[0211] The decoding subunit 412 is used to perform lane line decoding processing on the real-time BEV feature vector to obtain the lane line BEV grid coordinates;

[0212] The third conversion subunit 413 is used to convert the lane line BEV grid coordinates into the ego vehicle 3D coordinate system to obtain the lane line coordinates in the ego vehicle 3D coordinate system.

[0213] In this embodiment, the explanation of the 3D lane line detection device can refer to the description in Example 1 or Example 2, and will not be further elaborated in this embodiment.

[0214] It can be seen that the 3D lane line detection device described in this embodiment can effectively identify 3D lane lines, has low computational complexity and high computational efficiency, and can adapt to complex and changing driving scenarios.

[0215] An embodiment of the present application provides an electronic device, including a memory and a processor, wherein the memory is used to store a computer program, and the processor runs the computer program to enable the electronic device to perform the 3D lane line detection method in embodiment 1 or embodiment 2 of the present application.

[0216] An embodiment of the present application provides a computer-readable storage medium storing computer program instructions. When the computer program instructions are read and executed by a processor, the 3D lane line detection method in embodiment 1 or embodiment 2 of the present application is executed.

[0217] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can also be implemented in other ways. The device embodiments described above are merely illustrative. For example, the flowcharts and block diagrams in the accompanying drawings show the possible architectures, functions and operations of the devices, methods and computer program products according to the multiple embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of the code, and the module, program segment or a part of the code contains one or more executable instructions for implementing the specified logical functions. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in an order different from that marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart, can be implemented using a dedicated hardware-based system that performs the specified function or action, or can be implemented using a combination of dedicated hardware and computer instructions.

[0218] In addition, the functional modules in each embodiment of the present application can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.

[0219] If the functions are implemented in the form of software function modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0220] The foregoing is merely an embodiment of the present application and is not intended to limit the scope of protection of the present application. Various modifications and variations are possible for those skilled in the art. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application shall be included within the scope of protection of the present application. It should be noted that similar reference numerals and letters represent similar items in the following figures. Therefore, once an item is defined in one figure, it does not need to be further defined or explained in subsequent figures.

[0221] The above description is merely a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in the present application should be included in the scope of protection of the present application. Therefore, the scope of protection of the present application should be based on the scope of protection of the claims.

[0222] It should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply the existence of any such actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or device comprising the element.

Claims

1. A 3D lane line detection method, characterized in that: include: Build a detection model; Obtain road scene images and point cloud ground truth data containing lane lines; Obtaining an image semantic feature matrix and a multi-dimensional depth feature matrix according to the road scene image and the detection model; Converting the multidimensional depth feature matrix into depth probabilities at different depths of the camera frustum; Generate lane line BEV features according to the image semantic feature matrix and the depth probabilities of different depths of the camera cone; Supervising the lane line BEV feature according to the road scene image and the point cloud true value data to obtain a first loss value; Performing explicit depth supervision on the multi-dimensional depth feature matrix according to the point cloud true value data to obtain a second loss value; Updating the model parameters of the detection model according to the first loss value and the second loss value to obtain an updated detection model; When the model training termination condition is reached, the final updated detection model is determined as the trained 3D lane line detection model; Acquire a real-time road scene image, and perform 3D lane line detection based on the 3D lane line detection model and the real-time road scene image to obtain the lane line coordinates in the 3D coordinate system of the vehicle.

2. The 3D lane line detection method according to claim 1, characterized in that: The step of obtaining an image semantic feature matrix and a multi-dimensional depth feature matrix based on the road scene image and the detection model includes: Extracting features from the road scene image using the MobileNet feature extraction network of the detection model to obtain initial image features; Performing depth image coordinate conversion on the point cloud true value data to obtain a depth true value map; The FPN feature fusion network of the detection model is used to extract deep features from the initial image features to obtain an image semantic feature matrix and a multi-dimensional deep feature matrix.

3. The 3D lane line detection method according to claim 1, characterized in that: Generating lane line BEV features according to the image semantic feature matrix and the depth probabilities at different depths of the camera frustum includes: Performing an outer product operation on the depth probabilities at different depths of the camera frustum and the image semantic feature matrix to obtain a feature matrix; Mapping the feature matrix to position points corresponding to depth probabilities at different depths of the camera viewing cone to obtain a first feature point cloud; Convert the first feature point cloud to the vehicle's 3D coordinate system to obtain a second feature point cloud; The second feature point cloud is converted from the ego-vehicle 3D coordinate system to the BEV grid coordinate system to obtain the lane line BEV feature.

4. The 3D lane line detection method according to claim 1, characterized in that: The supervising the lane line BEV feature according to the road scene image and the point cloud true value data to obtain a first loss value includes: Performing true value annotation in the BEV space according to the road scene image and the point cloud true value data to obtain the BEV true value; Encoding the lane line BEV feature through a pre-built ResNet network to obtain an encoded lane line feature; Detecting the encoded lane line features using a pre-built BEV feature detection head to obtain lane line features in a BEV space; A first loss value is calculated according to a preset FocalLoss loss function, the BEV true value, and the lane line features in the BEV space.

5. The 3D lane line detection method according to claim 1, characterized in that: The step of performing explicit depth supervision on the multi-dimensional depth feature matrix according to the point cloud true value data to obtain a second loss value includes: Performing true value annotation on the point cloud true value data to obtain a depth true value map; A second loss value is calculated based on the depth truth map, a preset binary cross entropy loss function, and the multidimensional depth feature matrix.

6. The 3D lane line detection method according to claim 1, characterized in that: The method further comprises: Determine whether the model training termination condition has been reached; If yes, execute the above-mentioned method to determine the final updated detection model as the trained 3D lane line detection model; If not, execute the method of obtaining an image semantic feature matrix and a multi-dimensional depth feature matrix based on the road scene picture and the detection model.

7. The 3D lane line detection method according to claim 1, characterized in that: The performing 3D lane line detection based on the 3D lane line detection model and the real-time road scene image to obtain lane line coordinates in the 3D coordinate system of the vehicle includes: generating a real-time BEV feature vector based on the 3D lane detection model and the real-time road scene image; Performing lane line decoding processing on the real-time BEV feature vector to obtain lane line BEV grid coordinates; The lane line BEV grid coordinates are converted to the 3D coordinate system of the vehicle to obtain the lane line coordinates in the 3D coordinate system of the vehicle.

8. A 3D lane line detection device, characterized in that: The 3D lane line detection device includes: A construction unit for constructing a detection model; A first acquisition unit is used to acquire a road scene image containing lane lines and point cloud true value data; A second acquisition unit is configured to acquire an image semantic feature matrix and a multi-dimensional depth feature matrix based on the road scene image and the detection model; A conversion unit, configured to convert the multidimensional depth feature matrix into depth probabilities at different depths of a camera viewing cone; A second generating unit is configured to generate a lane line BEV feature according to the image semantic feature matrix and the depth probabilities at different depths of the camera viewing cone; A first supervision unit is configured to supervise the lane line BEV feature according to the road scene image and the point cloud true value data to obtain a first loss value; A second supervision unit is configured to perform explicit depth supervision on the multi-dimensional depth feature matrix according to the point cloud true value data to obtain a second loss value; an updating unit, configured to update model parameters of the detection model according to the first loss value and the second loss value to obtain an updated detection model; A determination unit, configured to determine the final updated detection model as the trained 3D lane line detection model when a model training termination condition is reached; A third acquisition unit is used to acquire real-time road scene images; The lane line detection unit is used to perform 3D lane line detection based on the 3D lane line detection model and the real-time road scene image to obtain the lane line coordinates in the 3D coordinate system of the vehicle.

9. An electronic device, characterized in that: The electronic device includes a memory and a processor, the memory is used to store a computer program, and the processor runs the computer program to enable the electronic device to perform the 3D lane line detection method according to any one of claims 1 to 7.

10. A readable storage medium, characterized in that: The readable storage medium stores computer program instructions, and when the computer program instructions are read and executed by a processor, the 3D lane line detection method according to any one of claims 1 to 7 is executed.

Images