3D lane line detection method and device

Abstract

The invention provides a 3D lane line detection method and device, and the method comprises the steps: obtaining an image semantic feature matrix and a multi-dimensional depth feature matrix according to a road scene image and a detection model, and converting the multi-dimensional depth feature matrix into depth probabilities of different depths of a camera view cone, lane line BEV features are generated based on the image semantic feature matrix and depth probabilities of different depths of a camera cone; then, according to a road scene picture and point cloud truth value data, lane line BEV features are supervised, and dominant depth supervision is performed on a multi-dimensional depth feature matrix, so that model parameters are updated until model training is completed; in practical application, 3D lane line detection is carried out according to the 3D lane line detection model and a real-time road scene picture, and lane line coordinates of a vehicle under a 3D coordinate system are obtained. According to the method and the device, the 3D lane line can be effectively identified, the calculation complexity is low, the calculation efficiency is high, and the method and the device can adapt to complex and changeable driving scenes.

Description

Technical Field

[0001] The present application relates to the field of autonomous driving technology, and more specifically, to a 3D lane line detection method and device. Background Art

[0002] In recent years, lane detection has played an important role in autonomous driving technology, especially in intelligent assisted driving systems. Accurate identification of lane lines is essential for the implementation of key driver assistance functions such as lane departure warning systems (LDW), blind spot monitoring systems (BSM), and adaptive cruise control systems (ACC). However, traditional 2D lane detection methods lack depth information and face the problem of reduced accuracy and insufficient robustness when projecting detection results into 3D space. To address this challenge, the research community has gradually shifted the focus of lane detection to the 3D field. Existing 3D lane detection methods usually capture lane line images first, then encode the images, map the features of the lane lines into a virtual top view, and predict the lane points represented by the view. Finally, these lane points are converted into the self-vehicle coordinate system through geometric transformation to obtain the position information of the 3D lane points. Despite this, current 3D lane detection methods still face many limitations in practice. They are often difficult to adapt to more complex and changeable driving scenarios, and there are bottlenecks in computational efficiency, resulting in high computational complexity. Summary of the invention

[0003] The purpose of the embodiments of the present application is to provide a 3D lane line detection method and device, which can effectively identify 3D lane lines, have low computational complexity, high computational efficiency, and can adapt to complex and changeable driving scenarios.

[0004] The first aspect of the present application provides a 3D lane line detection method, comprising:

[0005] Build a detection model;

[0006] Obtain road scene images and point cloud truth data containing lane lines;

[0007] According to the road scene picture and the detection model, an image semantic feature matrix and a multi-dimensional depth feature matrix are obtained;

[0008] Converting the multidimensional depth feature matrix into depth probabilities at different depths of the camera frustum;

[0009] Generate lane line BEV features according to the image semantic feature matrix and the depth probabilities of different depths of the camera cone;

[0010] Supervising the lane line BEV feature according to the road scene image and the point cloud true value data to obtain a first loss value;

[0011] According to the point cloud true value data, performing explicit deep supervision on the multi-dimensional deep feature matrix to obtain a second loss value;

[0012] Updating the model parameters of the detection model according to the first loss value and the second loss value to obtain an updated detection model;

[0013] When the model training termination condition is reached, the finally updated detection model is determined as the trained 3D lane line detection model;

[0014] A real-time road scene image is obtained, and 3D lane line detection is performed according to the 3D lane line detection model and the real-time road scene image to obtain the lane line coordinates in the 3D coordinate system of the vehicle.

[0015] In the above implementation process, the method can effectively identify 3D lane lines, has low computational complexity and high computational efficiency, and can adapt to complex and changeable driving scenarios.

[0016] Furthermore, the obtaining of an image semantic feature matrix and a multi-dimensional depth feature matrix according to the road scene picture and the detection model includes:

[0017] Extracting features from the road scene image using the MobileNet feature extraction network of the detection model to obtain initial image features;

[0018] Performing depth image coordinate conversion on the point cloud true value data to obtain a depth true value map;

[0019] The initial image features are subjected to deep feature extraction according to the FPN feature fusion network of the detection model to obtain an image semantic feature matrix and a multi-dimensional deep feature matrix.

[0020] Furthermore, generating lane line BEV features according to the image semantic feature matrix and the depth probabilities of different depths of the camera cone includes:

[0021] Performing an outer product operation on the depth probabilities at different depths of the camera cone and the image semantic feature matrix to obtain a feature matrix;

[0022] Mapping the feature matrix to position points corresponding to depth probabilities at different depths of the camera cone to obtain a first feature point cloud;

[0023] Convert the first feature point cloud to the 3D coordinate system of the vehicle to obtain a second feature point cloud;

[0024] The second feature point cloud is converted from the 3D coordinate system of the vehicle to the BEV grid coordinate system to obtain the lane line BEV feature.

[0025] Furthermore, the lane line BEV feature is supervised according to the road scene image and the point cloud true value data to obtain a first loss value, including:

[0026] According to the road scene image and the point cloud true value data, perform true value annotation in the BEV space to obtain the BEV true value;

[0027] Encoding the lane line BEV feature through a pre-built ResNet network to obtain an encoded lane line feature;

[0028] The encoded lane line features are detected by a pre-built BEV feature detection head to obtain lane line features in a BEV space;

[0029] A first loss value is calculated according to a preset FocalLoss loss function, the BEV true value, and the lane line features in the BEV space.

[0030] Furthermore, performing explicit deep supervision on the multi-dimensional deep feature matrix according to the point cloud true value data to obtain a second loss value includes:

[0031] Performing true value annotation on the point cloud true value data to obtain a depth true value map;

[0032] A second loss value is calculated according to the depth truth map, a preset binary cross entropy loss function and the multidimensional depth feature matrix.

[0033] Furthermore, the method further comprises:

[0034] Determine whether the model training termination condition is currently reached;

[0035] If yes, executing the above step of determining the finally updated detection model as the trained 3D lane line detection model;

[0036] If not, execute the steps of obtaining an image semantic feature matrix and a multi-dimensional depth feature matrix based on the road scene picture and the detection model.

[0037] Furthermore, the performing 3D lane line detection according to the 3D lane line detection model and the real-time road scene image to obtain the lane line coordinates in the 3D coordinate system of the vehicle includes:

[0038] Generate a real-time BEV feature vector according to the 3D lane line detection model and the real-time road scene picture;

[0039] Performing lane line decoding processing on the real-time BEV feature vector to obtain lane line BEV grid coordinates;

[0040] The lane line BEV grid coordinates are converted into the 3D coordinate system of the vehicle to obtain the lane line coordinates in the 3D coordinate system of the vehicle.

[0041] A second aspect of the present application provides a 3D lane line detection device, the 3D lane line detection device comprising:

[0042] A construction unit, used to construct a detection model;

[0043] A first acquisition unit is used to acquire a road scene image including lane lines and point cloud true value data;

[0044] A second acquisition unit is used to acquire an image semantic feature matrix and a multi-dimensional depth feature matrix according to the road scene picture and the detection model;

[0045] A conversion unit, used for converting the multi-dimensional depth feature matrix into depth probabilities at different depths of the camera cone;

[0046] A second generating unit is used to generate a lane line BEV feature according to the image semantic feature matrix and the depth probability of different depths of the camera cone;

[0047] A first supervision unit is used to supervise the lane line BEV feature according to the road scene image and the point cloud true value data to obtain a first loss value;

[0048] A second supervision unit is used to perform explicit deep supervision on the multi-dimensional deep feature matrix according to the point cloud true value data to obtain a second loss value;

[0049] an updating unit, configured to update a model parameter of the detection model according to the first loss value and the second loss value to obtain an updated detection model;

[0050] A determination unit, used for determining the finally updated detection model as the trained 3D lane line detection model when the model training termination condition is reached;

[0051] A third acquisition unit is used to acquire a real-time road scene picture;

[0052] The lane line detection unit is used to perform 3D lane line detection according to the 3D lane line detection model and the real-time road scene image to obtain the lane line coordinates in the 3D coordinate system of the vehicle.

[0053] Furthermore, the second acquiring unit includes:

[0054] An extraction subunit, configured to extract features from the road scene image through a MobileNet feature extraction network of the detection model to obtain initial image features;

[0055] A first conversion subunit is used to perform depth image coordinate conversion on the point cloud true value data to obtain a depth true value image;

[0056] The extraction subunit is also used to perform deep feature extraction on the initial image features according to the FPN feature fusion network of the detection model to obtain an image semantic feature matrix and a multi-dimensional deep feature matrix.

[0057] Furthermore, the second generating unit includes:

[0058] An outer product subunit, used for performing an outer product operation on the depth probabilities at different depths of the camera cone and the image semantic feature matrix to obtain a feature matrix;

[0059] A mapping subunit, used for mapping the feature matrix to position points corresponding to depth probabilities at different depths of the camera cone to obtain a first feature point cloud;

[0060] A second conversion subunit is used to convert the first feature point cloud into a 3D coordinate system of the vehicle to obtain a second feature point cloud;

[0061] The second conversion subunit is further used to convert the second feature point cloud from the vehicle 3D coordinate system to the BEV grid coordinate system to obtain the lane line BEV feature.

[0062] Furthermore, the first supervision unit includes:

[0063] A first annotation subunit is used to perform true value annotation in the BEV space according to the road scene image and the point cloud true value data to obtain a BEV true value;

[0064] An encoding subunit, used for encoding the lane line BEV feature through a pre-built ResNet network to obtain an encoded lane line feature;

[0065] A detection subunit, configured to detect the encoded lane line features through a pre-built BEV feature detection head to obtain lane line features in a BEV space;

[0066] The first calculation subunit is used to calculate a first loss value according to a preset FocalLoss loss function, the BEV true value and the lane line feature in the BEV space.

[0067] Furthermore, the second supervision unit includes:

[0068] A second annotation subunit is used to perform true value annotation on the point cloud true value data to obtain a depth true value map;

[0069] The second calculation subunit is used to calculate a second loss value according to the depth truth map, a preset binary cross entropy loss function and the multi-dimensional depth feature matrix.

[0070] Furthermore, the 3D lane line detection device further includes:

[0071] A judgment unit, used to judge whether the current model training termination condition is met;

[0072] The determining unit is specifically used to determine the finally updated detection model as the trained 3D lane line detection model when the model training termination condition is reached;

[0073] The second acquisition unit is also used to acquire an image semantic feature matrix and a multi-dimensional depth feature matrix according to the road scene image and the detection model when the model training termination condition is not met.

[0074] Furthermore, the lane line detection unit includes:

[0075] A generating subunit, configured to generate a real-time BEV feature vector according to the 3D lane line detection model and the real-time road scene picture;

[0076] A decoding subunit, used for performing lane line decoding processing on the real-time BEV feature vector to obtain lane line BEV grid coordinates;

[0077] The third conversion subunit is used to convert the lane line BEV grid coordinates into the 3D coordinate system of the vehicle to obtain the lane line coordinates in the 3D coordinate system of the vehicle.

[0078] A third aspect of the present application provides an electronic device, including a memory and a processor, wherein the memory is used to store a computer program, and the processor runs the computer program to enable the electronic device to perform the 3D lane line detection method described in any one of the first aspects of the present application.

[0079] A fourth aspect of the present application provides a computer-readable storage medium storing computer program instructions, wherein when the computer program instructions are read and executed by a processor, the 3D lane line detection method described in any one of the first aspect of the present application is executed.

[0080] The beneficial effects of the present application are as follows: the method and device can detect 3D lane lines based on BEV feature supervision and deep explicit supervision, thereby realizing effective recognition of 3D lane lines;

[0081] In addition, the method and device can also transform features from image-based view projection to BEV view by utilizing the internal and external parameters of the camera, and establish an implicit depth supervision by learning the BEV view features; at the same time, the true value depth of the point cloud is used to perform explicit joint supervision on the depth features, thereby effectively improving the accuracy of the model lane line depth prediction, and further promoting the lane line foreground point distribution predicted by the final model to be closer to the true value;

[0082] Finally, the method and device can fuse the lane line predictions of all cameras into a unified representation of a scene, thereby improving the comprehensiveness and accuracy of lane line recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0083] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments of the present application will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0084] Figure 1 A schematic diagram of a 3D lane detection method provided in an embodiment of the present application;

[0085] Figure 2 A schematic diagram of a flow chart of another 3D lane line detection method provided in an embodiment of the present application;

[0086] Figure 3 A schematic diagram of an example flow of a 3D lane line detection method provided in an embodiment of the present application;

[0087] Figure 4 A schematic diagram of the structure of a 3D lane line detection device provided in an embodiment of the present application;

[0088] Figure 5 A schematic diagram of the structure of another 3D lane detection device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0089] The technical solutions in the embodiments of the present application will be described below in conjunction with the drawings in the embodiments of the present application.

[0090] It should be noted that similar reference numerals and letters represent similar items in the following drawings, so once an item is defined in one drawing, it does not need to be further defined and explained in subsequent drawings. At the same time, in the description of this application, the terms "first", "second", etc. are only used to distinguish the description and cannot be understood as indicating or implying relative importance.

[0091] Example 1

[0092] Please see Figure 1 , Figure 1 A schematic diagram of a 3D lane line detection method provided in this embodiment. The 3D lane line detection method includes:

[0093] S101. Build a detection model.

[0094] S102: Obtain a road scene image containing lane lines and point cloud true value data.

[0095] S103. Obtain an image semantic feature matrix and a multi-dimensional depth feature matrix according to the road scene image and the detection model.

[0096] S104, converting the multi-dimensional depth feature matrix into depth probabilities at different depths of the camera cone.

[0097] S105 , generating lane line BEV features according to the image semantic feature matrix and the depth probabilities at different depths of the camera cone.

[0098] S106: Supervise the lane line BEV feature according to the road scene image and the point cloud true value data to obtain a first loss value.

[0099] S107. Perform explicit deep supervision on the multi-dimensional deep feature matrix according to the point cloud true value data to obtain a second loss value.

[0100] S108. Update the model parameters of the detection model according to the first loss value and the second loss value to obtain an updated detection model.

[0101] S109: When the model training termination condition is reached, the finally updated detection model is determined as the trained 3D lane line detection model.

[0102] S110, obtaining a real-time road scene image, and performing 3D lane line detection according to the 3D lane line detection model and the real-time road scene image to obtain the lane line coordinates in the 3D coordinate system of the vehicle.

[0103] In this embodiment, the 3D lane line detection method is specifically a 3D lane line detection method based on BEV feature supervision and explicit depth joint supervision scheme.

[0104] In this embodiment, the execution subject of the method may be a computing device such as a computer or a server, and no limitation is made in this embodiment.

[0105] In this embodiment, the execution subject of the method may also be a smart device such as a smart phone, a tablet computer, etc., which is not limited in this embodiment.

[0106] It can be seen that the 3D lane line detection method described in this embodiment can effectively identify 3D lane lines, has low computational complexity and high computational efficiency, and can adapt to complex and changeable driving scenarios.

[0107] Example 2

[0108] Please see Figure 2 , Figure 2 A schematic diagram of a 3D lane line detection method provided in this embodiment. The 3D lane line detection method includes:

[0109] S201, construct a detection model.

[0110] S202: Obtain a road scene image containing lane lines and point cloud true value data.

[0111] In this embodiment, the method can utilize six cameras (front, left front, right front, left rear, right rear, and rear) and one laser radar installed on the vehicle to collect road scene images containing lane lines and laser point cloud true value data.

[0112] In this embodiment, the method can label the lane line points in the point cloud true value data to obtain training and test data.

[0113] S203, extracting features of the road scene image through the MobileNet feature extraction network of the detection model to obtain initial image features.

[0114] S204 , performing depth image coordinate conversion on the point cloud truth data to obtain a depth truth map.

[0115] In this embodiment, the method can use the surround view images from the perspectives of the six cameras on the vehicle as input X k ∈R 3 ×H×W , k = 0, 1, ..., 5; at the same time, the corresponding camera internal parameter is determined as E k ∈3×4, external parameter is I k ∈3×3.

[0116] S205. Perform deep feature extraction on the initial image features according to the FPN feature fusion network of the detection model to obtain an image semantic feature matrix and a multi-dimensional deep feature matrix.

[0117] In this embodiment, the method can use MobileNet as the backbone network of the network structure to extract features from the image inputs under the six camera perspectives respectively, and then use the FPN network to fuse the features to obtain an image semantic feature matrix and a D-dimensional depth feature matrix.

[0118] S206: Convert the multi-dimensional depth feature matrix into depth probabilities at different depths of the camera frustum.

[0119] In this embodiment, the method can generate depth probabilities of different depths of the camera cone according to the 2D image.

[0120] In this embodiment, the method can define D discrete depth value distributions for each pixel of the input image (i.e., in the depth direction of the image, starting from 4m away from the camera plane, taking a depth value every 1m, up to the range of (D+4)m), then each 2D image can generate a depth probability of different depths of the camera cone, each point of which can be described by P(u, v, d), i.e., the coordinates and corresponding depth values ​​in the image coordinate system. Among them, the depth probability of different depths of the camera cone is a position matrix in a spatial relationship obtained according to the camera projection principle, and does not contain image feature information.

[0121] In this embodiment, the method can also use the camera internal and external parameters to convert the point P (u, v, d) in the depth probability of different depths of the camera cone to the coordinate P' (x e ,y e , z e ).

[0122] In this embodiment, the method can use the softmax function to convert the D-dimensional depth feature into depth probability information, that is, each feature map point corresponds to D depth probability values.

[0123] S207, performing an outer product operation on the depth probabilities at different depths of the camera cone and the image semantic feature matrix to obtain a feature matrix.

[0124] In this embodiment, the method may perform an outer product operation on the depth probability and the image semantic feature matrix, and may obtain a feature matrix in which D depth probability information is assigned to each image semantic feature point.

[0125] S208 , mapping the feature matrix to position points corresponding to depth probabilities at different depths of the camera's viewing cone to obtain a first feature point cloud.

[0126] In this embodiment, the given BEV grid is a three-dimensional space of 50 meters in front, behind, left and right of the corresponding vehicle and with no height limit, and is composed of 200×200 BEV grids.

[0127] In this embodiment, the method maps the image features endowed with depth probability information obtained in the above steps to the corresponding position points of depth probabilities at different depths of the camera cone obtained in the above steps according to the camera projection space relationship to obtain a first feature point cloud.

[0128] S209: Convert the first feature point cloud to the vehicle 3D coordinate system to obtain a second feature point cloud.

[0129] In this embodiment, the method converts the first feature point cloud into the 3D coordinate system of the vehicle to obtain the second feature point cloud.

[0130] S210 , converting the second feature point cloud from the ego-vehicle 3D coordinate system to the BEV grid coordinate system to obtain the lane line BEV feature.

[0131] In this embodiment, the method converts the 3D coordinates of the vehicle to the BEV grid coordinates, and performs an elimination operation in the height direction to remove points outside the space represented by the BEV grid, and distributes the remaining valid 3D feature point cloud to each BEV grid according to the coordinate position. Then, for each grid, the method performs cumulative summation on all features of the points belonging to the same grid to obtain the lane line BEV feature.

[0132] S211. Supervise the lane line BEV feature according to the road scene image and the point cloud true value data to obtain a first loss value.

[0133] As an optional implementation, the lane line BEV feature is supervised according to the road scene image and the point cloud true value data to obtain a first loss value, including:

[0134] According to the road scene images and point cloud truth data, the truth value is annotated in the BEV space to obtain the BEV truth value;

[0135] The lane line BEV features are encoded through the pre-built ResNet network to obtain the encoded lane line features;

[0136] The encoded lane line features are detected by a pre-built BEV feature detection head to obtain the lane line features in the BEV space;

[0137] The first loss value is calculated according to the preset FocalLoss loss function, the BEV true value and the lane line features in the BEV space.

[0138] In this embodiment, the method can use ResNet to encode and extract BEV features, and then use the BEV feature detection head to detect lane line features in the BEV space.

[0139] In this embodiment, the BEV feature detection head is a convolutional network. The detection head outputs the lane line foreground feature distribution state in the BEV space (i.e., the lane line features under the BEV space), and uses FocalLoss to supervise the lane line foreground background distribution state of the BEV feature map.

[0140] In this embodiment, the feedback of the feature detection loss corresponding to the detection head prompts the middle layer of the network to learn the depth of the lane line foreground point, thereby establishing an implicit lane line foreground point depth supervision.

[0141] S212. Perform explicit deep supervision on the multi-dimensional deep feature matrix based on the point cloud true value data to obtain a second loss value.

[0142] As an optional implementation, explicit deep supervision is performed on the multi-dimensional deep feature matrix according to the point cloud true value data to obtain a second loss value, including:

[0143] Annotate the point cloud truth data to obtain the depth truth map;

[0144] The second loss value is calculated based on the depth truth map, the preset binary cross entropy loss function and the multi-dimensional depth feature matrix.

[0145] In this embodiment, regarding the deep supervision:

[0146] This method uses the true value depth of the point cloud to jointly supervise the depth features, which will make the distribution of lane line foreground points predicted by the final model closer to the true value.

[0147] Specifically, this method first uses the lane line point cloud truth value to transform the depth image coordinates to obtain the depth truth map: the lane line point cloud truth value P(x e ,y e , z e ) is converted to the k-th camera view image coordinate system with depth through the internal and external parameter matrix of the k-th camera, and P is obtained k (u, v, d), where u and v represent pixel coordinates and d represents depth. k If (u, v, d) does not fall on the kth view, the coordinate is discarded. If the depth truth map has a true depth d conversion result at a certain coordinate (u, v), the probability value at depth d is filled with 1, otherwise the depth truth probability is 0.

[0148] Then, according to the depth truth map, Binary Cross Entropy Loss is used to perform display supervision on the depth features extracted from the lane line BEV features. Among them, the supervision process requires masking of non-lane line foreground points.

[0149] S213. Update the model parameters of the detection model according to the first loss value and the second loss value to obtain an updated detection model.

[0150] S214: Determine whether the model training termination condition is currently met. If so, execute step S215; if not, execute step S203.

[0151] S215: Determine the finally updated detection model as the trained 3D lane line detection model.

[0152] S216: Acquire a real-time road scene picture, and generate a real-time BEV feature vector based on the 3D lane line detection model and the real-time road scene picture.

[0153] S217: Perform lane line decoding processing on the real-time BEV feature vector to obtain lane line BEV grid coordinates.

[0154] S218, converting the lane line BEV grid coordinates into the 3D coordinate system of the own vehicle to obtain the lane line coordinates in the 3D coordinate system of the own vehicle.

[0155] In this embodiment, the 3D lane line detection model will solve the lane line BEV feature supervision and explicit depth joint supervision detection process. The points with feature values ​​greater than the threshold γ in the feature map output by the BEV feature detection head are determined as lane line points, thereby obtaining the BEV grid coordinates corresponding to the lane line. Then, it is converted to the 3D coordinate system of the vehicle to obtain the lane line coordinates in the 3D coordinate system of the vehicle. Among them, this process is the decoding process of the BEV lane line.

[0156] Please see Figure 3 , Figure 3 A schematic diagram of an example process of a 3D lane line detection method is shown.

[0157] In this embodiment, the execution subject of the method may be a computing device such as a computer or a server, and no limitation is made in this embodiment.

[0158] In this embodiment, the execution subject of the method may also be a smart device such as a smart phone, a tablet computer, etc., which is not limited in this embodiment.

[0159] It can be seen that the 3D lane line detection method described in this embodiment can effectively identify 3D lane lines, has low computational complexity and high computational efficiency, and can adapt to complex and changeable driving scenarios.

[0160] Example 3

[0161] Please see Figure 4 , Figure 4 A schematic diagram of the structure of a 3D lane line detection device provided in this embodiment. Figure 4 As shown, the 3D lane line detection device includes:

[0162] A construction unit 310, used to construct a detection model;

[0163] A first acquisition unit 320 is used to acquire a road scene image including lane lines and point cloud true value data;

[0164] A second acquisition unit 330 is used to acquire an image semantic feature matrix and a multi-dimensional depth feature matrix according to the road scene picture and the detection model;

[0165] A conversion unit 340, configured to convert the multi-dimensional depth feature matrix into depth probabilities at different depths of the camera frustum;

[0166] The second generating unit 350 is used to generate a lane line BEV feature according to the image semantic feature matrix and the depth probability of different depths of the camera cone;

[0167] A first supervision unit 360 is used to supervise the lane line BEV feature according to the road scene image and the point cloud true value data to obtain a first loss value;

[0168] A second supervision unit 370 is used to perform explicit deep supervision on the multi-dimensional deep feature matrix according to the point cloud true value data to obtain a second loss value;

[0169] An updating unit 380, configured to update a model parameter of the detection model according to the first loss value and the second loss value to obtain an updated detection model;

[0170] A determination unit 390, configured to determine the finally updated detection model as the trained 3D lane line detection model when a model training termination condition is reached;

[0171] The third acquisition unit 400 is used to acquire a real-time road scene picture;

[0172] The lane line detection unit 410 is used to perform 3D lane line detection based on the 3D lane line detection model and the real-time road scene image to obtain the lane line coordinates in the 3D coordinate system of the vehicle.

[0173] In this embodiment, the explanation of the 3D lane line detection device can refer to the description in Example 1 or Example 2, and will not be further elaborated in this embodiment.

[0174] It can be seen that the 3D lane line detection device described in this embodiment can effectively identify 3D lane lines, has low computational complexity and high computational efficiency, and can adapt to complex and changeable driving scenarios.

[0175] Example 4

[0176] Please see Figure 5 , Figure 5 A schematic diagram of the structure of a 3D lane line detection device provided in this embodiment. Figure 5 As shown, the 3D lane line detection device includes:

[0177] A construction unit 310, used to construct a detection model;

[0178] A first acquisition unit 320 is used to acquire a road scene image including lane lines and point cloud true value data;

[0179] A second acquisition unit 330 is used to acquire an image semantic feature matrix and a multi-dimensional depth feature matrix according to the road scene picture and the detection model;

[0180] A conversion unit 340, configured to convert the multi-dimensional depth feature matrix into depth probabilities at different depths of the camera frustum;

[0181] The second generating unit 350 is used to generate a lane line BEV feature according to the image semantic feature matrix and the depth probability of different depths of the camera cone;

[0182] A first supervision unit 360 is used to supervise the lane line BEV feature according to the road scene image and the point cloud true value data to obtain a first loss value;

[0183] A second supervision unit 370 is used to perform explicit deep supervision on the multi-dimensional deep feature matrix according to the point cloud true value data to obtain a second loss value;

[0184] An updating unit 380, configured to update a model parameter of the detection model according to the first loss value and the second loss value to obtain an updated detection model;

[0185] A determination unit 390, configured to determine the finally updated detection model as the trained 3D lane line detection model when a model training termination condition is reached;

[0186] The third acquisition unit 400 is used to acquire a real-time road scene picture;

[0187] The lane line detection unit 410 is used to perform 3D lane line detection based on the 3D lane line detection model and the real-time road scene image to obtain the lane line coordinates in the 3D coordinate system of the vehicle.

[0188] As an optional implementation manner, the second obtaining unit 330 includes:

[0189] An extraction subunit 331 is used to extract features from the road scene image through a MobileNet feature extraction network of the detection model to obtain initial image features;

[0190] The first conversion subunit 332 is used to perform depth image coordinate conversion on the point cloud truth data to obtain a depth truth map;

[0191] The extraction subunit 331 is further used to perform deep feature extraction on the initial image features according to the FPN feature fusion network of the detection model to obtain an image semantic feature matrix and a multi-dimensional deep feature matrix.

[0192] As an optional implementation manner, the second generating unit 350 includes:

[0193] An outer product subunit 351 is used to perform an outer product operation on the depth probabilities at different depths of the camera cone and the image semantic feature matrix to obtain a feature matrix;

[0194] A mapping subunit 352 is used to map the feature matrix to position points corresponding to depth probabilities at different depths of the camera's viewing cone to obtain a first feature point cloud;

[0195] The second conversion subunit 353 is used to convert the first feature point cloud into the 3D coordinate system of the vehicle to obtain a second feature point cloud;

[0196] The second conversion subunit 353 is further used to convert the second feature point cloud from the ego-vehicle 3D coordinate system to the BEV grid coordinate system to obtain the lane line BEV feature.

[0197] As an optional implementation, the first supervision unit 360 includes:

[0198] The first annotation subunit 361 is used to perform true value annotation in the BEV space according to the road scene image and the point cloud true value data to obtain the BEV true value;

[0199] The encoding subunit 362 is used to encode the lane line BEV feature through a pre-built ResNet network to obtain the encoded lane line feature;

[0200] The detection subunit 363 is used to detect the encoded lane line features through a pre-built BEV feature detection head to obtain the lane line features in the BEV space;

[0201] The first calculation subunit 364 is used to calculate a first loss value according to a preset FocalLoss loss function, a BEV true value, and lane line features in a BEV space.

[0202] As an optional implementation, the second supervision unit 370 includes:

[0203] The second annotation subunit 371 is used to perform truth annotation on the point cloud truth data to obtain a depth truth map;

[0204] The second calculation subunit 372 is used to calculate a second loss value according to the depth truth map, a preset binary cross entropy loss function and a multi-dimensional depth feature matrix.

[0205] As an optional implementation, the 3D lane line detection device further includes:

[0206] A judgment unit 420 is used to judge whether the model training termination condition is currently reached;

[0207] A determination unit 390 is specifically configured to determine the finally updated detection model as the trained 3D lane line detection model when the model training termination condition is reached;

[0208] The second acquisition unit 330 is also used to acquire an image semantic feature matrix and a multi-dimensional depth feature matrix according to the road scene image and the detection model when the model training termination condition is not met.

[0209] As an optional implementation, the lane line detection unit 410 includes:

[0210] A generating subunit 411, for generating a real-time BEV feature vector according to the 3D lane line detection model and the real-time road scene picture;

[0211] A decoding subunit 412 is used to perform lane line decoding processing on the real-time BEV feature vector to obtain the lane line BEV grid coordinates;

[0212] The third conversion subunit 413 is used to convert the lane line BEV grid coordinates into the 3D coordinate system of the vehicle to obtain the lane line coordinates in the 3D coordinate system of the vehicle.

[0213] In this embodiment, the explanation of the 3D lane line detection device can refer to the description in Example 1 or Example 2, and will not be further elaborated in this embodiment.

[0214] It can be seen that the 3D lane line detection device described in this embodiment can effectively identify 3D lane lines, has low computational complexity and high computational efficiency, and can adapt to complex and changeable driving scenarios.

[0215] An embodiment of the present application provides an electronic device, including a memory and a processor, wherein the memory is used to store a computer program, and the processor runs the computer program to enable the electronic device to execute the 3D lane line detection method in embodiment 1 or embodiment 2 of the present application.

[0216] An embodiment of the present application provides a computer-readable storage medium storing computer program instructions. When the computer program instructions are read and executed by a processor, the 3D lane line detection method in Embodiment 1 or Embodiment 2 of the present application is executed.

[0217] In several embodiments provided in the present application, it should be understood that the disclosed devices and methods can also be implemented in other ways. The device embodiments described above are merely schematic. For example, the flowcharts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the devices, methods and computer program products according to multiple embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of a code, and the module, a program segment or a part of a code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart can be implemented with a dedicated hardware-based system that performs a specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions.

[0218] In addition, the functional modules in the various embodiments of the present application may be integrated together to form an independent part, or each module may exist separately, or two or more modules may be integrated to form an independent part.

[0219] If the functions are implemented in the form of software function modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0220] The above description is only an embodiment of the present application and is not intended to limit the scope of protection of the present application. For those skilled in the art, the present application may have various changes and variations. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application should be included in the scope of protection of the present application. It should be noted that similar reference numerals and letters represent similar items in the following drawings, so once an item is defined in one drawing, it does not need to be further defined and explained in the subsequent drawings.

[0221] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0222] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

Claims

1. A 3D lane line detection method, characterized in that: include: Build a detection model; Obtain road scene images and point cloud truth data containing lane lines; According to the road scene picture and the detection model, an image semantic feature matrix and a multi-dimensional depth feature matrix are obtained; Converting the multidimensional depth feature matrix into depth probabilities at different depths of the camera frustum; Generate lane line BEV features according to the image semantic feature matrix and the depth probabilities of different depths of the camera cone; Supervising the lane line BEV feature according to the road scene image and the point cloud true value data to obtain a first loss value; According to the point cloud true value data, performing explicit deep supervision on the multi-dimensional deep feature matrix to obtain a second loss value; Updating the model parameters of the detection model according to the first loss value and the second loss value to obtain an updated detection model; When the model training termination condition is reached, the finally updated detection model is determined as the trained 3D lane line detection model; A real-time road scene image is obtained, and 3D lane line detection is performed according to the 3D lane line detection model and the real-time road scene image to obtain the lane line coordinates in the 3D coordinate system of the vehicle.

2. The 3D lane line detection method according to claim 1, characterized in that: The step of obtaining an image semantic feature matrix and a multi-dimensional depth feature matrix according to the road scene picture and the detection model includes: Extracting features from the road scene image using the MobileNet feature extraction network of the detection model to obtain initial image features; Performing depth image coordinate conversion on the point cloud true value data to obtain a depth true value map; The initial image features are subjected to deep feature extraction according to the FPN feature fusion network of the detection model to obtain an image semantic feature matrix and a multi-dimensional deep feature matrix.

3. The 3D lane line detection method according to claim 1, characterized in that: The generating of lane line BEV features according to the image semantic feature matrix and the depth probabilities of different depths of the camera cone includes: Performing an outer product operation on the depth probabilities at different depths of the camera cone and the image semantic feature matrix to obtain a feature matrix; Mapping the feature matrix to position points corresponding to depth probabilities at different depths of the camera cone to obtain a first feature point cloud; Convert the first feature point cloud to the 3D coordinate system of the vehicle to obtain a second feature point cloud; The second feature point cloud is converted from the 3D coordinate system of the vehicle to the BEV grid coordinate system to obtain the lane line BEV feature.

4. The 3D lane line detection method according to claim 1, characterized in that: The step of supervising the lane line BEV feature according to the road scene image and the point cloud true value data to obtain a first loss value includes: According to the road scene image and the point cloud true value data, perform true value annotation in the BEV space to obtain the BEV true value; Encoding the lane line BEV feature through a pre-built ResNet network to obtain an encoded lane line feature; The encoded lane line features are detected by a pre-built BEV feature detection head to obtain lane line features in a BEV space; A first loss value is calculated according to a preset FocalLoss loss function, the BEV true value, and the lane line features in the BEV space.

5. The 3D lane line detection method according to claim 1, characterized in that: The step of performing explicit deep supervision on the multi-dimensional deep feature matrix according to the point cloud true value data to obtain a second loss value includes: Performing true value annotation on the point cloud true value data to obtain a depth true value map; A second loss value is calculated according to the depth truth map, a preset binary cross entropy loss function and the multidimensional depth feature matrix.

6. The 3D lane line detection method according to claim 1, characterized in that: The method further comprises: Determine whether the model training termination condition is currently reached; If yes, executing the above step of determining the finally updated detection model as the trained 3D lane line detection model; If not, execute the steps of obtaining an image semantic feature matrix and a multi-dimensional depth feature matrix based on the road scene picture and the detection model.

7. The 3D lane line detection method according to claim 1, characterized in that: The performing 3D lane line detection according to the 3D lane line detection model and the real-time road scene image to obtain the lane line coordinates in the 3D coordinate system of the vehicle includes: Generate a real-time BEV feature vector according to the 3D lane line detection model and the real-time road scene picture; Performing lane line decoding processing on the real-time BEV feature vector to obtain lane line BEV grid coordinates; The lane line BEV grid coordinates are converted into the 3D coordinate system of the vehicle to obtain the lane line coordinates in the 3D coordinate system of the vehicle.

8. A 3D lane line detection device, characterized in that: The 3D lane line detection device comprises: A construction unit, used to construct a detection model; A first acquisition unit is used to acquire a road scene image including lane lines and point cloud true value data; A second acquisition unit is used to acquire an image semantic feature matrix and a multi-dimensional depth feature matrix according to the road scene picture and the detection model; A conversion unit, used for converting the multi-dimensional depth feature matrix into depth probabilities at different depths of the camera cone; A second generating unit is used to generate a lane line BEV feature according to the image semantic feature matrix and the depth probability of different depths of the camera cone; A first supervision unit is used to supervise the lane line BEV feature according to the road scene image and the point cloud true value data to obtain a first loss value; A second supervision unit is used to perform explicit deep supervision on the multi-dimensional deep feature matrix according to the point cloud true value data to obtain a second loss value; an updating unit, configured to update a model parameter of the detection model according to the first loss value and the second loss value to obtain an updated detection model; A determination unit, used for determining the finally updated detection model as the trained 3D lane line detection model when the model training termination condition is reached; A third acquisition unit is used to acquire a real-time road scene picture; The lane line detection unit is used to perform 3D lane line detection according to the 3D lane line detection model and the real-time road scene image to obtain the lane line coordinates in the 3D coordinate system of the vehicle.

9. An electronic device, characterized in that: The electronic device includes a memory and a processor, the memory is used to store a computer program, and the processor runs the computer program to enable the electronic device to perform the 3D lane line detection method according to any one of claims 1 to 7.

10. A readable storage medium, characterized in that: The readable storage medium stores computer program instructions, and when the computer program instructions are read and executed by a processor, the 3D lane line detection method according to any one of claims 1 to 7 is executed.

Images