3D lane line estimation method and device, computer readable storage medium, computer equipment and vehicle

Abstract

The invention discloses a 3D lane line estimation method and device, a computer readable storage medium, computer equipment and a vehicle, and the method comprises the steps: obtaining a lane line image of a front view angle of the vehicle; carrying out feature coding on the obtained lane line image of the front view angle of the vehicle through a backbone network, and generating a two-dimensional lane line feature graph and multi-level features in an image space; according to internal and external parameters of a vehicle-mounted camera, projecting a two-dimensional lane line segmentation image and multilevel features in an image space to a pseudo BEV space by adopting inverse perspective transformation; correcting the two-dimensional lane line segmentation graph and the multi-level features projected to the pseudo BEV space through a lane segmentation supervision signal in the BEV space to obtain a lane line feature graph in the real BEV space; and decoding the lane line feature map in the real BEV space by using the 3D lane line prediction head, and outputting a 3D lane line. According to the invention, 3D lane line estimation is carried out through a high-decoupling component structure, high-performance progressive learning is realized, and dependence on expensive real 3D data is eliminated.

Description

Technical Field

[0001] The present invention belongs to the field of intelligent driving technology, and specifically relates to a 3D lane line estimation method, device, computer-readable storage medium, computer equipment and vehicle. Background Art

[0002] Automotive autonomous driving technology can safely and reliably transport passengers to their destinations without human intervention. Accurate and real-time lane perception is essential for autonomous driving applications.

[0003] The current mainstream lane line perception technology can be roughly divided into two types: two-dimensional lane line perception technology and three-dimensional lane line perception technology. Two-dimensional lane line perception technology usually includes three categories: (1) Classification-based lane line detection method, which first classifies the image pixel by pixel or line by line, and then outputs the 2D lane result through post-processing method; (2) Lane curve prediction method, which directly outputs the polynomial expression of the 2D lane line by estimating the polynomial parameters of the lane line curve; (3) Anchor point-based lane line prediction method, which performs position regression and category division on the nodes of the lane line based on the pre-set anchor points, and finally outputs a series of lane line points. Although the above 2D lane line detection methods have achieved good results in detection accuracy and speed, the unreasonable forced setting of plane assumptions due to the lack of vertical height information will cause greater safety hazards and application difficulties.

[0004] The 3D-based lane detection method is an improvement over the 2D lane detection method because it does not make a planar assumption on the ground and is therefore more realistic. A common strategy is to use IPM on a single image from a monocular camera to predict lanes in 3D space, such as 3D-LaneNet, which is an end-to-end network that combines image encoding, bird's-eye view BEV (Bird's Eye View) spatial transformation, and 3D lane prediction. Gen-LaneNet is a continuation of 3D-LaneNet, which divides the network model into an image segmentation subnetwork and a 3D lane prediction subnetwork. It trains the subnetworks separately to reduce the need for 3D lane annotation. In this way, it is also possible to prevent the 2D image feature encoding and the feature encoding in the overhead view coordinate system from interfering with each other. However, its defects are still quite obvious: compared with the end-to-end training method, the two-stage training method is more cumbersome and not conducive to the application of real scenes; the image segmentation sub-network model parameters are fixed in the second-stage training. This two-stage isolated training method causes the final 3D lane prediction result of the network to be heavily dependent on the lane segmentation result of the first stage, that is, the network's capabilities will be limited by the lane segmentation results, especially in extreme or unknown scenarios; transforming the two-dimensional segmentation results through inverse perspective transformation IPM (Inverse Perspective Mapping) is inaccurate, especially the distant lane features will be mixed together, misleading the 3D lane encoding downstream of the network. Summary of the invention

[0005] The technical problem to be solved by the embodiments of the present invention is to provide a 3D lane line estimation method, apparatus, computer-readable storage medium, computer equipment and vehicle to improve the performance and efficiency of 3D lane line estimation.

[0006] In order to solve the above technical problems, the present invention provides a 3D lane line estimation method, comprising:

[0007] Obtaining a lane line image from the front perspective of the vehicle;

[0008] The backbone network is used to encode the features of the lane line image obtained from the front view of the vehicle, and a two-dimensional lane line segmentation map and multi-level features are generated in the image space.

[0009] According to the internal and external parameters of the vehicle-mounted camera, the inverse perspective transformation is used to project the two-dimensional lane line segmentation map and multi-level features in the image space into the pseudo-BEV space.

[0010] The lane segmentation supervision signal in the BEV space is used to correct the two-dimensional lane line segmentation map and multi-level features projected into the pseudo-BEV space, and the lane line feature map in the real BEV space is obtained.

[0011] Use the 3D lane line prediction head to decode the lane line feature map in the real BEV space and output the 3D lane line.

[0012] Through the above steps, the 3D lane line estimation method of this embodiment utilizes a highly decoupled network structure to disassemble the 3D lane line estimation task, which has the advantage of being highly flexible; it corrects the shortcomings of the IPM method and ultimately achieves better 3D lane structure expression.

[0013] Preferably, the lane segmentation supervision signal in the BEV space is used to correct the two-dimensional lane line segmentation map and multi-level features projected into the pseudo BEV space to obtain the lane line feature map in the real BEV space, specifically including:

[0014] A three-layer module is used, each layer includes a feature fusion layer and a 3×3 convolution layer with a stride of 2, to process the 2D lane segmentation map and multi-level features in the pseudo-BEV space, and learn and extract preliminary features containing the 3D coordinate information of the lane lines;

[0015] Based on the extracted preliminary features, a preset segmentation head is used to correct the three-dimensional coordinate information of the lane line, and combined with the lane line segmentation and annotation in the real BEV space for supervision, to obtain the lane line feature map in the real BEV space, wherein the segmentation head includes three groups of modules, each group of modules consists of three convolutional layers and one upsampling layer.

[0016] Through the above correction mechanism, this embodiment adds a lane segmentation supervision signal in the BEV space to correct the pseudo BEV space of the IPM result to obtain the real BE space, thereby overcoming the limitation of the IPM transformation in two-dimensional segmentation.

[0017] Preferably, the feature encoding of the lane line image acquired from the front perspective of the vehicle is performed through the backbone network, and the network model of generating a two-dimensional lane line segmentation map in the image space and a multi-level feature is trained by real 2D data and simulated 2D data; the decoding of the lane line feature map in the real BEV space using the 3D lane line prediction head is performed, and the network model of the output 3D lane line is trained by simulated 3D data.

[0018] Preferably, the simulation scene is set as the source domain, the real scene is set as the target domain, and the training process is divided into two workflows according to different inputs: if the input is an image of lane lines with 3D annotations in the source domain, it will be completely trained through f θ and At the same time, the corresponding data annotations are used to calculate the segmentation loss and 3D lane loss in the image and BEV space respectively; when the input is only the lane line image with 2D annotations in the target domain, only f θAnd only the segmentation loss in the image space is calculated, where f θ In order to perform feature encoding of the lane line image acquired from the front perspective of the vehicle through the backbone network, a network model function for generating a two-dimensional lane line segmentation map and multi-level features in the image space is generated. In order to perform the decoding of the lane feature map in the real BEV space using the 3D lane prediction head, a network model function of the 3D lane is output.

[0019] Preferably, the two workflows are executed alternately, and the 3D data from the source domain and the 2D data from the target domain will alternately guide the function f θ Learn the feature F in the common domain a , when the next source domain data is input, F a Will be from f θ Output to Complete cross-domain expression from image space to BEV space and finally to 3D lane lines.

[0020] Through the above-mentioned cross-domain training mechanism, this embodiment can reduce the high economic cost, high labor cost, and high time cost caused by the difficulties in collecting, labeling, and reviewing real 3D data, and eliminate the demand for 3D real data.

[0021] Preferably, the loss function of the network model is:

[0022]

[0023]

[0024] in, Represent the total loss function under different inputs of the source domain and the target domain respectively; Represent the segmentation loss in image space and BEV space respectively, Represents the loss of 3D lane line output.

[0025] Preferably, for a lane feature map, use the annotation y i and the prospect probability p i , the cross entropy loss is calculated as follows:

[0026]

[0027] Given a 3D lane prediction result (x i ,z i ,v i ,c i ) and its corresponding real annotation Calculate as follows:

[0028]

[0029]

[0030] in, Used to calculate the L1 loss for regression in the x and z directions, while passing the class probability and visibility probability Filter invalid points; and are the cross entropy losses for visibility and category, respectively.

[0031] Preferably, the calculated cross entropy loss is transferred backward from back to front, the gradient of each network structure parameter is calculated respectively, and the network structure parameters are dynamically adjusted according to the direction and size of the gradient.

[0032] The present invention also provides a non-transitory computer-readable storage medium, which includes a stored computer program; wherein, when the computer program is running, it controls the device where the non-transitory computer-readable storage medium is located to execute the 3D lane line estimation method.

[0033] The present invention also provides a 3D lane line estimation device, comprising:

[0034] An acquisition module, used to acquire a lane line image from a front perspective of the vehicle;

[0035] The first network model is used to perform feature encoding on the lane line image acquired from the front perspective of the vehicle through the backbone network, and generate a two-dimensional lane line segmentation map and multi-level features in the image space;

[0036] The second network model is used to project the two-dimensional lane line segmentation map and multi-level features in the image space into the pseudo-BEV space using inverse perspective transformation according to the internal and external parameters of the vehicle-mounted camera;

[0037] The third network model is used to correct the two-dimensional lane line segmentation map and multi-level features projected into the pseudo BEV space through the lane segmentation supervision signal in the BEV space to obtain the lane line feature map in the real BEV space;

[0038] The fourth network model is used to decode the lane line feature map in the real BEV space using the 3D lane line prediction head and output the 3D lane line.

[0039] The 3D lane line estimation device of this embodiment provides a highly decoupled network structure, which disassembles the 3D lane line estimation task and has the advantage of being highly flexible; it corrects the shortcomings of the IPM method and ultimately achieves better 3D lane structure expression.

[0040] Preferably, the third network model is specifically used to: use a three-layer module, each layer includes a feature fusion layer and a 3×3 convolution layer with a step size of 2, process the two-dimensional lane line segmentation map and multi-level features in the pseudo-BEV space, and learn and extract preliminary features containing the three-dimensional coordinate information of the lane line; based on the extracted preliminary features, use a preset segmentation head to correct the three-dimensional coordinate information of the lane line, and combine it with the lane line segmentation annotation in the real BEV space for supervision to obtain the lane line feature map in the real BEV space, wherein the segmentation head includes three groups of modules, each group of modules consists of three convolution layers and one upsampling layer.

[0041] Through the above correction mechanism, this embodiment adds a lane segmentation supervision signal in the BEV space to correct the pseudo BEV space of the IPM result to obtain the real BE space, thereby overcoming the limitation of the IPM transformation in two-dimensional segmentation.

[0042] Preferably, the first network model is trained by real 2D data and simulated 2D data; and the fourth network model is trained by simulated 3D data.

[0043] Preferably, the simulation scene is set as the source domain, the real scene is set as the target domain, and the training process is divided into two workflows according to different inputs: if the input is an image of lane lines with 3D annotations in the source domain, it will be completely trained through f θ and At the same time, the corresponding data annotations are used to calculate the segmentation loss and 3D lane loss in the image and BEV space respectively; when the input is only the lane line image with 2D annotations in the target domain, only f θ And only the segmentation loss in the image space is calculated, where f θ In order to perform feature encoding of the lane line image acquired from the front perspective of the vehicle through the backbone network, a network model function for generating a two-dimensional lane line segmentation map and multi-level features in the image space is generated. In order to perform the decoding of the lane feature map in the real BEV space using the 3D lane prediction head, a network model function of the 3D lane is output.

[0044] Preferably, the two workflows are executed alternately, and the 3D data from the source domain and the 2D data from the target domain will alternately guide the function f θ Learn the feature F in the common domain a , when the next source domain data is input, F a Will be from f θ Output to Complete cross-domain expression from image space to BEV space and finally to 3D lane lines.

[0045] Through the above-mentioned cross-domain training mechanism, this embodiment can reduce the high economic cost, high labor cost, and high time cost caused by the difficulties in collecting, labeling, and reviewing real 3D data, and eliminate the demand for 3D real data.

[0046] Preferably, the loss function of the network model is:

[0047]

[0048]

[0049] in, Represent the total loss function under different inputs of the source domain and the target domain respectively; Represent the segmentation loss in image space and BEV space respectively, Represents the loss of 3D lane line output.

[0050] Preferably, for a lane feature map, use the annotation y i and the prospect probability p i , the cross entropy loss is calculated as follows:

[0051]

[0052] Given a 3D lane prediction result (x i ,z i ,v i ,c i ) and its corresponding real annotation Calculate as follows:

[0053]

[0054]

[0055] in, Used to calculate the L1 loss for regression in the x and z directions, while passing the class probability and visibility probability Filter invalid points; and are the cross entropy losses for visibility and category, respectively.

[0056] Preferably, the calculated cross entropy loss is transferred backward from back to front, the gradient of each network structure parameter is calculated respectively, and the network structure parameters are dynamically adjusted according to the direction and size of the gradient.

[0057] The present invention also provides a computer device, the computer device comprising:

[0058] one or more processors;

[0059] Memory;

[0060] One or more applications, wherein the one or more applications are stored in the memory and configured to be executed by the one or more processors, and the one or more programs are configured to execute the 3D lane line estimation method.

[0061] The present invention also provides a vehicle, characterized in that it comprises the 3D lane line estimation device.

[0062] The implementation of the present invention has the following beneficial effects: providing a highly decoupled component structure for 3D lane line estimation, and the entire network structure is highly maintainable and scalable; correcting the pseudo-BEV space after IPM through BEV lane segmentation supervision to obtain the real BEV space, thereby providing an effective reference for feature encoding downstream of the model; in the model training stage, by adding multi-level supervision signals, the features are subjected to progressive multi-level supervised learning from 2D pixel level to 3D lane level, so as to achieve ultimately better 3D lane structure expression; through the cross-domain training mechanism, the dependence on expensive real 3D data is completely eliminated, and converted into the demand for cheap simulated 3D data and real 2D data, greatly reducing the cost of algorithm training. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0064] Figure 1 It is a flowchart of a 3D lane line estimation method according to Embodiment 1 of the present invention.

[0065] Figure 2 It is a schematic diagram of the geographical expression of lane line detection in an embodiment of the present invention.

[0066] Figure 3 3D lane line estimation in an embodiment of the present invention.

[0067] Figure 4 It is a schematic diagram of the cross-domain training mechanism in an embodiment of the present invention. DETAILED DESCRIPTION

[0068] The following descriptions of the embodiments refer to the accompanying drawings to illustrate specific embodiments in which the present invention may be implemented.

[0069] Please refer to Figure 1As shown, the first embodiment of the present invention provides a 3D lane line estimation method, including:

[0070] Acquire the lane line image from the front perspective of the vehicle;

[0071] The backbone network is used to encode the features of the lane line image obtained from the front view of the vehicle, and a two-dimensional lane line segmentation map and multi-level features are generated in the image space.

[0072] According to the internal and external parameters of the vehicle-mounted camera, the inverse perspective transformation is used to project the two-dimensional lane line segmentation map and multi-level features in the image space into the pseudo-BEV space.

[0073] The lane segmentation supervision signal in the BEV space is used to correct the two-dimensional lane line segmentation map and multi-level features projected into the pseudo-BEV space, and the lane line feature map in the real BEV space is obtained.

[0074] Use the 3D lane line prediction head to decode the lane line feature map in the real BEV space and output the 3D lane line.

[0075] From the above steps, it can be seen that the present invention proposes a highly decoupled network structure, which disassembles the 3D lane line estimation task and has the advantage of being highly flexible; it corrects the shortcomings of the IPM method and ultimately achieves better 3D lane structure expression.

[0076] Generally speaking, the settings of the vehicle camera are as follows: Figure 2 As shown in the figure, the 3D lane lines are represented by the vehicle geographic coordinate system consisting of the x, y, z axes and the origin O. Specifically, the origin O is the projection of the camera center on the road plane, and x, y, and z are used to describe the position information of the left and right, front and back, and top and bottom of the vehicle respectively. The camera coordinate system consists of the origin C and x c ,y c 、z c The origin C represents the center of the camera, x c ,y c 、z c Point to the front, right, and bottom of the camera respectively. c The camera will tilt downward from the horizontal direction to capture more objects near the vehicle. This tilt angle θ and the height h of point C compared to point O together constitute the camera's extrinsic parameters. This extrinsic parameter can be used to achieve the conversion between the vehicle's geographic coordinate system and the camera's coordinate system. In addition, the actual physical size (d x d y ) can realize the conversion from the camera coordinate system to the image coordinate system (u,v). Specifically, their mathematical meaning can be expressed by the following formula, where R and T represent the rotation matrix and translation matrix respectively composed of the camera extrinsic parameters:

[0077]

[0078]

[0079] In addition, in order to more accurately represent the detected lane line position information, the present invention lays out grid anchor points based on the vehicle geographic coordinate plane. The network model will regress the lane position based on the anchor point position to output the final 3D lane line expression. Specifically, in the vehicle geographic coordinate plane P of a preset size, N positions are laid out at equal intervals on the x-axis. These anchor points are also spaced at regular intervals in the vertical direction. Laying, so each The anchor point is composed of 3·K attributes Add 1 lane credibility p i Composition, of which Represents the horizontal offset of the lane point relative to the anchor point position. Indicates the height of the lane point relative to the ground plane, It represents the visibility of the lane point. In the training phase of the model, the real representation of the 3D lane is usually matched with the anchor point representation to obtain the best positive sample. Then, the offset and scale change between the 3D lane line representation and the anchor point are calculated, and these data are used to train the deep learning model. In the model inference phase, the trained model is used to predict each anchor point to obtain the offset and scale change of the lane line, and then these changes are applied to the anchor point to obtain the predicted 3D lane line representation, and also the confidence of the existence of a lane. Usually a threshold is set to retain only the lane prediction results with a confidence higher than the threshold.

[0080] The present invention can first use the backbone network to extract features from the lane line image of the vehicle's front view, and then set anchor points based on these features to predict the 3D lane line representation. Alternatively, the 3D lane line representation is first predicted through the anchor points, and then the prediction results are used to guide the feature extraction of the lane line image of the vehicle's front view and the binary segmentation of the lane.

[0081] It can be understood that the 3D lane line estimation method of the embodiment of the present invention can be decomposed into the following four stages: image to segmentation, image to pseudo-BEV space, pseudo-BEV space to real BEV space, and 3D lane line estimation, which are described in detail below.

[0082] (I) Image to Segmentation

[0083] Please combine Figure 3As shown, in this embodiment, the lane line image from the front perspective of the vehicle is obtained by a vehicle-mounted camera (such as a single forward-looking camera), and the obtained image can be preprocessed: including random rotation, fixed-size scaling, etc.; the rotation angle and scaling size are set according to actual needs.

[0084] It should be noted that different tasks can share the backbone network and improve the final output results of the task. The lane segmentation map can provide pixel-level lane references in the image space for the downstream network, thereby obtaining a better 3D lane representation. As described above, the embodiment of the present invention will perform lane line segmentation in the image space. After preprocessing the lane line image obtained from the front perspective of the vehicle, the backbone network (such as ErfNet) will perform feature encoding on the image, and output a two-dimensional lane line segmentation map and multi-level features in the image space to provide pixel-level geometric references for downstream modules.

[0085] Multi-level features are features with different scales and semantic information extracted from the input lane line image, which can be used to guide lane line segmentation and feature learning; different levels correspond to feature representations of different scales. In the lane line segmentation task, multi-level features can refer to the following aspects:

[0086] 1. Low-level features: mainly include low-level image details such as edges, corners, textures, etc. They can help the model capture the subtle edge information of lane lines and the texture and geometric correspondence with the surrounding environment.

[0087] 2. Mid-level features: Involving higher-level semantic information, such as the shape, relative position, and length of lane lines. Mid-level detail features can be used to capture the global structure and geometric properties of lane lines and provide higher-level semantic constraints.

[0088] 3. High-level features: Associated with the semantic categories of lane lines or more abstract concepts, such as the classification, starting point and end point of lane lines. High-level features can help the model to annotate lane lines more accurately and capture more complex semantic information of lane lines.

[0089] (II) Image to Pseudo-BEV Space

[0090] The embodiment of the present invention adopts the classic IPM method to project the two-dimensional lane segmentation map and multi-level features from the image space to the BEV space. Specifically, given a 2D image pixel (u, v), it can be projected under the BEV space reference system using formulas (1) and (2). However, the traditional IPM method is flawed, especially when the feature map is projected onto the BEV plane, the far-side features are blurred and different lane features are mixed together, thereby limiting the network downstream module from accurately obtaining the structural information contained in the feature map. For the convenience of description, the embodiment of the present invention refers to this imperfect projection reference as a pseudo-BEV space. In the process of projecting to the pseudo-BEV space, problems such as far-side feature blurring and feature mixing will occur, which have a greater impact on downstream tasks. In the model training stage, these pseudo-BEV features are learned, and in theory, the coordinate information of the three-dimensional lane can be learned, but due to the inaccuracy of the projection, the learned coordinate information is also inaccurate.

[0091] It should be noted that before the IPM transformation, the internal pixel values ​​of the lane segmentation results need to be normalized and scaled.

[0092] (III) From Pseudo-BEV Space to Real BEV Space

[0093] After IPM, the two-dimensional lane segmentation map and multi-level features in the pseudo-BEV space are obtained. On this basis, these features will be further encoded in the BEV space to provide a basis for the downstream 3D lane representation. Specifically, the two-dimensional lane segmentation map and multi-level features in the pseudo-BEV space first pass through a three-layer module. Each layer of this module consists of a feature fusion layer and a 3×3 convolution layer with a step size of 2. In each layer, the data stream from the previous layer will be fused with the low-level features in the pseudo-BEV space before the convolution operation. Through the fusion and convolution operations, the lane segmentation results and low-level features are further fused and reduced in dimension to obtain a feature expression containing richer information. On the one hand, by using three-dimensional annotations to supervise the output of the "3D lane prediction head", the module can learn the depth information about the three-dimensional lane in the two-dimensional lane segmentation map and multi-level features in the pseudo-BEV space. This part of information is an abstract state after feature encoding, and is based on the pseudo-BEV space, which is not accurate enough, but can provide a primary feature description related to the spatial position for subsequent processing. Therefore, on the other hand, the present invention corrects this three-dimensional coordinate information through additional modules and supervisory signals: an auxiliary head is added to this part of the network structure to output the lane line feature map in the real BEV space, and its structure is similar to the decoder part in U-Net. Specifically, a segmentation head is added after the module, and the segmentation head includes three groups of modules, each group of modules consists of three layers of convolution layers and one layer of upsampling layer, and the output results are supervised by the lane line segmentation annotation in the real BEV space, so that the output results are expected to be as close as possible from the pseudo-BEV space to the real BEV space, so as to correct the features in the pseudo-BEV space and obtain the real BEV space. Its main operation is to extract more detailed feature information through these convolution and upsampling layers, and restore the feature size to the original resolution. Then, through the lane line segmentation annotation in the real BEV space, the output results are supervised, so that the features learned by the model contain higher-precision lane line information and spatial position information, so that its output results are as close as possible from the pseudo-BEV space to the real BEV space. In other words, this process is to calibrate and correct the features learned in the pseudo-BEV space through the guidance of the network and real annotations, solve the far-side mixing problem, and obtain a lane line description that is closer to the real BEV space.

[0094] 4. 3D Lane Estimation

[0095] The lane line feature map in the real BEV space is decoded through a 3D lane line prediction head similar to that in Gen-LaneNet, and a 3D lane representation containing lane line position information is output.

[0096] The following further introduces the training mechanism of the network model on which the 3D lane line estimation method according to the embodiment of the present invention is based when being implemented.

[0097] Input data: Input a single picture of a single front-view camera; specifically, the picture is a sample image of the lane line at the front-view angle with its corresponding three-dimensional annotation. The three-dimensional annotation represents the three-dimensional position information of the lane line appearing in the front view, which is composed of a series of three-dimensional points. The three-dimensional annotation is projected to the two-dimensional space under the front view and the auxiliary view respectively through the internal and external parameters of the camera to generate two-dimensional segmentation annotations of the front view (image space) and the top view (BEV space). It can be understood that annotation is not required for actual lane line estimation. Annotation only exists in the training stage. The fixed model parameters are obtained by annotating the training model, which is then applied in practice.

[0098] Data preprocessing: Randomly rotate and resize the input images.

[0099] Image to segmentation: Use backbone to encode image features, use the two-dimensional lane line segmentation annotation in image space to guide feature learning in image space, and generate a two-dimensional lane line segmentation map and multi-level features.

[0100] Image to pseudo-BEV space: The two-dimensional lane segmentation map and multi-level features are projected into the pseudo-BEV space using the traditional IPM method based on the internal and external parameters of the vehicle-mounted camera.

[0101] From pseudo-BEV to real BEV space: As mentioned above, on the one hand, by using three-dimensional annotations to supervise the output of the "3D lane line prediction head", the module can learn the coordinate information of the three-dimensional lane from the two-dimensional lane segmentation map and multi-level features in the pseudo-BEV space; on the other hand, this three-dimensional coordinate information is corrected through additional modules and supervision signals: an auxiliary head is added to the network structure of this part to output the lane line feature map in the real BEV space, and its structure is similar to the decoder part in U-Net. Specifically, a segmentation head is added after the module, and the segmentation head includes three groups of modules, each of which consists of three convolutional layers and one upsampling layer, and its output results are supervised by the lane line segmentation annotations in the real BEV space, in the hope that the output results are as close as possible from the pseudo-BEV space to the real BEV space, so as to correct the features in the pseudo-BEV space and obtain the real BEV space.

[0102] Backward transfer: The network has a total of three output results, namely the segmentation map in image space and BEV space, and the final 3D lane structure. The three output results correspond to the three annotations in the input data - three-dimensional annotations, two-dimensional segmentation annotations of the front view (image space) and the top view (BEV space). The current loss value can be calculated from the three annotations and the model output results as a basis for judging whether the network output results are ideal. The network will pass the loss value backward from back to front, calculate the gradient of each parameter respectively, and dynamically adjust the network structure parameters according to the direction and size of the gradient.

[0103] Repeat the above process until the network converges.

[0104] It should be noted that this training method is end-to-end, and each part of the network structure is jointly trained under the action of multiple supervisory signals to give full play to the coordination and joint advantages of the entire network. The present invention proposes a highly decoupled network structure, which decomposes the 3D lane line estimation task into four parts, which has the advantage of being highly flexible. In particular, the four parts correspond to four progressive stages from 2D images to 3D lane line expression. By defining auxiliary learning objectives for multiple stages, sufficient guidance can be provided for the feature learning of the middle layer of the network to output better 3D lane expressions.

[0105] It should also be noted that the working mode of the above-mentioned training mechanism is the same as most general 3D lane detection work currently, relying on real 3D lane annotation to train the network. However, this strict data requirement often brings about problems such as difficulty in collecting real 3D data annotation, difficulty in annotation, and difficulty in review in actual use. In order to overcome these problems, an embodiment of the present invention provides a cross-domain training mechanism that combines simulation and real data, and divides network training into two parts: "two-dimensional lane line segmentation training" and "3D lane line estimation training". In the "two-dimensional lane line segmentation training" part, real 2D data and simulated 2D data are used for training, while in the "3D lane line estimation training" part, simulated 3D data is used for training. In this way, the need for real 3D data is completely eliminated, and it is replaced by the need for simulated 3D data that can be produced automatically and at low cost.

[0106] like Figure 4 As shown in Figure 1, the simulation scene is set as the source domain and the real scene is set as the target domain. Based on the 3D lane data in the source domain, cheap 2D lane data is introduced from the target domain to better obtain the 3D lane representation in the target domain. For convenience, the “image to segmentation” part in the network architecture is simplified to function f θ , the rest is simplified to function

[0107] For clarity, the training process is divided into two workflows according to different inputs: (1) If the input is a lane image with 3D annotations in the source domain, it will be fully trained through f θ and At the same time, the corresponding data annotations are used to calculate the segmentation loss and 3D lane loss in the image and BEV space respectively. (2) When the input is only a lane image with 2D annotations in the target domain, it will only pass f θ And only the segmentation loss in the image space is calculated.

[0108] It should be noted that these two workflows are performed alternately rather than synchronously. Therefore, the 3D data from the source domain and the 2D data from the target domain will alternately guide the function f θ Learn the features F in the public domain a When the next source domain data is input, F a Will be from f θ Output to Complete the cross-domain expression from image space to BEV space and finally to 3D lane lines. "Common domain" refers to the common area of ​​the target domain and the source domain. Although one is a simulation scene and the other is a real scene, for the network model, there are some common features between the two (such as the most basic shape, color, texture, etc.). Since they come from two different image data, the expression of this feature is inconsistent at the beginning and needs to be aligned and unified during the training process.

[0109] Combining 2D real data (target domain data) and 3D simulated data (source domain data) for training during the training phase has two benefits:

[0110] First, the model can handle both simulated and real scenes: Since the training process includes 3D simulated data in the source domain and 2D real data in the target domain, the learning model is capable of handling different scenes. The model learns the features of the simulated scene from the 3D simulated data and learns the mapping from these features to 3D lanes. At the same time, it learns the features of the real scene from the 2D real data and how to map these features to 2D lane representations.

[0111] Second, only a large amount of 2D real data (not dependent on 3D real data) is needed for training to output 3D lane results in real scenes: it does not rely on 3D data in real scenes to output 3D results. Thanks to cross-domain learning, even with only a large amount of 2D real data and a small amount of 3D simulation data, 3D lane results can be output in real scenes. This is mainly because the model has learned how to generate 3D lane representations from 3D simulation data, and can apply this understanding to features extracted from 2D real data.

[0112] According to the aforementioned training process, in the “image to segmentation” stage, by applying the function f on the lane line images of the source domain and the target domain θ , extracting 2D lane segmentation maps and multi-level features; used to generate pseudo-BEV space representations in the "image to pseudo-BEV space" stage. This is usually optimized by calculating the segmentation loss in image space.

[0113] In the "image to pseudo-BEV space" stage, the extracted two-dimensional lane line segmentation map is projected to a "top-down" or "bird's-eye" perspective by converting the perspective and embedding 3D information, preparing for the next step of "pseudo-BEV to real BEV space".

[0114] Finally, in the step of "pseudo-BEV to real BEV space", the pseudo-BEV space data obtained in the previous step is needed and converted through the function After further processing and transformation, we can finally get the real BEV space representation, that is, the real 3D lane line.

[0115] The design of the network training objective function of the embodiment of the present invention is introduced below.

[0116] The loss function of the embodiment of the present invention has two modes, depending on whether the input is from the source domain or the target domain, which corresponds to the aforementioned cross-domain training mechanism. When a source domain image with 3D annotations is input, its loss function consists of three parts: segmentation loss in image and BEV space, and the final 3D lane prediction loss. When a target domain image with 2D annotations is input, its loss function only has image space segmentation loss. They can be mathematically expressed as:

[0117]

[0118]

[0119] Represent different total loss functions under source domain and target domain input respectively. In addition, Represent the segmentation loss in image space and BEV space respectively, and they both calculate cross entropy loss in pixels. Given a lane feature map, we can use the annotation y i and the prospect probability p i , the cross entropy loss is calculated as follows:

[0120]

[0121] Represents the loss of the three-dimensional lane output. Given the three-dimensional lane prediction result (x i ,z i ,v i ,c i) and its corresponding real annotation It can be calculated as follows:

[0122]

[0123]

[0124] in, Used to calculate the L1 loss (mean absolute error MAE, Mean Absolute Error) for regression in the x and z directions, and at the same time pass the category probability and visibility probability Invalid points are filtered. In addition, and are the cross entropy losses for visibility and category, respectively, and their calculations are similar to formula (5).

[0125] In the Apollo-Sim-3d public dataset, the present invention achieved improvements of 4.3, 8.9, and 5.0 in F-score indicators compared with Gen-LaneNet in three data subsets.

[0126] Corresponding to the 3D lane line estimation method described in the aforementioned embodiment 1 of the present invention, embodiment 2 of the present invention provides that the present invention also provides a non-transitory computer-readable storage medium, and the non-transitory computer-readable storage medium includes a stored computer program; wherein, the computer program controls the device where the non-transitory computer-readable storage medium is located to execute the 3D lane line estimation method when running.

[0127] Preferably, the computer program can be divided into one or more modules / units (such as computer program 1, computer program 2, ...), and the one or more modules / units are stored in the memory and executed by the processor to complete the present invention. The one or more modules / units can be a series of computer program instruction segments that can complete specific functions, and the instruction segments are used to describe the execution process of the computer program in the device.

[0128] The processor may be a central processing unit (CPU), or other general-purpose processors, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor, or the processor may be any conventional processor. The processor is the control center of the device, and various parts of the device are connected using various interfaces and lines.

[0129] The memory mainly includes a program storage area and a data storage area, wherein the program storage area can store an operating system, an application required for at least one function, etc., and the data storage area can store related data, etc. In addition, the memory can be a high-speed random access memory, or a non-volatile memory, such as a plug-in hard disk, a smart memory card (SmartMedia Card, SMC), a secure digital (Secure Digital, SD) card, and a flash card (Flash Card), etc., or the memory can also be other volatile solid-state storage devices.

[0130] It should be noted that the above-mentioned device may include but is not limited to a processor and a memory, which can be understood by those skilled in the art.

[0131] Corresponding to the 3D lane line estimation method described in the first embodiment of the present invention, the third embodiment of the present invention provides a 3D lane line estimation device, including:

[0132] An acquisition module, used to acquire a lane line image from a front perspective of the vehicle;

[0133] The first network model is used to perform feature encoding on the lane line image acquired from the front perspective of the vehicle through the backbone network, and generate a two-dimensional lane line segmentation map and multi-level features in the image space;

[0134] The second network model is used to project the two-dimensional lane line segmentation map and multi-level features in the image space into the pseudo-BEV space using inverse perspective transformation according to the internal and external parameters of the vehicle-mounted camera;

[0135] The third network model is used to correct the two-dimensional lane line segmentation map and multi-level features projected into the pseudo BEV space through the lane segmentation supervision signal in the BEV space to obtain the lane line feature map in the real BEV space;

[0136] The fourth network model is used to decode the lane line feature map in the real BEV space using the 3D lane line prediction head and output the 3D lane line.

[0137] Preferably, the third network model is specifically used to: use a three-layer module, each layer includes a feature fusion layer and a 3×3 convolution layer with a step size of 2, process the two-dimensional lane line segmentation map and multi-level features in the pseudo-BEV space, and learn and extract preliminary features containing the three-dimensional coordinate information of the lane line; based on the extracted preliminary features, use a preset segmentation head to correct the three-dimensional coordinate information of the lane line, and combine it with the lane line segmentation annotation in the real BEV space for supervision to obtain the lane line feature map in the real BEV space, wherein the segmentation head includes three groups of modules, each group of modules consists of three convolution layers and one upsampling layer.

[0138] Preferably, the first network model is trained by real 2D data and simulated 2D data; and the fourth network model is trained by simulated 3D data.

[0139] Preferably, the simulation scene is set as the source domain, the real scene is set as the target domain, and the training process is divided into two workflows according to different inputs: if the input is an image of lane lines with 3D annotations in the source domain, it will be completely trained through f θ and At the same time, the corresponding data annotations are used to calculate the segmentation loss and 3D lane loss in the image and BEV space respectively; when the input is only the lane line image with 2D annotations in the target domain, only f θ And only the segmentation loss in the image space is calculated, where f θ In order to perform feature encoding of the lane line image acquired from the front perspective of the vehicle through the backbone network, a network model function for generating a two-dimensional lane line segmentation map and multi-level features in the image space is generated. In order to perform the decoding of the lane feature map in the real BEV space using the 3D lane prediction head, a network model function of the 3D lane is output.

[0140] Preferably, the two workflows are executed alternately, and the 3D data from the source domain and the 2D data from the target domain will alternately guide the function f θ Learn the feature F in the common domain a , when the next source domain data is input, F a Will be from f θ Output to Complete cross-domain expression from image space to BEV space and finally to 3D lane lines.

[0141] Preferably, the loss function of the network model is:

[0142]

[0143]

[0144] in, Represent the total loss function under different inputs of the source domain and the target domain respectively; Represent the segmentation loss in image space and BEV space respectively, Represents the loss of 3D lane line output.

[0145] Preferably, for a lane feature map, use the annotation y i and the prospect probability p i , the cross entropy loss is calculated as follows:

[0146]

[0147] Given a 3D lane prediction result (x i ,z i ,v i ,c i ) and its corresponding real annotation Calculate as follows:

[0148]

[0149]

[0150] in, Used to calculate the L1 loss for regression in the x and z directions, while passing the class probability and visibility probability Filter invalid points; and are the cross entropy losses for visibility and category, respectively.

[0151] Preferably, the calculated cross entropy loss is transferred backward from back to front, the gradient of each network structure parameter is calculated respectively, and the network structure parameters are dynamically adjusted according to the direction and size of the gradient.

[0152] Corresponding to the 3D lane line estimation method described in the first embodiment of the present invention, the fourth embodiment of the present invention provides a computer device, the computer device comprising:

[0153] one or more processors;

[0154] Memory;

[0155] One or more applications, wherein the one or more applications are stored in the memory and configured to be executed by the one or more processors, and the one or more programs are configured to execute the 3D lane line estimation method described in Embodiment 1 of the present invention.

[0156] Corresponding to the 3D lane line estimation device described in the aforementioned third embodiment of the present invention, the fifth embodiment of the present invention provides a vehicle, comprising the 3D lane line estimation device described in the third embodiment of the present invention.

[0157] For the working principle and process of the above embodiment, please refer to the description of the above embodiment of the present invention, which will not be repeated here.

[0158] It can be seen from the above description that compared with the prior art, the beneficial effects of the present invention are: providing a highly decoupled component structure for 3D lane line estimation, and the entire network structure is highly maintainable and scalable; the pseudo-BEV space after IPM is corrected through BEV lane segmentation supervision to obtain the real BEV space, thereby providing an effective reference for the feature encoding of the downstream model; in the model training stage, by adding multi-level supervision signals, the features are progressively multi-level supervised learning from 2D pixel level to 3D lane level, so as to achieve ultimately better 3D lane structure expression; through the cross-domain training mechanism, the dependence on expensive real 3D data is completely eliminated, and converted into the demand for cheap simulated 3D data and real 2D data, greatly reducing the cost of algorithm training.

[0159] The above disclosure is only the preferred embodiment of the present invention, which certainly cannot be used to limit the scope of the present invention. Therefore, equivalent changes made according to the claims of the present invention are still within the scope of the present invention.

Claims

1. A 3D lane line estimation method, characterized in that: include: Obtaining a lane line image from the front perspective of the vehicle; The backbone network is used to encode the features of the lane line image obtained from the front view of the vehicle, and a two-dimensional lane line segmentation map and multi-level features are generated in the image space. According to the internal and external parameters of the vehicle-mounted camera, the inverse perspective transformation is used to project the two-dimensional lane line segmentation map and multi-level features in the image space into the pseudo-BEV space. The lane segmentation supervision signal in the BEV space is used to correct the two-dimensional lane line segmentation map and multi-level features projected into the pseudo-BEV space, and the lane line feature map in the real BEV space is obtained. Use the 3D lane line prediction head to decode the lane line feature map in the real BEV space and output the 3D lane line.

2. The 3D lane line estimation method according to claim 1, characterized in that: The lane segmentation supervision signal in the BEV space is used to correct the two-dimensional lane line segmentation map and multi-level features projected into the pseudo BEV space to obtain the lane line feature map in the real BEV space, specifically including: A three-layer module is used, each layer includes a feature fusion layer and a 3×3 convolution layer with a stride of 2, to process the 2D lane segmentation map and multi-level features in the pseudo-BEV space, and learn and extract preliminary features containing the 3D coordinate information of the lane lines; Based on the extracted preliminary features, a preset segmentation head is used to correct the three-dimensional coordinate information of the lane line, and combined with the lane line segmentation and annotation in the real BEV space for supervision, to obtain the lane line feature map in the real BEV space, wherein the segmentation head includes three groups of modules, each group of modules consists of three convolutional layers and one upsampling layer.

3. The 3D lane line estimation method according to claim 1, characterized in that: The method of performing feature encoding on the lane line image acquired from the front perspective of the vehicle through the backbone network, generating a two-dimensional lane line segmentation map in the image space and a network model with multi-level features trained by real 2D data and simulated 2D data; the method of using the 3D lane line prediction head to decode the lane line feature map in the real BEV space, and outputting a network model of 3D lane lines trained by simulated 3D data.

4. The 3D lane line estimation method according to claim 3, characterized in that: The simulation scene is set as the source domain, the real scene is set as the target domain, and the training process is divided into two workflows according to different inputs: If the input is an image of lane lines with 3D annotations in the source domain, it will be completely trained by f θ and At the same time, the corresponding data annotations are used to calculate the segmentation loss and 3D lane loss in the image and BEV space respectively; when the input is only the lane line image with 2D annotations in the target domain, only f θ And only the segmentation loss in the image space is calculated, where f θ In order to perform feature encoding of the lane line image acquired from the front perspective of the vehicle through the backbone network, a network model function for generating a two-dimensional lane line segmentation map and multi-level features in the image space is generated. In order to perform the decoding of the lane feature map in the real BEV space using the 3D lane prediction head, a network model function of the 3D lane is output.

5. The 3D lane line estimation method according to claim 4, characterized in that: The two workflows are executed alternately, and the 3D data from the source domain and the 2D data from the target domain will alternately guide the function f θ Learn the feature F in the common domain a , when the next source domain data is input, F a Will be from f θ Output to Complete cross-domain expression from image space to BEV space and finally to 3D lane lines.

6. The 3D lane line estimation method according to claim 4, characterized in that: The loss function of the network model is: in, Represent the total loss function under different inputs of the source domain and the target domain respectively; Represent the segmentation loss in image space and BEV space respectively, Represents the loss of 3D lane line output.

7. The 3D lane line estimation method according to claim 6, characterized in that: For a lane feature map, use the annotation y i and the prospect probability p i , the cross entropy loss is calculated as follows: Given a 3D lane prediction result (x i ,z i ,v i ,c i ) and its corresponding real annotation Calculate as follows: in, Used to calculate the L1 loss for regression in the x and z directions, while passing the class probability and visibility probability Filter invalid points; and are the cross entropy losses for visibility and category, respectively.

8. The 3D lane line estimation method according to claim 7, characterized in that: The calculated cross entropy loss is transferred backward from back to front, and the gradient of each network structure parameter is calculated respectively, and the network structure parameters are dynamically adjusted according to the direction and size of the gradient.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored computer program; wherein, when the computer program is run, it controls the device where the computer-readable storage medium is located to execute the 3D lane line estimation method according to any one of claims 1 to 8.

10. A 3D lane line estimation device, characterized in that: include: An acquisition module, used to acquire a lane line image from a front perspective of the vehicle; The first network model is used to perform feature encoding on the lane line image acquired from the front perspective of the vehicle through the backbone network, and generate a two-dimensional lane line segmentation map and multi-level features in the image space; The second network model is used to project the two-dimensional lane line segmentation map and multi-level features in the image space into the pseudo-BEV space using inverse perspective transformation according to the internal and external parameters of the vehicle-mounted camera; The third network model is used to correct the two-dimensional lane line segmentation map and multi-level features projected into the pseudo BEV space through the lane segmentation supervision signal in the BEV space to obtain the lane line feature map in the real BEV space; The fourth network model is used to decode the lane line feature map in the real BEV space using the 3D lane line prediction head and output the 3D lane line.

11. A computer device, characterized in that: The computer device comprises: one or more processors; Memory; One or more applications, wherein the one or more applications are stored in the memory and configured to be executed by the one or more processors, and the one or more programs are configured to execute the 3D lane line estimation method as described in any one of claims 1-8.

12. A vehicle, characterized in that: Includes the 3D lane line estimation device as described in claim 10.

Images