Lane line detection method and device, electronic equipment and storage medium

Abstract

The invention discloses a lane line detection method and device, electronic equipment and a storage medium, the lane line detection method can be applied to the technical field of automatic driving, and the method comprises the steps: obtaining a multi-view image collected by an image collection device of a target vehicle; projecting the multi-view image to a bird's-eye view plane to form a bird's-eye view corresponding to the multi-view image; performing feature fusion on the multi-view image and the aerial view to obtain a target feature image; performing lane line multi-task processing based on the target feature image, and determining target pixel points belonging to a lane line area in the target feature image, offset information of the target pixel points relative to a reference line in the target feature image and a relationship among the target pixel points; and performing lane line aggregation based on the target pixel points, the offset information corresponding to the target pixel points and the relationship between the target pixel points to obtain a lane line detection result. Under the condition of expanding the receptive field of the lane line, the efficient performance of lane line detection can still be kept.

Description

Technical Field

[0001] The present application relates to the field of autonomous driving technology, and in particular to a lane line detection method, device, electronic device and storage medium. Background Art

[0002] Lane detection is one of the core tasks in the fields of autonomous driving, robot motion control, and Advanced Driver Assistance Systems (ADAS). The detection accuracy of lanes directly affects the accuracy of motion planning of mobile agents, as well as the stability and safety of the system in different motion scenarios. Especially in complex urban environments, due to the presence of various interference factors such as occlusion, lighting changes, and dynamic obstacles, it is particularly difficult to obtain a lane detection model with high robustness and strong generalization.

[0003] For autonomous driving tasks, the ultimate goal is not only to identify the lane lines themselves, but also to capture relevant information in the vehicle coordinate system or the world coordinate system. Pure two-dimensional (2D) perception results are often not enough to support high-level decision-making tasks, because these results need to be further mapped to three-dimensional (3D) space to obtain accurate three-dimensional positioning information. Only by accurately locating the lane lines in three-dimensional space can subsequent prediction, path planning, and control tasks be supported. However, there is still a significant gap between existing 2D lane line detection methods and actual application requirements, especially in terms of positioning accuracy and robustness in three-dimensional space. Summary of the invention

[0004] In order to solve the problems of the prior art, the embodiments of the present application provide a lane line detection method, device, electronic device and storage medium. The technical solution is as follows:

[0005] On the one hand, a lane line detection method is provided, the method comprising:

[0006] Acquire multi-view images acquired by an image acquisition device of a target vehicle;

[0007] Projecting the multi-view image onto a bird's-eye view plane to form a bird's-eye view corresponding to the multi-view image;

[0008] Performing feature fusion on the multi-view image and the bird's-eye view to obtain a target feature image;

[0009] Perform lane line multi-task processing based on the target feature image, determine target pixel points in the target feature image that belong to the lane line area, offset information of the target pixel points relative to the reference line in the target feature image, and the relationship between the target pixel points;

[0010] Lane line aggregation is performed based on the target pixel point, the offset information corresponding to the target pixel point, and the relationship between the target pixel points to obtain a lane line detection result.

[0011] In another aspect, a lane line detection device is provided, the device comprising:

[0012] An image acquisition module, used to acquire multi-view images acquired by an image acquisition device of a target vehicle;

[0013] An image projection module, used for projecting the multi-view image onto a bird's-eye view plane to form a bird's-eye view corresponding to the multi-view image;

[0014] A feature fusion module, used for performing feature fusion on the multi-view image and the bird's-eye view to obtain a target feature image;

[0015] a multi-task processing module, configured to perform lane line multi-task processing based on the target feature image, determine target pixel points in the target feature image belonging to the lane line area, offset information of the target pixel points relative to the reference line in the target feature image, and the relationship between the target pixel points;

[0016] The lane line aggregation module is used to perform lane line aggregation based on the target pixel point, the offset information corresponding to the target pixel point and the relationship between the target pixel points to obtain a lane line detection result.

[0017] In an exemplary embodiment, the feature fusion module includes:

[0018] A feature extraction module, used to perform multi-level feature extraction on the multi-view images to obtain multi-level lane line features;

[0019] A feature enhancement module, used to fuse the multi-level lane line features layer by layer to obtain a feature enhancement map corresponding to the multi-view image;

[0020] A feature bird's-eye view module, used to project the feature enhancement image onto the bird's-eye view plane and fuse it with the bird's-eye view to obtain a bird's-eye view feature image;

[0021] The extraction and compression module is used to perform feature extraction and spatial dimension compression on the bird's-eye view feature image to obtain the target feature image.

[0022] In an exemplary embodiment, the multi-tasking processing module includes:

[0023] A region segmentation module, used to distinguish a lane line region from a non-lane line region from the target feature image to obtain a segmentation mask; the segmentation mask indicates the target pixel points in the target feature image that belong to the lane line region;

[0024] A pixel offset module, used to estimate the vertical offset of the target pixel relative to the reference line in the target feature image to obtain the offset information;

[0025] The extraction and dimensionality reduction module is used to perform feature extraction and dimensionality reduction processing on the target feature image to obtain an embedded feature matrix; the embedded feature matrix represents the relationship between the target pixel points.

[0026] In an exemplary embodiment, the feature extraction module includes:

[0027] A model acquisition module, used to acquire a pre-trained lane detection model; the pre-trained lane detection model includes a feature perception framework, and the feature perception framework includes a backbone network;

[0028] A backbone network module, used to use the multi-view image as an input of the backbone network to perform multi-level feature extraction on the multi-view image to obtain the multi-level lane line features;

[0029] Accordingly, the feature perception framework further includes a neck network; the feature enhancement module includes:

[0030] A neck network module, used for taking the multi-level lane line features as input of the neck network to fuse the multi-level lane line features layer by layer to obtain the feature enhancement map corresponding to the multi-view image;

[0031] Accordingly, the feature perception framework further includes a fusion layer; the feature bird's-eye view module includes:

[0032] A fusion layer module, used to use the feature enhancement map and the bird's-eye view as inputs of the fusion layer, so as to project the feature enhancement map onto the bird's-eye view plane and fuse it with the bird's-eye view to obtain the bird's-eye feature map;

[0033] Accordingly, the feature perception framework further includes a coding layer; the extraction and compression module includes:

[0034] The coding layer module is used to use the bird's-eye view feature map as the input of the coding layer to perform feature extraction and spatial dimension compression on the bird's-eye view feature map to obtain the target feature image.

[0035] In an exemplary embodiment, the pre-trained lane detection model further includes a target detection network, and the target detection network includes a segmentation layer; the region segmentation module includes:

[0036] A segmentation layer module, used to use the target feature image as an input of the segmentation layer to distinguish the lane line area from the non-lane line area from the target feature image to obtain the segmentation mask;

[0037] Accordingly, the target detection network further includes an offset layer; the pixel offset module includes:

[0038] An offset layer module, used to use the target feature image as an input of the offset layer to estimate the vertical offset of the target pixel point belonging to the lane line area in the target feature image relative to the reference line in the target feature image, so as to obtain the offset information;

[0039] Accordingly, the target detection network further includes an embedding layer; the extraction and dimensionality reduction module includes:

[0040] The embedding layer module is used to use the target feature image as the input of the embedding layer to perform feature extraction and dimensionality reduction processing on the target feature image to obtain the embedded feature matrix.

[0041] In an exemplary embodiment, the apparatus further includes a model training module for training the initial lane line detection model to obtain the pre-trained lane line detection model, the model training module including:

[0042] A sample acquisition module, used to acquire multi-view sample images and real lane line information corresponding to the multi-view sample images;

[0043] A sample projection module, used for projecting the multi-view sample image onto a bird's-eye view plane to form a sample bird's-eye view corresponding to the multi-view sample image;

[0044] A lane line prediction module, used to input the multi-view sample image and the sample bird's-eye view into an initial lane line detection model to obtain predicted lane line information;

[0045] A model loss module, configured to determine a model loss of the lane line detection model based on a difference between the predicted lane line information and the actual lane line information;

[0046] A parameter adjustment module is used to adjust the parameters of the initial lane line detection model based on the model loss until a preset training end condition is met to obtain the pre-trained lane line detection model.

[0047] In an exemplary embodiment, the real lane line information includes a real segmentation mask, real offset information and a real embedded feature matrix, and the predicted lane line information includes a predicted segmentation mask, predicted offset information and a predicted embedded feature matrix; the model loss module includes:

[0048] A segmentation loss module, configured to determine a segmentation loss based on a difference between the predicted segmentation mask and the true segmentation mask, and an overlap degree between the predicted segmentation mask and the true segmentation mask;

[0049] an offset module, configured to determine an offset loss based on a difference between the predicted offset information and the actual offset information;

[0050] A feature vector module, for determining, based on the real embedded feature matrix, the proximity between feature vectors belonging to the same lane line in the predicted embedded feature matrix, and the distance between feature vectors belonging to different lane lines in the predicted embedded feature matrix;

[0051] a push-pull determination module, configured to determine a determination loss based on the proximity degree and the distance degree; and determine a push-pull loss based on the proximity degree and the distance degree;

[0052] An embedding loss module, configured to determine an embedding loss based on the discrimination loss and the push-pull loss;

[0053] A loss determination module is used to determine the model loss based on the segmentation loss, the offset loss and the embedding loss.

[0054] On the other hand, an electronic device is provided, including a processor and a memory, wherein the memory stores at least one instruction or at least one program, and the at least one instruction or the at least one program is loaded and executed by the processor to implement the lane line detection method of any one of the above aspects.

[0055] On the other hand, a computer-readable storage medium is provided, in which at least one instruction or at least one program is stored, and the at least one instruction or the at least one program is loaded and executed by a processor to implement a lane line detection method as described in any of the above aspects.

[0056] On the other hand, a computer program product or a computer program is provided, the computer program product or the computer program includes computer instructions, the computer instructions are stored in a computer-readable storage medium. A processor of an electronic device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the electronic device performs the lane line detection method of any aspect above.

[0057] The embodiment of the present application obtains multi-perspective images acquired by an image acquisition device of the target vehicle, and projects these multi-perspective images onto a bird's-eye view plane to form a corresponding bird's-eye view, then performs feature fusion on the multi-perspective image and the bird's-eye view to obtain a target feature image, performs lane line multi-task processing based on the target feature image, determines target pixel points belonging to the lane line area in the target feature image, as well as the offset information of these target pixel points relative to the reference line and the relationship between the target pixel points, and performs lane line aggregation based on this to obtain accurate lane line detection results. The two-dimensional target feature image in the bird's-eye view plane indirectly provides an information representation in a three-dimensional space, and there is no need to map the target feature image to a three-dimensional space. Therefore, when the lane line receptive field is expanded, the lane line detection can still maintain high efficiency, improve detection accuracy and robustness, and enhance the adaptability to complex environments and occlusion conditions. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0059] Figure 1 It is a flowchart of a lane line detection method provided in an embodiment of the present application;

[0060] Figure 2 Schematic diagram of the framework of the lane detection model provided in the embodiment of the present application;

[0061] Figure 3 It is a real lane line feature map provided by the embodiment of the present application;

[0062] Figure 4 is an embedded feature map provided in an embodiment of the present application;

[0063] Figure 5 is another embedded feature map provided in an embodiment of the present application;

[0064] Figure 6 is a structural block diagram of a lane line detection device provided in an embodiment of the present application;

[0065] Figure 7 It is a hardware structure block diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0066] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0067] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or server that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0068] It is understandable that in the specific implementation of this application, related data such as user information is involved. When the above embodiments of this application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of relevant data need to comply with relevant laws, regulations and standards of relevant countries and regions.

[0069] See also Figure 1 , which shows a flow chart of a lane line detection method provided in an embodiment of the present application. It should be noted that this specification provides method operation steps such as the embodiments or flow charts, but may include more or fewer operation steps based on conventional or non-creative labor. The order of steps listed in the embodiments is only one way of executing the steps among many orders, and does not represent the only order of execution. When the system or product is executed in practice, it can be executed in the order of the methods shown in the embodiments or drawings or in parallel (for example, in a parallel processor or multi-threaded processing environment). Specifically, Figure 1 As shown, the method may include:

[0070] S101, acquiring multi-view images captured by an image capture device of a target vehicle.

[0071] The target vehicle refers to the vehicle that is performing lane detection and related tasks. Specifically, the target vehicle is equipped with relevant sensors (such as cameras, lidar, etc.) and processing systems, which are used to collect data from the surrounding environment and process it to achieve functions such as lane detection.

[0072] The image acquisition device refers to a device installed on the target vehicle to capture images of the surrounding environment. Specifically, the image acquisition device is installed at different positions on the vehicle body to provide multi-view images to cover a larger field of view. In specific implementations, the image acquisition device usually includes but is not limited to cameras (monocular cameras, multi-camera cameras, surround cameras, etc.), laser radars, millimeter-wave radars, infrared cameras, etc.

[0073] Among them, multi-view images refer to multiple images taken by image acquisition devices at different positions of the vehicle body in the same scene, covering all directions around the vehicle and providing a full range of perspectives. Specifically, images from different perspectives can provide complementary information, helping to eliminate blind spots or occlusions that may exist in a single perspective. Even if there is occlusion or difficulty in identification in one perspective, images from another perspective may provide additional clues.

[0074] Exemplarily, cameras may be installed in four directions, front, back, left, and right, of the target vehicle to obtain front, rear, left, and right images.

[0075] S103, projecting the multi-view image onto a bird's-eye view plane to form a bird's-eye view corresponding to the multi-view image.

[0076] The bird's-eye view plane refers to a virtual plane that is a vertical downward view of the target vehicle and its surroundings. This plane is usually parallel to the ground and is used to project multi-view images into the same reference system. In the bird's-eye view plane, the layout of the target vehicle and its surroundings can be clearly seen.

[0077] Specifically, by projecting the multi-view images onto the bird's-eye view plane, a unified bird's-eye view is formed, eliminating the impact of perspective differences, and can intuitively display the layout of the target vehicle and the surrounding environment, including lane lines. In the bird's-eye view, features such as lane lines are easier to identify and process because they present a simpler geometric form.

[0078] Specifically, the distortion problem of the multi-view image is corrected according to the calibration parameters of the image acquisition device, such as radial distortion. The corrected multi-view image is converted from the original front view to a bird's-eye view through a perspective transformation algorithm. This step requires determining the points in the image corresponding to the ground plane and calculating the affine matrix for the transformation. After the images of different viewpoints are converted to the same bird's-eye reference system, they are fused to obtain the bird's-eye view corresponding to the multi-view image.

[0079] S105, performing feature fusion on the multi-view image and the bird's-eye view to obtain a target feature image.

[0080] Among them, the target feature image contains the processed feature information for subsequent lane line detection.

[0081] Specifically, features are extracted from the multi-view image and the bird's-eye view respectively, the extracted features are fused to enhance the feature expression capability, and the fused feature map is used as the target feature image.

[0082] In an exemplary embodiment, the above step S105 may include the following steps:

[0083] Perform multi-level feature extraction on the multi-view images to obtain multi-level lane line features. Specifically, extract multi-level features from the multi-view images, for example, low-level features (such as edges, textures, colors, etc.), mid-level features (such as the local shape of the lane line), and high-level features (such as the overall shape and position of the lane line). In the specific implementation, basic edge features, local features, more complex features, and global features are extracted from the multi-view images.

[0084] The multi-level lane features are fused layer by layer to obtain the feature enhancement map corresponding to the multi-view image. Specifically, the multi-level lane features are integrated to better capture the details of the lanes. In the specific implementation, the features of different levels can be fused by upsampling and layer-by-layer fusion to enhance the feature expression capability. Specifically, the deep feature map is upsampled to a higher resolution to restore the detail information, and the upsampled feature map is added element by element to the shallow feature map to enhance the feature expression capability.

[0085] The feature enhancement map is projected onto the bird's-eye view plane and fused with the bird's-eye view to obtain a bird's-eye view feature map. Specifically, the enhanced feature map is converted into a unified bird's-eye view feature map for subsequent processing. In the specific implementation, the enhanced feature maps of different viewing angles are projected into a unified BEV (Bird's Eye View) coordinate system through perspective transformation, superimposed with the bird's-eye view, and the superimposed feature maps are further processed using convolutional layers or other methods to enhance the feature expression capability.

[0086] The bird's-eye view feature map is subjected to feature extraction and spatial dimension compression to obtain a target feature image. Specifically, a more compact and abstract feature representation is extracted from the bird's-eye view feature map for lane line multi-task processing. In a specific implementation, the bird's-eye view feature map can be further encoded using a convolutional layer to extract higher-level features, a pooling layer can be used to reduce the spatial dimension of the feature map to further compress the features, and a fully connected layer or other methods can be used to further aggregate features to obtain a compact feature representation.

[0087] Specifically, the lane line detection model can be used to perform feature fusion on the multi-view image and the bird's-eye view to obtain a target feature image, including: obtaining a pre-trained lane line detection model; Figure 2 As shown, the pre-trained lane line detection model includes a feature perception framework, which includes a backbone network, a neck network, a fusion layer, and a coding layer; the multi-view image is used as the input of the backbone network to perform multi-level feature extraction on the multi-view image to obtain multi-level lane line features; the multi-level lane line features are used as the input of the neck network to fuse the multi-level lane line features layer by layer to obtain a feature enhancement map corresponding to the multi-view image; the feature enhancement map and the bird's-eye view are used as the input of the fusion layer to project the feature enhancement map to the bird's-eye view plane and fuse it with the bird's-eye view to obtain a bird's-eye feature map; the bird's-eye feature map is used as the input of the coding layer to extract features and compress the spatial dimension of the bird's-eye feature map to obtain a target feature image.

[0088] Specifically, FastBEV (Fast Bird's Eye View) can be used as a feature perception framework. FastBEV provides a lightweight BEV feature extraction method and feature encoding network, which aims to improve computational efficiency and enable the model to run quickly on mobile devices. It also introduces temporal feature fusion technology, which can be used to complete 3D target detection. The design of FastBEV takes into account the real-time requirements of autonomous driving systems and the need to deploy on resource-constrained devices.

[0089] Among them, the main function of the backbone network is to extract multi-level features from multi-view images. Specifically, ResNet-18 can be used as the backbone network. It has four main network blocks, each of which is responsible for extracting features at different levels. The first layer is the convolutional layer, which is used to extract basic edge features, the second layer is the residual block, which is used to extract local features, the third layer is a deeper residual block, which is used to extract more complex features, and the fourth layer is the deepest residual block, which is used to extract global features.

[0090] The main function of the neck network is to improve the feature expression ability through feature fusion and enhancement. It integrates the features of different levels extracted by the backbone network to better capture the details of objects such as lane lines. Specifically, the feature pyramid network (FPN) can be used as the neck network to fuse features of different levels through upsampling and layer-by-layer fusion.

[0091] The function of the fusion layer is to convert the feature enhancement map output by the neck network into a unified bird's-eye view feature map, and fuse it with the bird's-eye view corresponding to the multi-view image for subsequent processing. The multi-view two-dimensional image information is integrated into a unified bird's-eye view, thereby indirectly providing a three-dimensional information representation.

[0092] The function of the encoding layer is to extract a more compact and abstract feature representation from the bird's-eye view feature map for subsequent lane line multi-task processing.

[0093] It can be seen from the above technical solutions of the embodiments of the present application that the embodiments of the present application obtain multi-level lane line features by performing multi-level feature extraction on multi-view images, and fuse these features layer by layer to obtain a feature enhancement map corresponding to the multi-view image, project the feature enhancement map to the bird's-eye view plane and fuse it with the bird's-eye view to obtain a bird's-eye feature map, perform feature extraction and spatial dimension compression on the bird's-eye view feature map to obtain a target feature image, thereby achieving enhanced expression of lane line features in the bird's-eye view plane, indirectly providing a three-dimensional information representation, which not only improves the detection accuracy, but also enhances the robustness and generalization ability of the system, while ensuring the real-time processing capability of the system.

[0094] S107, performing lane line multi-task processing based on the target feature image, determining target pixel points in the target feature image that belong to the lane line area, offset information of the target pixel points relative to the reference line in the target feature image, and the relationship between the target pixel points.

[0095] Among them, lane line multi-task processing includes image segmentation, offset estimation, and instance differentiation. Specifically, the task goal of image segmentation is to determine which pixels belong to the lane line. The segmentation result can be used to define the location of the lane line and help the target vehicle understand the road boundary it should follow; the task goal of offset estimation is to estimate the offset of each pixel on the lane line relative to the reference line. The obtained offset information helps to more accurately describe the actual shape of the lane line; the purpose of instance differentiation is to distinguish different lane line instances, even if two lane lines are close to each other, they can be correctly distinguished.

[0096] The reference line refers to a virtual line used for reference and benchmark, which is usually used to describe the position and shape of the lane line. Specifically, the reference line provides a position benchmark for the lane line relative to the vehicle or other reference objects. For example, the reference line can be the center line of the lane line, or the center line of the vehicle.

[0097] For the relationship between target pixels, specifically, determine which pixels are adjacent, that is, which pixels are close to each other in the image and belong to the same lane line. The pixels belonging to the same lane line can be treated as a connected component; the similarity between pixels is measured by calculating the distance between the feature vectors corresponding to the pixels. A closer feature vector distance means that the pixel is more likely to belong to the same lane line, and the pixels between different lane lines have a larger distance in the feature space.

[0098] In an exemplary embodiment, the above step S107 may include the following steps:

[0099] The lane line area and the non-lane line area are distinguished from the target feature image to obtain a segmentation mask; the segmentation mask indicates the target pixel points in the target feature image that belong to the lane line area. Specifically, one or more convolutional layers are used to extract features. These convolutional layers can capture the local features of the lane line. The target feature image is restored to the size of the original input image through upsampling operations (such as transposed convolution or bilinear interpolation). Classification is performed at each pixel position, usually through a 1x1 convolutional layer (also called point-by-point convolution) to generate a segmentation mask, and a sigmoid function is used to generate the probability of each pixel belonging to the lane line area (ranging between 0 and 1). Among them, the segmentation mask is represented as a two-dimensional matrix, and each element represents the probability that the corresponding pixel belongs to the lane line area. In specific implementation, it is possible to parse which pixels belong to the lane line area from the segmentation mask. Usually, a threshold is set to mark pixels with a segmentation probability higher than a certain threshold as lane lines.

[0100] Estimate the vertical offset of the target pixel relative to the reference line in the target feature image to obtain the offset information. Specifically, use one or more convolutional layers to extract features. These convolutional layers can capture the position information of the lane line. The target feature image is restored to the size of the original input image through upsampling operations. Regression is performed at each pixel position, usually through a 1x1 convolutional layer to generate the offset, and a linear activation function (i.e., no activation function) is used to generate the offset information of each pixel. The offset information is expressed as a two-dimensional matrix, and each element represents the vertical offset of the corresponding pixel relative to the reference line. The vertical offset refers to the offset distance of the target pixel relative to the reference line in the vertical direction (y-axis direction).

[0101] The target feature image is subjected to feature extraction and dimensionality reduction processing to obtain an embedded feature matrix; the embedded feature matrix represents the relationship between the target pixels. Specifically, one or more convolutional layers are used to extract features. These convolutional layers can capture the relationship information of the lane lines. The target feature map is restored to the size of the original input image through upsampling operations, and a low-dimensional embedding vector is generated at each pixel position. The embedded feature map is usually generated through a 1x1 convolutional layer, and a linear activation function (i.e., no activation function) is used to generate the embedding vector of each pixel. The embedded feature matrix is ​​represented as a three-dimensional matrix, and each element represents the embedding vector of the corresponding pixel. In a specific implementation, it is possible to identify from the embedded feature matrix which pixels corresponding to the segmentation mask belong to the same lane line, which is usually achieved by calculating the similarity between the embedded vectors in the embedded feature matrix.

[0102] Specifically, lane line multi-task processing can be performed through the lane line detection model, including: the pre-trained lane line detection model also includes a target detection network, such as Figure 2 As shown in the figure, the target detection network includes a segmentation layer, an offset layer, and an embedding layer; the target feature image is used as the input of the segmentation layer to distinguish the lane line area and the non-lane line area from the target feature image to obtain a segmentation mask; the target feature image is used as the input of the offset layer to estimate the vertical offset of the target pixel points belonging to the lane line area in the target feature image relative to the reference line in the target feature image to obtain the offset information; the target feature image is used as the input of the embedding layer to perform feature extraction and dimensionality reduction on the target feature image to obtain an embedded feature matrix.

[0103] Specifically, BEVLaneDet (Bird's Eye View Lane Detection) can be used as the target detection network. BEVLaneDet is a network focused on lane detection. It uses virtual camera technology to ensure spatial consistency and can adapt to more complex scenarios based on key points.

[0104] The main function of the segmentation layer is to distinguish the lane line area from other background areas in the input target feature image and generate a segmentation mask to indicate which pixels belong to the lane line area. The segmentation layer can include convolutional layers, upsampling layers, classification layers, and activation functions, which respectively perform the tasks of feature extraction, image resizing, pixel classification, and probability generation.

[0105] The main function of the offset layer is to estimate the vertical offset of each point on the lane line relative to the reference line. The offset layer can include a convolution layer, an upsampling layer, a regression layer, and an activation function, which respectively perform the tasks of feature extraction, image resizing, regression calculation, and offset generation.

[0106] The main function of the embedding layer is to generate an embedded feature map to capture the relationship between lane lines and help determine which pixels corresponding to the segmentation mask belong to the same lane line. The embedding layer can include a convolutional layer, an upsampling layer, an embedding vector layer, and an activation function, which respectively perform the tasks of feature extraction, image resizing, embedded feature generation, and embedding vector generation.

[0107] It can be seen from the above technical solutions of the embodiment of the present application that the embodiment of the present application distinguishes the lane line area from the non-lane line area from the target feature image, and obtains a segmentation mask indicating the target pixel points belonging to the lane line area in the target feature image; estimates the vertical offset of the target pixel points relative to the reference line in the target feature image, performs feature extraction and dimensionality reduction processing on the target feature image, and obtains an embedded feature matrix that characterizes the relationship between the target pixel points, thereby realizing accurate recognition and description of the lane lines and enhancing the accuracy and robustness of detection.

[0108] S109, performing lane line aggregation based on the target pixel points, the offset information corresponding to the target pixel points, and the relationship between the target pixel points to obtain a lane line detection result.

[0109] In the specific implementation, a clustering algorithm is used to cluster the pixels belonging to the same lane line in the segmentation mask, such as DBSCAN (Density-Based Spatial Clustering of Applications with Noise), MeanShift, etc., and curve fitting is performed on the clustered lane line pixels to obtain a smooth lane line contour, such as polynomial fitting, Spline interpolation, etc. Filters (such as median filtering, mean filtering, etc.) are used to denoise and smooth the lane line pixels, remove abnormal points, check for isolated points or noise points in the lane line contour, and remove them, and verify whether the detected lane line meets the expected shape and logic (for example, the lane line should be continuous and reasonable). If the system processes continuous video frames, the lane line detection results of multiple frames need to be fused to improve the consistency and stability of detection.

[0110] The lane detection result can be presented in any format required by the system, such as visualizing the lane detection result in the form of an image, or sending the position information of the lane to a path planning module of the target vehicle.

[0111] It can be seen from the above technical solutions of the embodiments of the present application that the embodiments of the present application obtain multi-perspective images collected by the image acquisition device of the target vehicle, and project these multi-perspective images onto the bird's-eye view plane to form a corresponding bird's-eye view, and then perform feature fusion on the multi-perspective image and the bird's-eye view to obtain a target feature image, and perform lane line multi-task processing based on the target feature image to determine the target pixel points belonging to the lane line area in the target feature image, as well as the offset information of these target pixel points relative to the reference line and the relationship between the target pixel points, and perform lane line aggregation based on this to obtain accurate lane line detection results. The two-dimensional target feature image in the bird's-eye view plane indirectly provides an information representation in a three-dimensional space, and there is no need to map the target feature image to the three-dimensional space. Therefore, when the lane line receptive field is expanded, the lane line detection can still maintain high efficiency, improve detection accuracy and robustness, and enhance the adaptability to complex environments and occlusion conditions.

[0112] The following is an explanation of the training process of the lane detection model, which may include:

[0113] Obtain multi-view sample images and real lane line information corresponding to the multi-view sample images;

[0114] Projecting the multi-view sample image onto a bird's-eye view plane to form a sample bird's-eye view corresponding to the multi-view sample image;

[0115] Input the multi-view sample images and the sample bird's-eye view into the initial lane line detection model to obtain the predicted lane line information;

[0116] Determine the model loss of the lane line detection model based on the difference between the predicted lane line information and the actual lane line information;

[0117] The parameters of the initial lane line detection model are adjusted based on the model loss until the preset training end conditions are met to obtain a pre-trained lane line detection model.

[0118] The multi-view sample images are obtained from image acquisition devices installed at different positions of the target vehicle.

[0119] The real lane line information refers to the marked lane line position information. Specifically, the real lane line information is usually obtained through manual or semi-automatic labeling. This information includes the precise position, shape and other related attributes of the lane line, which is used as the standard answer for training the model. During the training process, the model's prediction results will be compared with the real lane line information to evaluate the accuracy of the model.

[0120] The bird's-eye view of the sample is formed by projecting the multi-view sample image into the BEV coordinate system through geometric transformation (such as perspective transformation).

[0121] The predicted lane line information refers to the lane line position information predicted by the model based on the input multi-view sample images and sample bird's-eye view. During the training process, the model will try to generate predicted lane line information based on the input images. These predicted information will be compared with the real lane line information to evaluate the prediction accuracy of the model and adjust the model parameters accordingly.

[0122] Among them, model loss is a quantitative indicator used to measure the difference between the model prediction results and the actual lane line information during the training process. It is used to guide the optimization algorithm (such as gradient descent) to adjust the model parameters to minimize the difference between the predicted lane line information and the actual lane line information.

[0123] Among them, the preset training end conditions refer to a set of conditions used to decide when to stop training during the training process. These conditions usually include but are not limited to the following: reaching a predetermined number of iterations, loss function convergence, validation set performance, and early stopping strategy. Specifically, a maximum number of iterations is set, and when the model iterates to the specified number of times, the training ends; when the change in the loss function is less than a preset threshold, it indicates that the model has converged, and training can be stopped at this time; set performance standards on the validation set, such as when the accuracy or loss on the validation set reaches a certain level, stop training; if the performance on the validation set does not improve within a certain number of iterations, terminate the training early. By setting these training end conditions, overfitting can be avoided and the model can be ensured to reach the optimal state within a reasonable time.

[0124] It can be seen from the above technical solutions of the embodiments of the present application that the embodiments of the present application train the initial lane line detection model, and the pre-trained lane line detection model obtained is used for efficient processing of multi-view sample images and accurate detection of lane line information, ensuring that the lane line detection model can maintain high efficiency and reliability in practical applications.

[0125] In an exemplary embodiment, the real lane line information includes a real segmentation mask, real offset information and a real embedded feature matrix, and the predicted lane line information includes a predicted segmentation mask, predicted offset information and a predicted embedded feature matrix; the step of determining the model loss of the lane line detection model based on the difference between the predicted lane line information and the real lane line information may include the following steps:

[0126] The segmentation loss is determined based on the difference between the predicted segmentation mask and the true segmentation mask, as well as the degree of overlap between the predicted segmentation mask and the true segmentation mask. Specifically, binary cross entropy and intersection over union (IoU) are used as segmentation losses. Binary cross entropy is used to measure the difference between the predicted segmentation mask output by the model and the true segmentation mask, encouraging the model to generate the correct segmentation mask and distinguish lane lines from the background; IoU is used to measure the degree of overlap between the predicted segmentation mask output by the model and the true segmentation mask, ensuring that the model can accurately distinguish lane lines from the background.

[0127] Based on the difference between the predicted offset information and the actual offset information, the offset loss is determined. Specifically, binary cross entropy is used as the offset loss to measure the difference between the predicted offset information output by the model and the predicted offset information, ensuring that the model can accurately estimate the offset of each pixel in the Y direction.

[0128] Based on the real embedded feature matrix, determine the closeness between the feature vectors belonging to the same lane line in the predicted embedded feature matrix, and determine the distance between the feature vectors belonging to different lane lines in the predicted embedded feature matrix; determine the discriminative loss based on the closeness and distance, and determine the push-pull loss based on the closeness and distance; determine the embedding loss based on the discriminative loss and the push-pull loss. Specifically, the discriminative loss (DiscriminativeLoss) is used to ensure that the feature vectors in the same lane line are as close as possible, and the feature vectors between different lane lines are as far away as possible, helping the model learn better lane line distinction capabilities; the push-pull loss (NDPushPullLoss, Normalized Distance Push-Pull Loss) combines the "push" and "pull" effects to optimize the feature distribution in the embedding space, ensuring that the feature vectors between different lane lines are as far away as possible, and the feature vectors within the same lane line are as close as possible, improving the quality of feature representation.

[0129] Specifically, the embedding loss is obtained by weighting DiscriminativeLoss and NDPushPullLoss. Compared with the single NDPushPullLoss, the embedded feature map generated by the embedding layer is enhanced, and the lane detection accuracy is improved. Figures 3 to 5 , Figure 3 It is a real lane feature map provided by the embodiment of the present application. The model is trained by using a single NDPushPullLoss as the embedding loss. The corresponding embedded feature map generated is as follows Figure 4As shown in Figure 2, the model is trained using DiscriminativeLoss and NDPushPullLoss as embedding losses, and the corresponding embedded feature maps generated are as follows: Figure 5 As shown, comparison Figure 4 and Figure 5 It can be seen that the lane line features of the latter are significantly enhanced.

[0130] Based on the segmentation loss, offset loss and embedding loss, the model loss is determined. In specific implementation, the model loss of the lane detection model is usually obtained by weighted summing of the segmentation loss, offset loss and embedding loss.

[0131] It can be seen from the above technical solutions of the embodiment of the present application that the embodiment of the present application determines the model loss by respectively calculating the segmentation loss, offset loss and embedding loss, wherein the embedding loss is obtained by weighting the discriminant loss and the push-pull loss. Compared with the single discriminant loss, the embedded feature map generated by the embedding layer is enhanced, thereby improving the lane line detection accuracy.

[0132] Corresponding to the lane line detection methods provided in the above-mentioned embodiments, the embodiments of the present application also provide a lane line detection device. Since the lane line detection device provided in the embodiments of the present application corresponds to the lane line detection methods provided in the above-mentioned embodiments, the implementation methods of the aforementioned lane line detection methods are also applicable to the lane line detection device provided in this embodiment and will not be described in detail in this embodiment.

[0133] See also Figure 6 , which is a schematic diagram of the structure of a lane line detection device provided in an embodiment of the present application, the device has the function of implementing the lane line detection method in the above method embodiment, and the function can be implemented by hardware or by hardware executing corresponding software. Figure 6 As shown, the device may include:

[0134] An image acquisition module 610 is used to acquire multi-view images acquired by an image acquisition device of a target vehicle;

[0135] An image projection module 620, configured to project the multi-view image onto a bird's-eye view plane to form a bird's-eye view corresponding to the multi-view image;

[0136] A feature fusion module 630 is used to perform feature fusion on the multi-view image and the bird's-eye view to obtain a target feature image;

[0137] A multi-task processing module 640 is used to perform lane line multi-task processing based on the target feature image, determine the target pixel points belonging to the lane line area in the target feature image, the offset information of the target pixel points relative to the reference line in the target feature image, and the relationship between the target pixel points;

[0138] The lane line aggregation module 650 is used to perform lane line aggregation based on target pixels, offset information corresponding to the target pixels, and the relationship between the target pixels to obtain a lane line detection result.

[0139] In an exemplary embodiment, the feature fusion module includes:

[0140] A feature extraction module is used to perform multi-level feature extraction on multi-view images to obtain multi-level lane line features;

[0141] The feature enhancement module is used to fuse the multi-level lane line features layer by layer to obtain the feature enhancement map corresponding to the multi-view image;

[0142] A feature bird's-eye view module is used to project the feature enhancement map onto the bird's-eye view plane and fuse it with the bird's-eye view to obtain a bird's-eye view feature map;

[0143] The extraction and compression module is used to extract features and compress the spatial dimension of the bird's-eye view feature map to obtain a target feature image.

[0144] In an exemplary embodiment, the multi-tasking processing module includes:

[0145] A region segmentation module is used to distinguish the lane line region from the non-lane line region from the target feature image to obtain a segmentation mask; the segmentation mask indicates the target pixel points belonging to the lane line region in the target feature image;

[0146] A pixel offset module is used to estimate the vertical offset of the target pixel relative to the reference line in the target feature image to obtain offset information;

[0147] The extraction and dimensionality reduction module is used to perform feature extraction and dimensionality reduction processing on the target feature image to obtain an embedded feature matrix; the embedded feature matrix represents the relationship between the target pixel points.

[0148] In an exemplary embodiment, the feature extraction module includes:

[0149] A model acquisition module, used to acquire a pre-trained lane detection model; the pre-trained lane detection model includes a feature perception framework, and the feature perception framework includes a backbone network;

[0150] The backbone network module is used to use the multi-view image as the input of the backbone network to perform multi-level feature extraction on the multi-view image to obtain multi-level lane line features;

[0151] Accordingly, the feature perception framework also includes a neck network; a feature enhancement module, including:

[0152] The neck network module is used to use the multi-level lane line features as the input of the neck network to fuse the multi-level lane line features layer by layer to obtain a feature enhancement map corresponding to the multi-view image;

[0153] Accordingly, the feature perception framework also includes a fusion layer; a feature bird's-eye view module, including:

[0154] A fusion layer module, used for taking the feature enhancement map and the bird's-eye view as inputs of the fusion layer, so as to project the feature enhancement map onto the bird's-eye view plane and fuse it with the bird's-eye view to obtain a bird's-eye view feature map;

[0155] Accordingly, the feature perception framework also includes an encoding layer; an extraction and compression module, including:

[0156] The coding layer module is used to use the bird's-eye view feature map as the input of the coding layer to extract features and compress the spatial dimension of the bird's-eye view feature map to obtain a target feature image.

[0157] In an exemplary embodiment, the pre-trained lane detection model further includes a target detection network, which includes a segmentation layer; and a region segmentation module, including:

[0158] A segmentation layer module is used to use the target feature image as an input of the segmentation layer to distinguish the lane line area from the non-lane line area in the target feature image to obtain a segmentation mask;

[0159] Accordingly, the target detection network also includes an offset layer; a pixel offset module, including:

[0160] An offset layer module is used to use the target feature image as an input of the offset layer to estimate the vertical offset of the target pixel points belonging to the lane line area in the target feature image relative to the reference line in the target feature image to obtain offset information;

[0161] Accordingly, the target detection network also includes an embedding layer; an extraction and dimensionality reduction module, including:

[0162] The embedding layer module is used to take the target feature image as the input of the embedding layer to perform feature extraction and dimensionality reduction on the target feature image to obtain an embedded feature matrix.

[0163] In an exemplary embodiment, the apparatus further includes a model training module for training the initial lane line detection model to obtain a pre-trained lane line detection model, the model training module including:

[0164] A sample acquisition module is used to acquire multi-view sample images and real lane line information corresponding to the multi-view sample images;

[0165] A sample projection module, used to project the multi-view sample image onto a bird's-eye view plane to form a sample bird's-eye view corresponding to the multi-view sample image;

[0166] The lane line prediction module is used to input the multi-view sample images and the sample bird's-eye view into the initial lane line detection model to obtain the predicted lane line information;

[0167] A model loss module, for determining a model loss of a lane detection model based on a difference between predicted lane information and actual lane information;

[0168] The parameter adjustment module is used to adjust the parameters of the initial lane line detection model based on the model loss until the preset training end conditions are met to obtain a pre-trained lane line detection model.

[0169] In an exemplary embodiment, the real lane line information includes a real segmentation mask, a real offset information and a real embedded feature matrix, and the predicted lane line information includes a predicted segmentation mask, a predicted offset information and a predicted embedded feature matrix; the model loss module includes:

[0170] A segmentation loss module, configured to determine a segmentation loss based on a difference between the predicted segmentation mask and the true segmentation mask, and an overlap between the predicted segmentation mask and the true segmentation mask;

[0171] A migration module, configured to determine a migration loss based on a difference between the predicted migration information and the actual migration information;

[0172] A feature vector module, for determining the proximity between feature vectors belonging to the same lane line in the predicted embedded feature matrix, and determining the distance between feature vectors belonging to different lane lines in the predicted embedded feature matrix based on the real embedded feature matrix;

[0173] A push-pull discrimination module, used to determine a discrimination loss based on a proximity degree and a distance degree; and to determine a push-pull loss based on a proximity degree and a distance degree;

[0174] Embedding loss module, used to determine the embedding loss based on the discriminant loss and the push-pull loss;

[0175] The loss determination module is used to determine the model loss based on the segmentation loss, offset loss and embedding loss.

[0176] It should be noted that the device provided in the above embodiment, when implementing its functions, is only illustrated by the division of the above functional modules. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. In addition, the device and method embodiments provided in the above embodiment belong to the same concept, and the specific implementation process is detailed in the method embodiment, which will not be repeated here.

[0177] An embodiment of the present application provides an electronic device, which includes a processor and a memory, wherein the memory stores at least one instruction or at least one program, and the at least one instruction or the at least one program is loaded and executed by the processor to implement any lane line detection method provided in the above method embodiments.

[0178] The memory can be used to store software programs and modules. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory. The memory can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, application programs required for functions, etc.; the data storage area can store data created according to the use of the device, etc. In addition, the memory can include a high-speed random access memory and can also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage devices. Accordingly, the memory can also include a memory controller to provide the processor with access to the memory.

[0179] The method embodiments provided in the embodiments of the present application may be executed in a computer terminal, a server or a similar computing device, that is, the above-mentioned electronic device may include a computer terminal, a server or a similar computing device. Figure 7 is a hardware structure block diagram of a computer device running a lane line detection method provided by an embodiment of the present invention, such as Figure 7 As shown, the internal structure of the computer device may include but is not limited to: a processor, a network interface and a memory. The processor, network interface and memory in the computer device may be connected via a bus or other means. Figure 7 The example of connecting through bus is taken in the following.

[0180] Among them, the processor (or CPU (Central Processing Unit)) is the computing core and control core of the computer device. The network interface may optionally include a standard wired interface, a wireless interface (such as WI-FI, a mobile communication interface, etc.). Memory is a memory device in a computer device for storing programs and data. It is understandable that the memory here can be a high-speed RAM storage device or a non-volatile memory device (non-volatile memory), such as at least one disk storage device; optionally, it can also be at least one storage device located away from the aforementioned processor. The memory provides a storage space, which stores the operating system of the electronic device, which may include but is not limited to: Windows system (an operating system), Linux (an operating system), Android (Android, a mobile operating system) system, IOS (a mobile operating system) system, etc., and the present invention is not limited to this; and, in the storage space, one or more instructions suitable for being loaded and executed by the processor are also stored, and these instructions may be one or more computer programs (including program codes). In the embodiment of this specification, the processor loads and executes one or more instructions stored in the memory to implement the lane line detection method provided in the above method embodiment.

[0181] An embodiment of the present application also provides a computer-readable storage medium, which can be set in an electronic device to store at least one instruction or at least one program related to implementing a lane line detection method. The at least one instruction or the at least one program is loaded and executed by the processor to implement any lane line detection method provided by the above method embodiments.

[0182] Optionally, in this embodiment, the above-mentioned storage medium may include but is not limited to: a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk, and other media that can store program codes.

[0183] It should be noted that the above-mentioned sequence of the embodiments of the present application is for description only and does not represent the advantages and disadvantages of the embodiments. The above-mentioned specific embodiments of this specification are described. Other embodiments are within the scope of the attached claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0184] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0185] A person skilled in the art will understand that all or part of the steps to implement the above embodiments may be accomplished by hardware or by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, and the above-mentioned storage medium may be a read-only memory, a disk or an optical disk, etc.

[0186] The above are only preferred embodiments of the present application and are not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application should be included in the protection scope of the present application.

Claims

1. A lane line detection method, characterized in that: The method comprises: Acquire multi-view images acquired by an image acquisition device of a target vehicle; Projecting the multi-view image onto a bird's-eye view plane to form a bird's-eye view corresponding to the multi-view image; Performing feature fusion on the multi-view image and the bird's-eye view to obtain a target feature image; Perform lane line multi-task processing based on the target feature image, determine target pixel points in the target feature image that belong to the lane line area, offset information of the target pixel points relative to the reference line in the target feature image, and the relationship between the target pixel points; Lane line aggregation is performed based on the target pixel point, the offset information corresponding to the target pixel point, and the relationship between the target pixel points to obtain a lane line detection result.

2. The lane line detection method according to claim 1, characterized in that: The step of performing feature fusion on the multi-view image and the bird's-eye view to obtain a target feature image includes: Performing multi-level feature extraction on the multi-view images to obtain multi-level lane line features; The multi-level lane line features are integrated layer by layer to obtain a feature enhancement map corresponding to the multi-view image; Projecting the feature enhancement map onto the bird's-eye view plane and fusing it with the bird's-eye view to obtain a bird's-eye view feature map; Feature extraction and spatial dimension compression are performed on the bird's-eye view feature image to obtain the target feature image.

3. The lane line detection method according to claim 2, characterized in that: The lane line multi-task processing is performed based on the target feature image to determine the target pixel points in the target feature image belonging to the lane line area, the offset information of the target pixel points relative to the reference line in the target feature image, and the relationship between the target pixel points, including: Distinguishing a lane line region from a non-lane line region from the target feature image to obtain a segmentation mask; the segmentation mask indicates the target pixel points in the target feature image that belong to the lane line region; estimating a vertical offset of the target pixel relative to a reference line in the target feature image to obtain the offset information; The target feature image is subjected to feature extraction and dimensionality reduction processing to obtain an embedded feature matrix; the embedded feature matrix represents the relationship between the target pixel points.

4. The lane line detection method according to claim 3, characterized in that: The performing multi-level feature extraction on the multi-view images to obtain multi-level lane line features includes: Obtain a pre-trained lane detection model; the pre-trained lane detection model includes a feature perception framework, and the feature perception framework includes a backbone network; Using the multi-view image as the input of the backbone network to perform multi-level feature extraction on the multi-view image to obtain the multi-level lane line features; Correspondingly, the feature perception framework further includes a neck network; the multi-level lane line features are fused layer by layer to obtain a feature enhancement map corresponding to the multi-view image, including: Using the multi-level lane line features as input of the neck network to fuse the multi-level lane line features layer by layer to obtain the feature enhancement map corresponding to the multi-view image; Correspondingly, the feature perception framework further includes a fusion layer; the projecting of the feature enhancement map onto the bird's-eye view plane and fusing it with the bird's-eye view to obtain a bird's-eye view feature map includes: The feature enhancement map and the bird's-eye view are used as inputs of the fusion layer to project the feature enhancement map onto the bird's-eye view plane and fuse it with the bird's-eye view to obtain the bird's-eye view feature map; Correspondingly, the feature perception framework further includes a coding layer; the step of extracting features and compressing spatial dimensions of the bird's-eye view feature image to obtain the target feature image includes: The bird's-eye view feature map is used as the input of the encoding layer to perform feature extraction and spatial dimension compression on the bird's-eye view feature map to obtain the target feature image.

5. The lane line detection method according to claim 4, characterized in that: The pre-trained lane line detection model further includes a target detection network, and the target detection network includes a segmentation layer; the distinguishing the lane line area from the non-lane line area from the target feature image to obtain a segmentation mask includes: Using the target feature image as an input of the segmentation layer to distinguish a lane line area from a non-lane line area from the target feature image, thereby obtaining the segmentation mask; Correspondingly, the target detection network further includes an offset layer; the estimating the vertical offset of the target pixel relative to the reference line in the target feature image to obtain the offset information includes: Using the target feature image as an input of the offset layer to estimate a vertical offset of the target pixel point belonging to the lane line area in the target feature image relative to a reference line in the target feature image, thereby obtaining the offset information; Correspondingly, the target detection network further includes an embedding layer; the feature extraction and dimensionality reduction processing of the target feature image to obtain an embedded feature matrix includes: The target feature image is used as the input of the embedding layer to perform feature extraction and dimensionality reduction processing on the target feature image to obtain the embedded feature matrix.

6. The lane line detection method according to claim 5, characterized in that: The method further comprises: Acquire multi-view sample images and real lane line information corresponding to the multi-view sample images; Projecting the multi-view sample image onto a bird's-eye view plane to form a sample bird's-eye view corresponding to the multi-view sample image; Inputting the multi-view sample image and the sample bird's-eye view into an initial lane line detection model to obtain predicted lane line information; Determining a model loss of the lane line detection model based on a difference between the predicted lane line information and the actual lane line information; The parameters of the initial lane line detection model are adjusted based on the model loss until a preset training end condition is met to obtain the pre-trained lane line detection model.

7. The lane line detection method according to claim 5, characterized in that: The real lane line information includes a real segmentation mask, real offset information and a real embedded feature matrix, and the predicted lane line information includes a predicted segmentation mask, predicted offset information and a predicted embedded feature matrix; The determining, based on the difference between the predicted lane line information and the actual lane line information, a model loss of the lane line detection model comprises: Determining a segmentation loss based on a difference between the predicted segmentation mask and the true segmentation mask and a degree of overlap between the predicted segmentation mask and the true segmentation mask; determining an offset loss based on a difference between the predicted offset information and the actual offset information; Based on the real embedded feature matrix, determining the proximity between feature vectors belonging to the same lane line in the predicted embedded feature matrix, and determining the distance between feature vectors belonging to different lane lines in the predicted embedded feature matrix; Determine a discrimination loss based on the proximity and the distance; and determine a push-pull loss based on the proximity and the distance; Determining an embedding loss based on the discrimination loss and the push-pull loss; The model loss is determined based on the segmentation loss, the offset loss, and the embedding loss.

8. A lane line detection device, characterized in that: The device comprises: An image acquisition module, used to acquire multi-view images acquired by an image acquisition device of a target vehicle; An image projection module, used for projecting the multi-view image onto a bird's-eye view plane to form a bird's-eye view corresponding to the multi-view image; A feature fusion module, used for performing feature fusion on the multi-view image and the bird's-eye view to obtain a target feature image; a multi-task processing module, configured to perform lane line multi-task processing based on the target feature image, determine target pixel points in the target feature image belonging to the lane line area, offset information of the target pixel points relative to the reference line in the target feature image, and the relationship between the target pixel points; The lane line aggregation module is used to perform lane line aggregation based on the target pixel point, the offset information corresponding to the target pixel point and the relationship between the target pixel points to obtain a lane line detection result.

9. An electronic device, characterized in that: It comprises a processor and a memory, wherein the memory stores at least one instruction or at least one program, and the at least one instruction or the at least one program is loaded and executed by the processor to implement the lane line detection method according to any one of claims 1 to 7.

10. A computer-readable storage medium, wherein at least one instruction or at least one program is stored in the storage medium, and the at least one instruction or the at least one program is loaded and executed by a processor to implement the lane line detection method according to any one of claims 1 to 7.

Images