Lane detection methods, devices, electronic equipment and computer-readable storage media

By converting features from current and historical road images to a bird's-eye view and fusing the features, the problem of unstable lane line recognition caused by camera shake is solved, and more stable lane line detection is achieved.

CN119068437BActive Publication Date: 2026-11-10ZHEJIANG GEELY HLDG GRP CO LTD +1
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
CN202411199010.2
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-08-29
Publication Date
2026-11-10
Estimated Expiration
2044-08-29

AI Technical Summary

Technical Problem

In existing technologies, there are jitters and flickering between continuous road images captured by cameras, which leads to unstable lane line recognition and skipping phenomena.

Method used

The features of the current road image and historical road image are transformed to a bird's-eye view and then fused to generate a fused bird's-eye view feature map for lane line detection.

Benefits of technology

By using feature fusion, lane line jumps between adjacent road images are reduced, improving the stability and detection accuracy of lane line recognition, especially in cases of occlusion or loss.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119068437B_ABST
    Figure CN119068437B_ABST
Patent Text Reader

Abstract

This application provides a lane line detection method, apparatus, electronic device, and computer-readable storage medium. The lane line detection method includes: acquiring a current road image; converting the viewpoint of the current road image to a bird's-eye view to obtain a current bird's-eye view feature map; acquiring historical bird's-eye view feature maps of at least one historical road image; wherein the at least one historical road image and the current road image are adjacent road images collected by photographing the road on which the vehicle travels; fusing the current bird's-eye view feature map and the historical bird's-eye view feature map to obtain a fused bird's-eye view feature map; and performing lane line detection based on the fused bird's-eye view feature map to obtain the lane line detection result of the current road image. This application can reduce lane line jumps between adjacent road images and improve the stability of the identified lane lines.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This application relates to the field of lane line recognition, specifically to a lane line detection method, apparatus, electronic device, and computer-readable storage medium. Background Technology

[0002] Lane detection refers to the process of automatically detecting and identifying lane lines on a road. Lane lines are markings on the road used to indicate the direction and position of vehicles. Lane detection provides prior information for downstream vehicle control functions, such as automatic navigation, lane keeping assist, and lane departure warning. Currently, lane lines can be detected from road images captured by cameras.

[0003] However, there may be jitter or flicker between continuous road images captured by the camera, which can cause the lane lines identified based on continuous road images to jump, resulting in unstable lane lines. Summary of the Invention

[0004] In view of the above, embodiments of this application provide a lane line detection method, apparatus, electronic device, and computer-readable storage medium, which aim to reduce lane line jumps between adjacent road images and improve the stability of the identified lane lines.

[0005] In a first aspect, embodiments of this application provide a lane line detection method, including:

[0006] Get the current road image;

[0007] The viewpoint of the current road image is converted to a bird's-eye view to obtain the current bird's-eye feature map;

[0008] Obtain historical bird's-eye view feature maps from at least one historical road image;

[0009] Wherein, the at least one historical road image and the current road image are adjacent road images collected by taking pictures of the road on which the vehicle is traveling;

[0010] The current bird's-eye view feature map and the historical bird's-eye view feature map are fused to obtain a fused bird's-eye view feature map.

[0011] Lane line detection is performed based on the fused bird's-eye view feature map to obtain the lane line detection result of the current road image.

[0012] This application embodiment fuses the current bird's-eye view feature map and the historical bird's-eye view feature map to obtain a fused bird's-eye view feature map. The fused bird's-eye view feature map can integrate the features of adjacent road images, making the lane lines identified based on adjacent road images smoother. This can effectively reduce the jumps in lane lines between adjacent road images and improve the stability of the identified lane lines.

[0013] In some embodiments, obtaining a historical bird's-eye view feature map of at least one historical road image includes:

[0014] Obtain the historical lane line representation of the historical road image;

[0015] Based on the historical lane line representation, the lane lines are extended to obtain a predicted lane line representation group, which includes at least one predicted lane line representation.

[0016] The historical bird's-eye view feature map is determined based on the predicted lane line representation group.

[0017] This embodiment extends the lane lines based on the historical lane line representation to obtain a historical bird's-eye view feature map. This can predict and supplement lane lines in cases of occlusion or loss, thereby improving the detection accuracy of lane lines in occluded states.

[0018] In some embodiments, determining the historical bird's-eye view feature map based on the predicted lane line representation group includes:

[0019] When the predicted lane line representation group includes multiple predicted lane line representations, the multiple predicted lane line representations are fused to obtain the historical bird's-eye view feature map.

[0020] The embodiments of this application can integrate various extensions of historical lane lines to obtain a historical bird's-eye view feature map, thereby improving the accuracy of historical lane line prediction.

[0021] In some embodiments, the lane lines are extended based on the historical lane line representation to obtain a predicted lane line representation group, including:

[0022] Based on the preset trajectory prediction model and the historical lane line representation, the lane lines are extended to obtain the predicted lane line representation.

[0023] In some embodiments, the fused bird's-eye view feature map includes multiple cells, each cell including at least one pixel. The process of detecting lane lines based on the fused bird's-eye view feature map to obtain the lane line detection result of the current road image includes:

[0024] Extract the three-dimensional lane information from the fused bird's-eye view feature map;

[0025] The three-dimensional lane information includes: lane line instances, confidence scores of each cell, offsets of each cell, and average height of each cell; the confidence score of a cell is used to characterize whether a lane line exists in the cell; the offset of a cell is used to characterize the offset of the center of the cell from the lane line in a preset direction;

[0026] The lane line detection result is determined based on the three-dimensional lane information.

[0027] In some embodiments, after extracting the three-dimensional lane information from the fused bird's-eye view feature map, the method further includes:

[0028] The confidence level and offset of each cell are stored in a preset storage space;

[0029] The acquisition of a historical bird's-eye view feature map of at least one historical road image includes:

[0030] The confidence level and offset of each cell corresponding to the historical road image are obtained from the storage space.

[0031] Based on the confidence level and offset of each cell corresponding to the historical road image, the historical lane line representation of the historical road image is determined;

[0032] The historical bird's-eye view feature map is determined based on the historical lane line representation.

[0033] In some embodiments, the viewpoint of the current road image is transformed to a bird's-eye view to obtain a current bird's-eye feature map, including:

[0034] Based on the current road image and the preset image coding network, a two-dimensional view feature map is obtained;

[0035] Based on the two-dimensional view feature map and the preset bird's-eye view feature transformation network, the current bird's-eye view feature map is obtained.

[0036] Secondly, embodiments of this application also provide a lane line detection device, comprising:

[0037] The image acquisition module is used to acquire images of the current road.

[0038] The perspective conversion module is used to convert the perspective of the current road image to a bird's-eye view to obtain the current bird's-eye feature map;

[0039] The feature acquisition module is used to acquire historical bird's-eye view feature maps of at least one historical road image;

[0040] Wherein, the at least one historical road image and the current road image are adjacent road images obtained by taking pictures of the road on which the vehicle is traveling;

[0041] The feature fusion module is used to fuse the current bird's-eye view feature map and the historical bird's-eye view feature map to obtain a fused bird's-eye view feature map.

[0042] The detection module is used to perform lane line detection based on the fused bird's-eye view feature map to obtain the lane line detection result of the current road image.

[0043] Thirdly, embodiments of this application also provide an electronic device, the electronic device including a processor and a memory, the memory being used to store instructions, and the processor being used to call the instructions in the memory, causing the electronic device to execute the lane line detection method described in the first aspect.

[0044] This application also provides a computer-readable storage medium that stores computer instructions that, when executed on an electronic device, cause the electronic device to perform the lane detection method described in the first aspect. Attached Figure Description

[0045] Figure 1 This is a schematic diagram of the implementation environment of the lane line detection method provided according to an embodiment of this application.

[0046] Figure 2 This is a flowchart of the steps of a lane line detection method according to an embodiment of this application.

[0047] Figure 3 This is a schematic diagram of the model structure of a lane detection model provided according to an embodiment of this application.

[0048] Figure 4 This is a flowchart of the sub-steps of step 203 provided according to an embodiment of this application.

[0049] Figure 5 This is a schematic diagram of the model structure of a historical bird's-eye view feature generation network according to an embodiment of this application.

[0050] Figure 6 This is a heat map schematic diagram of three-dimensional lane information provided according to an embodiment of this application.

[0051] Figure 7 This is a schematic diagram of the training phase of a lane detection model provided according to an embodiment of this application.

[0052] Figure 8 This is a schematic diagram of the device structure of a lane line detection device according to an embodiment of this application.

[0053] Figure 9 This is a schematic diagram of the structure of an electronic device provided according to an embodiment of this application. Detailed Implementation

[0054] To better understand the above-mentioned objectives, features, and advantages of this application, the application will be described in detail below with reference to the accompanying drawings and specific embodiments. It should be noted that, unless otherwise specified, the embodiments and features described in these embodiments can be combined with each other.

[0055] The following description sets forth many specific details to provide a full understanding of this application. The described embodiments are only some, not all, of the embodiments of this application.

[0056] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein in the specification of this application is for the purpose of describing particular embodiments only and is not intended to be limiting of this application.

[0057] It should be further noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0058] In this application, "at least one" means one or more, and "more than one" means two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone, where A and B can be singular or plural. The terms "first," "second," "third," "fourth," etc. (if present) in the specification, claims, and drawings of this application are used to distinguish similar objects, not to describe a specific order or sequence.

[0059] In the embodiments of this application, the terms "exemplary" or "for example" are used to indicate that something is an example, illustration, or description. Any embodiment or design that is described as "exemplary" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design. Specifically, the use of the terms "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.

[0060] Lane detection refers to the process of automatically detecting and identifying lane lines on a road. Lane lines are markings on the road used to indicate the direction and position of vehicles. Lane detection provides prior information for downstream vehicle control functions, such as automatic navigation, lane keeping assist, and lane departure warning. Currently, lane lines can be detected from road images captured by cameras.

[0061] However, there may be jitter or flicker between continuous road images captured by the camera, which can cause the lane lines identified based on continuous road images to jump, resulting in unstable lane lines.

[0062] In view of the above, embodiments of this application provide a lane line detection method, apparatus, electronic device, and computer-readable storage medium.

[0063] refer to Figure 1 As shown, Figure 1 This is a schematic diagram illustrating the implementation environment of a lane detection method provided in an embodiment of this application. The implementation environment may include electronic devices and a camera, which may be configured in a vehicle.

[0064] Cameras can be used to capture images and videos of roads, and to transmit each frame of the road images and videos to electronic devices.

[0065] The electronic device can be used to acquire a current road image; convert the view of the current road image to a bird's-eye view to obtain a current bird's-eye view feature map, and acquire a historical bird's-eye view feature map of at least one historical road image, wherein the at least one historical road image and the current road image are adjacent road images obtained by taking pictures of the road on which the vehicle is traveling; then, the current bird's-eye view feature map and the historical bird's-eye view feature map are fused to obtain a fused bird's-eye view feature map; and lane line detection is performed based on the fused bird's-eye view feature map to obtain the lane line detection result of the current road image.

[0066] This electronic device is capable of automatically performing numerical calculations and / or information processing according to pre-set or stored instructions. Its hardware includes, but is not limited to, processors, microprogrammed controllers (MCUs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), digital signal processors (DSPs), and embedded devices. This electronic device can be a vehicle controller, such as a central controller or area controller, but is not limited thereto, and the embodiments of this application do not impose such limitations.

[0067] Figure 2 This is a flowchart illustrating the steps of an embodiment of the lane line detection method of this application. Depending on different requirements, the order of the steps in the flowchart can be changed, and some steps can be omitted.

[0068] See Figure 2 As shown, the lane line detection method may include the following steps.

[0069] Step 201: Obtain the current road image.

[0070] The current road image refers to the image currently used for lane line detection.

[0071] For example, a vehicle is equipped with a camera that can be used to capture the road during driving to obtain road video. The road video includes multiple road images, and these multiple road images are arranged according to the acquisition time sequence of the road images. Electronic devices can select road images from the road video in sequence according to the acquisition time sequence to perform lane line detection, and use the currently selected road image as the current road image.

[0072] For example, a camera positioned in the lane is used to capture images of the road during travel at preset intervals. That is, the camera can capture a road image at preset intervals, and the camera can transmit the road image to an electronic device. The electronic device can then perform lane line detection based on the currently received road image, which is the current road image.

[0073] Step 202: Convert the viewpoint of the current road image to a bird's-eye view to obtain the current bird's-eye feature map.

[0074] The current bird's-eye view feature map is a vectorized representation of the current road as shown in the current road image from a bird's-eye view perspective.

[0075] In some embodiments, the present application may use a lane detection model to transform the view of the current road image to a bird's-eye view to obtain the current bird's-eye feature map. The lane detection model may use the BEV-LaneDet framework.

[0076] Further reference Figure 3 As shown, the lane line detection model may include an image encoding network and a bird's-eye view feature transformation network. Step 202 can be implemented as follows: the electronic device obtains a two-dimensional view feature map based on the current road image and the preset image encoding network, and then obtains the current bird's-eye view feature map based on the two-dimensional view feature map and the preset bird's-eye view feature transformation network.

[0077] For example, the camera in the vehicle is a monocular camera. The current road image captured by the camera is a road image from a forward-looking perspective. A preset image encoder can extract features from the current road image to obtain a two-dimensional view feature map. This two-dimensional view feature map is used to describe the features of the current road as shown in the current road image from a forward-looking perspective. Then, based on a preset bird's-eye view feature transformation network, the two-dimensional view feature map is transformed into a bird's-eye view to obtain the current bird's-eye view feature map.

[0078] This preset image coding network can be created based on a residual neural network, but is not limited to this. For example, this preset image coding network can be created based on ResNet18.

[0079] Residual neural networks, trained on large-scale detection datasets, are capable of effectively constructing features for different objects in images. They can extract features from the input current road image to obtain a shallow feature map, then perform secondary deep feature extraction on the shallow feature map through downsampling to obtain a deep feature map. Finally, the shallow and deep feature maps are fused to obtain a two-dimensional viewpoint feature map.

[0080] This bird's-eye view feature transformation network can perform spatial transformations. It learns the pixel mapping relationship between forward-looking features and bird's-eye view features, thereby transforming a two-dimensional view feature map into a bird's-eye view feature map. This bird's-eye view feature transformation network can be created based on multi-layer convolutional networks, MLP networks, or transformers, etc., and this application embodiment does not limit it in this way.

[0081] Step 203: Obtain a historical bird's-eye view feature map of at least one historical road image.

[0082] Among them, at least one historical road image and the current road image mentioned above are adjacent road images collected by taking pictures of the road the vehicle is currently traveling on.

[0083] That is, assuming that in step 203 the electronic device acquires historical bird's-eye view feature maps of K historical road images, and the current road image is the t-th road image for lane line detection, then the K historical road images are the (t-1)-th road image for lane line detection, the (t-2)-th road image for lane line detection, ..., the tK-th road image for lane line detection. The value of K can be configured according to actual application requirements. For example, it can be configured as 2 or 3. This application embodiment does not limit this.

[0084] The historical bird's-eye view feature map includes the bird's-eye view features of lane lines detected based on the historical road image. The historical bird's-eye view feature map and the current bird's-eye view feature map are located in the same feature space, which facilitates subsequent feature fusion.

[0085] In some embodiments, step 203 can be implemented as follows: a set of historical bird's-eye view feature maps is created in the electronic device, and the electronic device can extract at least one historical bird's-eye view feature map from the set of historical bird's-eye view feature maps.

[0086] For example, after receiving a road image frame for lane line detection, the electronic device can switch the view of the road image frame to a bird's-eye view to obtain a bird's-eye view feature map, and store the bird's-eye view feature map of the road image in a historical bird's-eye view feature map set. When the historical bird's-eye view feature map is needed, it can be retrieved from the historical bird's-eye view feature map set.

[0087] In other embodiments, step 203 can also be implemented by the following method: the electronic device can acquire the historical lane line representation of the at least one historical road image, and then determine the historical bird's-eye view feature map based on the historical lane line representation of the historical road image.

[0088] This application embodiment uses historical lane lines to represent and determine historical bird's-eye view feature maps, so that the historical bird's-eye view feature maps can fully reflect the characteristics of historical lane lines and eliminate redundant information.

[0089] Specifically, refer to Figure 4 As shown, step 203 may include:

[0090] Step 401: Obtain the historical lane line representation of the historical road image.

[0091] Historical lane line representation is used to describe the vector representation of lane lines detected based on the historical road image.

[0092] In other words, the historical lane line representation can be determined based on the lane line detection results of the historical road image. The lane line detection results of the historical road image can characterize the lane line position of each lane line detected based on the historical road image, and the historical lane line representation can be obtained based on the lane line position.

[0093] For example, refer to Figure 5 As shown, assuming the current road image is the t-th frame road image, and the ti-th frame road image is a historical road image, i∈(1,k), the historical lane line representation of the ti-th frame road image is denoted as Ht-i, and the historical lane line representation of the K historical road images acquired by the electronic device can be denoted as:

[0094]

[0095] After acquiring the historical lane line representation of a historical road image, the electronic device can generate a network based on the historical lane line representation and historical bird's-eye view features to obtain the historical bird's-eye view features of the historical road image.

[0096] For example, electronic devices can convert the historical lane line representation into bird's-eye view features to obtain the historical bird's-eye view features of the historical road image.

[0097] For example, the electronic device may perform steps 402 to 403 to obtain the historical bird's-eye view features of the historical road image.

[0098] Step 402: Extend the lane lines based on the historical lane line representation to obtain the predicted lane line representation group.

[0099] In some embodiments, the historical bird's-eye view feature generation network includes a trajectory prediction model. Step 402 may include: extending the lane lines based on the preset trajectory prediction model and the historical lane line representation to obtain a predicted lane line representation.

[0100] The predicted lane line representation group includes at least one predicted lane line representation.

[0101] The trajectory prediction model can use the lane lines represented by historical lane line representations as trajectories to predict various possible extensions of the lane lines, thereby obtaining a set of predicted lane line representations.

[0102] The trajectory prediction model can be configured according to actual application requirements, and the embodiments of this application do not limit it.

[0103] Assuming the predicted lane line representation group includes N+1 different extension scenarios, Let Ht-i be the j-th predicted lane line representation of the historical lane line representation, j∈(0,N). Then the set of predicted lane line representations of the historical lane line representation Ht-i can be denoted as Ht-i.

[0104] For example, in Figure 5 In the context of historical lane line representation, each predicted lane line representation group includes three predicted lane line representations. The predicted lane line representation group for historical lane line representation Ht-1 can include:

[0105] Step 403: Determine the historical bird's-eye view feature map based on the predicted lane line representation group.

[0106] In some embodiments, when the predicted lane line representation group includes a predicted lane line representation, the electronic device can convert the predicted lane representation to a bird's-eye view to obtain a historical bird's-eye view feature map.

[0107] When the predicted lane line representation group includes multiple predicted lane line representations, the electronic device can perform feature fusion on the multiple predicted lane line representations to obtain the historical bird's-eye view feature map.

[0108] Specifically, the electronic device can input the multiple predicted lane line representations into a convolutional layer, which can then perform feature fusion on the multiple predicted lane line representations to form a historical bird's-eye view feature map.

[0109] For example, continue to refer to Figure 5 As shown, the electronic device represents the historical lane lines as H. t-1 Predicted lane line representation group Inputting the convolutional layer yields the historical bird's-eye view feature map F of the road image in frame t-1. t-1 .

[0110] The embodiments of this application can integrate various extensions of historical lane lines to obtain a historical bird's-eye view feature map, thereby improving the accuracy of historical lane line prediction.

[0111] In other embodiments, when the predicted lane line representation group includes multiple predicted lane line representations, a predicted lane line representation is selected from the multiple predicted lane line representations, and a historical bird's-eye view feature map is obtained based on the selected predicted lane line representation.

[0112] This embodiment extends the lane lines based on the historical lane line representation, and can predict and supplement lane lines in cases of occlusion or loss, thereby improving the detection accuracy of lane lines in occluded states.

[0113] For example, during rush hour commuting in the city, the lane lines in the road images captured by the camera are severely obscured, or the lane lines are damaged due to prolonged use. In this embodiment of the application, steps 401 to 403 are taken to obtain historical bird's-eye view features, which can output more complete lane lines and provide more accurate and comprehensive prior information for downstream control functions.

[0114] After acquiring the current bird's-eye view feature map and the historical bird's-eye view feature map, the electronic device can execute step 204.

[0115] Step 204: Perform feature fusion between the current bird's-eye view feature map and the historical bird's-eye view feature map to obtain the fused bird's-eye view feature map.

[0116] Specifically, the electronic device can perform channel alignment and size alignment between the current bird's-eye view feature map and each historical bird's-eye view feature map, and then use channel stitching to fuse the current bird's-eye view feature map and the historical bird's-eye view feature map to obtain the fused bird's-eye view feature map.

[0117] For example, continue to refer to Figure 5 As shown, assuming the historical lane lines represent the historical bird's-eye view feature map of Ht-i, denoted as F... t-i The historical bird's-eye view feature map represented by K historical lane lines, i∈(0,k), can be denoted as {F t-i} i∈(0,k) The current bird's-eye view feature is denoted as F. t Electronic devices can {F t-i} i∈(0,k) Historical aerial view feature maps and current aerial view feature maps F t Channel splicing is performed to obtain a fused bird's-eye view feature map.

[0118] Step 205: Determine the lane line detection results of the current road image based on the fused bird's-eye view feature map.

[0119] Lane detection results are used to describe the position of lane lines, such as the three-dimensional coordinates of the lane lines.

[0120] In some embodiments, step 205 can be implemented as follows: the electronic device extracts the three-dimensional lane information from the fused bird's-eye view feature map, for example, by extracting information from the fused bird's-eye view feature map based on a three-dimensional lane information extraction network to obtain three-dimensional lane information, and then determining the lane line detection result based on the three-dimensional lane information.

[0121] The 3D lane information extraction network can employ a multi-layer convolutional network, such as an upsampling encoder, but is not limited to this. That is, the electronic device can extract the 3D lane information from the fused bird's-eye view feature map based on the upsampling encoder.

[0122] The fused bird's-eye view feature map includes multiple cells, each cell corresponding to at least one pixel, which is used to display a location on the driving road from a bird's-eye view perspective.

[0123] The 3D lane information may include: lane line instances, confidence level of each cell, offset of each cell, and average height of each cell.

[0124] Lane line instances are used to distinguish different lane lines in the fused bird's-eye view feature map. For example, the upsampling encoder can predict an embedding for each cell, which is used for clustering, with different categories corresponding to different lane lines (i.e., lane line instances).

[0125] Continue to refer to Figure 6 As shown, lane line instances can be presented in the form of an instantiated heatmap 61.

[0126] The confidence score of a cell indicates whether a lane line exists within that cell. In other words, the confidence score indicates whether a lane line crosses the cell. This confidence score can be determined using a binary classification method; for example, a confidence score of 1 indicates the presence of a lane line, and a confidence score of 0 indicates the absence of a lane line. The confidence scores of each cell can be presented using a keypoint heatmap. That is, the keypoint heatmap includes the confidence scores of each cell.

[0127] The cell offset represents the amount of deviation between the cell's center and the lane line in a preset direction. For example, the preset direction is as follows: Figure 6 As shown, draw a straight line from the center of the cell towards the y-axis. This line will intersect the lane line. The relative position of the intersection point with the center of the cell is denoted as Δy, which is the offset. The offset of each cell can be presented in the form of an offset heatmap, that is, the offset heatmap includes the offset of each cell.

[0128] Continue to refer to Figure 6 As shown, Figure 6 The comprehensive heatmap 62 shown reflects the confidence level and offset of each cell. Figure 6 A shaded cell indicates that the confidence level of that cell is 1. Figure 6 In this context, Δy represents the offset.

[0129] The average height of each cell represents the average height of the driving road corresponding to that cell.

[0130] Electronic devices can distinguish the cells containing lane lines based on cell confidence, distinguish the lane line instances to which the cells containing lane lines belong based on lane line instances, determine the position of the lane line passing through the cell in the horizontal plane based on cell offset, and determine the height of the lane line relative to the horizontal plane based on the average height of the cell. Therefore, the horizontal position and height of each lane line (i.e., lane line detection result) can be obtained through the above three-dimensional lane information.

[0131] In some embodiments, after extracting the three-dimensional lane information from the fused bird's-eye view feature map, the electronic device stores the confidence level of each cell and the offset in a preset storage space.

[0132] The confidence level (e.g., key point heatmap) and offset of each cell in the storage space (e.g., offset heatmap) can be used to calculate the lane line representation of the road image.

[0133] That is, in some embodiments, step 401 above may include: obtaining the confidence and offset of each cell corresponding to the historical road image from the storage space; determining the historical lane line representation of the historical road image based on the confidence and offset of each cell corresponding to the historical road image, for example, performing point fitting based on the confidence and offset of each cell to obtain the historical lane line representation.

[0134] The embodiments of this application obtain historical lane line representations based on cell confidence and offset, which can reduce the complexity of historical lane line representations and reduce the amount of computation.

[0135] In this embodiment of the application, the confidence level and offset of the cell corresponding to a certain road image are stored in the storage space each time they are obtained, so as to facilitate lane line detection based on the subsequently obtained road images.

[0136] In this embodiment, the electronic device can fuse the current bird's-eye view feature map and the historical bird's-eye view feature map to obtain a fused bird's-eye view feature map. The fused bird's-eye view feature map can integrate the features of adjacent road images, making the lane lines identified based on adjacent road images smoother. This can effectively reduce the jumps in lane lines between adjacent road images and improve the stability of the identified lane lines.

[0137] Furthermore, this embodiment extends the lane lines based on the historical lane line representation to obtain a historical bird's-eye view feature map, which can predict and supplement lane lines in cases of occlusion or loss, thereby improving the detection accuracy of lane lines in occluded states.

[0138] The lane detection method in this application embodiment can be implemented based on a lane line detection model, which can adopt the BEV-LaneDet framework.

[0139] In some embodiments, the training steps of the lane detection model may include: a model training device obtaining a three-dimensional lane loss based on road video samples and the lane detection model; and updating the model parameters of the lane detection model based on the three-dimensional lane loss.

[0140] The 3D lane line loss is the sum of confidence loss, offset loss, first lane line instance loss (i.e., embedding vector loss), and lane line height loss.

[0141] The confidence loss can be calculated using the binary cross entropy loss function.

[0142] Offset loss can be calculated using mean square error loss.

[0143] Lane instance loss can be reduced by push-pull loss to decrease the point set loss within the same lane and increased to increase the point set loss between different lanes. For example, minimize the mean of the embedding vector of the grid belonging to the same lane and maximize the variance of the embedding vector of the grids of different lanes.

[0144] Lane height loss is calculated based on the detected average height of the grid and the height label of that grid.

[0145] The above calculation methods for each loss value are merely examples. In practical applications, other methods may also be used for calculation, and this application does not limit this.

[0146] Further reference Figure 7 As shown, during the training phase of the lane detection model, an auxiliary training branch can also be configured in the lane detection model. This auxiliary training branch includes a two-dimensional lane information extraction network, which can use an upsampling decoder, but is not limited to this.

[0147] After obtaining a fused bird's-eye view feature map based on road video samples and a lane detection model, the model training device can input this fused bird's-eye view feature map into the two-dimensional lane information extraction network to obtain a lane line segmentation map and a lane line instance map. Based on the lane line segmentation map and the lane line instance map, the two-dimensional view lane line loss is calculated. The network parameters of the lane detection model are updated based on the two-dimensional view lane line loss and the three-dimensional lane line loss. For example, the sum of the two-dimensional view lane line loss and the three-dimensional lane line loss is used as the comprehensive loss of the lane detection model, and the network parameters of the lane detection model are updated based on the comprehensive loss.

[0148] The two-dimensional lane line loss includes the sum of lane line segmentation loss and second lane line instance loss.

[0149] The lane segmentation loss is based on the detected two-dimensional lane segmentation map and the lane segmentation truth value.

[0150] The second lane line instance loss is based on the detected lane line instance map from a two-dimensional perspective and the truth value of that lane line instance.

[0151] That is, the total loss of the lane detection model = confidence loss + offset loss + first lane instance loss + lane height loss + lane segmentation loss + second lane instance loss.

[0152] The embodiments of this application add an auxiliary training structure to train the lane detection model, which can improve the lane detection model's ability to extract lane line features from a two-dimensional perspective.

[0153] Based on the same idea as the lane line detection method in the above embodiments, this application also provides a lane line detection device, which can be used to perform the above lane line detection method. For ease of explanation, the structural schematic diagram of the lane line detection device embodiment only shows the parts related to the embodiments of this application. Those skilled in the art will understand that the illustrated structure does not constitute a limitation on the device, and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0154] As shown in the figure, the lane line detection device includes an image acquisition module 801, a view conversion module 802, a feature acquisition module 803, a feature fusion module 804, and a detection module 805. In some embodiments, the above modules can be programmable software instructions stored in memory and executable by a processor. It is understood that in other embodiments, the above modules can also be program instructions or firmware embedded in the processor.

[0155] Image acquisition module 801 is used to acquire the current road image;

[0156] The perspective conversion module 802 is used to convert the perspective of the current road image to a bird's-eye view to obtain the current bird's-eye feature map;

[0157] The feature acquisition module 803 is used to acquire a historical bird's-eye view feature map of at least one historical road image; wherein, the at least one historical road image and the current road image are adjacent road images obtained by taking pictures of the road on which the vehicle is traveling;

[0158] The feature fusion module 804 is used to fuse the current bird's-eye view feature map and the historical bird's-eye view feature map to obtain a fused bird's-eye view feature map.

[0159] The detection module 805 is used to perform lane line detection based on the fused bird's-eye view feature map to obtain the lane line detection result of the current road image.

[0160] Figure 9 This is a schematic diagram of an embodiment of the electronic device of this application.

[0161] The electronic device 100 includes a memory 20, a processor 30, and a computer program 40 stored in the memory 20 and executable on the processor 30. When the processor 30 executes the computer program 40, it implements the steps described in the lane detection method embodiments above, for example... Figure 2 Steps 201 to 205 are shown.

[0162] For example, computer program 40 can also be divided into one or more modules / units, which are stored in memory 20 and executed by processor 30. The one or more modules / units can be a series of computer program instruction segments capable of performing a specific function, which describe the execution process of computer program 40 in electronic device 100. For example, it can be divided into... Figure 8 The image acquisition module 801, viewpoint conversion module 802, feature acquisition module 803, feature fusion module 804, and detection module 805 are shown.

[0163] Those skilled in the art will understand that the schematic diagram is merely an example of the electronic device 100 and does not constitute a limitation on the electronic device 100. It may include more or fewer components than shown in the diagram, or combine certain components, or different components. For example, the electronic device 100 may also include input / output devices, network access devices, buses, etc.

[0164] Processor 30 can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor, a single-chip microcomputer, or any conventional processor.

[0165] The memory 20 can be used to store computer programs 40 and / or modules / units. The processor 30 implements various functions of the electronic device 100 by running or executing the computer programs and / or modules / units stored in the memory 20 and by calling data stored in the memory 20. The memory 20 may mainly include a program storage area and a data storage area. The program storage area may store the operating system, application programs required for at least one function (such as sound playback function, image playback function, etc.), etc.; the data storage area may store data created according to the use of the electronic device 100 (such as audio data), etc. In addition, the memory 20 may include high-speed random access memory, and may also include non-volatile memory, such as hard disk, RAM, plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, at least one disk storage device, flash memory device, or other non-volatile solid-state storage device.

[0166] If the modules / units integrated in the electronic device 100 are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include: any entity or device capable of carrying the computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in the computer-readable medium can be appropriately added or removed according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electrical carrier signals and telecommunication signals.

[0167] In the several embodiments provided in this application, it should be understood that the disclosed electronic devices and methods can be implemented in other ways. For example, the electronic device embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and other division methods may be used in actual implementation.

[0168] Furthermore, the functional units in the various embodiments of this application can be integrated into the same processing unit, or each unit can exist physically separately, or two or more units can be integrated into the same unit. The integrated units described above can be implemented in hardware or in the form of hardware plus software functional modules.

[0169] It will be apparent to those skilled in the art that this application is not limited to the details of the exemplary embodiments described above, and that this application can be implemented in other specific forms without departing from the spirit or essential characteristics of this application. Therefore, the embodiments should be considered exemplary and not restrictive in all respects. Furthermore, it is clear that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. Multiple units or electronic devices recited in the electronic device claims may also be implemented by the same unit or electronic device through software or hardware. The terms "first," "second," etc., are used to indicate names and do not indicate any particular order.

[0170] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application and are not intended to limit it. Although this application has been described in detail with reference to the above embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of this application without departing from the spirit and scope of the technical solutions of this application.

Claims

1. A lane line detection method, characterized in that, include: Get the current road image; The viewpoint of the current road image is converted to a bird's-eye view to obtain the current bird's-eye feature map; Obtain historical bird's-eye view feature maps from at least one historical road image; Wherein, the at least one historical road image and the current road image are adjacent road images collected by taking pictures of the road on which the vehicle is traveling; The current bird's-eye view feature map and the historical bird's-eye view feature map are fused to obtain a fused bird's-eye view feature map. Lane line detection is performed based on the fused bird's-eye view feature map to obtain the lane line detection result of the current road image; The step of obtaining a historical bird's-eye view feature map of at least one historical road image includes: Obtain the historical lane line representation of the historical road image; Based on the historical lane line representation, the lane lines are extended to obtain a predicted lane line representation group, which includes at least one predicted lane line representation. The historical bird's-eye view feature map is determined based on the predicted lane line representation group.

2. The lane line detection method as described in claim 1, characterized in that, Determining the historical bird's-eye view feature map based on the predicted lane line representation group includes: When the predicted lane line representation group includes multiple predicted lane line representations, the multiple predicted lane line representations are fused to obtain the historical bird's-eye view feature map.

3. The lane line detection method as described in claim 1, characterized in that, The process of extending lane lines based on the historical lane line representation to obtain a predicted lane line representation group includes: Based on the preset trajectory prediction model and the historical lane line representation, the lane lines are extended to obtain the predicted lane line representation.

4. The lane line detection method according to any one of claims 1 to 3, characterized in that, The fused bird's-eye view feature map includes multiple cells, each cell including at least one pixel. Lane detection based on the fused bird's-eye view feature map to obtain the lane detection result of the current road image includes: Extract the three-dimensional lane information from the fused bird's-eye view feature map; The three-dimensional lane information includes: lane line instances, confidence scores of each cell, offsets of each cell, and average height of each cell; the confidence score of a cell is used to characterize whether a lane line exists in the cell; the offset of a cell is used to characterize the offset of the center of the cell from the lane line in a preset direction; The lane line detection result is determined based on the three-dimensional lane information.

5. The lane line detection method as described in claim 4, characterized in that, After extracting the 3D lane information from the fused bird's-eye view feature map, the method further includes: The confidence level and offset of each cell are stored in a preset storage space; The acquisition of a historical bird's-eye view feature map of at least one historical road image includes: The confidence level and offset of each cell corresponding to the historical road image are obtained from the storage space. Based on the confidence level and offset of each cell corresponding to the historical road image, the historical lane line representation of the historical road image is determined; The historical bird's-eye view feature map is determined based on the historical lane line representation.

6. The lane line detection method according to any one of claims 1 to 3, characterized in that, The step of converting the viewpoint of the current road image to a bird's-eye view to obtain the current bird's-eye feature map includes: Based on the current road image and the preset image coding network, a two-dimensional view feature map is obtained; Based on the two-dimensional view feature map and the preset bird's-eye view feature transformation network, the current bird's-eye view feature map is obtained.

7. A lane line detection device, characterized in that, include: The image acquisition module is used to acquire images of the current road. The perspective conversion module is used to convert the perspective of the current road image to a bird's-eye view to obtain the current bird's-eye feature map; The feature acquisition module is used to acquire historical bird's-eye view feature maps of at least one historical road image; Wherein, the at least one historical road image and the current road image are adjacent road images obtained by taking pictures of the road on which the vehicle is traveling; The feature fusion module is used to fuse the current bird's-eye view feature map and the historical bird's-eye view feature map to obtain a fused bird's-eye view feature map. The detection module is used to perform lane line detection based on the fused bird's-eye view feature map to obtain the lane line detection result of the current road image; The step of obtaining a historical bird's-eye view feature map of at least one historical road image includes: Obtain the historical lane line representation of the historical road image; Based on the historical lane line representation, the lane lines are extended to obtain a predicted lane line representation group, which includes at least one predicted lane line representation. The historical bird's-eye view feature map is determined based on the predicted lane line representation group.

8. An electronic device, the electronic device comprising a processor and a memory, characterized in that, The memory is used to store instructions, and the processor is used to call the instructions in the memory to cause the electronic device to perform the lane detection method as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that, when executed on an electronic device, cause the electronic device to perform the lane detection method as described in any one of claims 1 to 6.

Citation Information

Patent Citations

  • Lane line detection method and system, and computer readable storage medium

    CN115049994A

  • Lane line end-to-end real-time detection method for automatic driving

    CN116580369A