Lane line detection method and device, electronic equipment and computer readable storage medium
The lane line detection method stabilizes lane line recognition by fusing current and historical bird's-eye view feature maps, addressing jittering and flickering issues in camera-captured images to improve detection accuracy.
Patent Information
- Application Number
- CN202411199010.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-29
- Publication Date
- 2024-12-03
- Estimated Expiration
- 2044-08-29
AI Technical Summary
Lane lines detected from continuous road images captured by cameras exhibit jittering and flickering, leading to unstable recognition due to jumps between adjacent images.
A lane line detection method involving feature fusion of current and historical bird's-eye view feature maps from adjacent road images, integrating lane line features to stabilize detection and improve accuracy.
The method reduces lane line jumps between adjacent images, enhancing the stability and accuracy of lane line detection by predicting and supplementing blocked or lost lane lines.
Smart Images

Figure CN119068437A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of lane line recognition, and specifically to a lane line detection method, device, electronic device and computer-readable storage medium. Background Art
[0002] Lane detection refers to the process of automatically detecting and identifying lane lines on the road. Lane lines are markings on the road that indicate the direction and position of the vehicle. Lane detection can provide prior information for downstream control functions of the vehicle, such as automatic navigation, lane keeping, and lane departure warning. Currently, lane lines can be detected from road images captured by cameras.
[0003] However, there may be jitters, flickers, etc. between the continuous road images taken by the camera, which may cause the lane lines recognized based on the continuous road images to jump, resulting in unstable recognized lane lines. Summary of the invention
[0004] In view of the above, embodiments of the present application provide a lane line detection method, device, electronic device and computer-readable storage medium, which aim to reduce lane line jumping between adjacent road images and improve the stability of identified lane lines.
[0005] In a first aspect, an embodiment of the present application provides a lane line detection method, comprising:
[0006] Get the current road image;
[0007] Converting the perspective of the current road image to a bird's-eye view to obtain a current bird's-eye view feature map;
[0008] Obtaining a historical bird's-eye view feature map of at least one historical road image;
[0009] Wherein, the at least one historical road image and the current road image are adjacent road images collected by photographing the road on which the vehicle is traveling;
[0010] Performing feature fusion on the current bird's-eye view feature map and the historical bird's-eye view feature map to obtain a fused bird's-eye view feature map;
[0011] Lane line detection is performed based on the fused bird's-eye view feature map to obtain a lane line detection result of the current road image.
[0012] The embodiment of the present application fuses the features of the current bird's-eye view feature map and the historical bird's-eye view feature map to obtain a fused bird's-eye view feature map. The fused bird's-eye view feature map can integrate the features of adjacent road images, so that the lane lines identified based on the adjacent road images are smoother, which can effectively reduce the lane line jumps between adjacent road images and improve the stability of the identified lane lines.
[0013] In some embodiments, obtaining a historical bird's-eye view feature map of at least one historical road image includes:
[0014] Obtaining a historical lane line representation of the historical road image;
[0015] Extending the lane line based on the historical lane line representation to obtain a predicted lane line representation group, wherein the predicted lane line representation group includes at least one predicted lane line representation;
[0016] The historical bird's-eye view feature map is determined based on the predicted lane line representation group.
[0017] This embodiment extends the lane lines for the historical lane line representation to obtain a historical bird's-eye view feature map, which can predict and supplement the lane lines that are blocked or lost, thereby improving the detection accuracy of the lane lines in the blocked state.
[0018] In some embodiments, determining the historical bird's-eye view feature map based on the predicted lane line representation group includes:
[0019] In the case where the predicted lane line representation group includes multiple predicted lane line representations, the multiple predicted lane line representations are feature fused to obtain the historical bird's-eye view feature map.
[0020] The embodiment of the present application can integrate various extension situations represented by historical lane lines to obtain a historical bird's-eye view feature map, thereby improving the accuracy of historical lane line prediction.
[0021] In some embodiments, the lane line is extended based on the historical lane line representation to obtain a predicted lane line representation group, including:
[0022] The lane line is extended based on the preset trajectory prediction model and the historical lane line representation to obtain a predicted lane line representation.
[0023] In some embodiments, the fused bird's-eye view feature map includes a plurality of cells, each of which includes at least one pixel point, and the lane line detection is performed based on the fused bird's-eye view feature map to obtain the lane line detection result of the current road image, including:
[0024] Extracting three-dimensional lane information from the fused bird's-eye view feature image;
[0025] The three-dimensional lane information includes: lane line instances, confidence of each cell, offset of each cell and average height of each cell; the confidence of the cell is used to indicate whether there is a lane line in the cell; the offset of the cell is used to indicate the offset between the center of the cell and the lane line in a preset direction;
[0026] The lane line detection result is determined based on the three-dimensional lane information.
[0027] In some embodiments, after extracting the three-dimensional lane information in the fused bird's-eye view feature map, the method further includes:
[0028] Storing the confidence of each cell and the offset in a preset storage space;
[0029] The step of obtaining a historical bird's-eye view feature map of at least one historical road image comprises:
[0030] Acquire the confidence and offset of each cell corresponding to the historical road image from the storage space;
[0031] Determining a historical lane line representation of the historical road image based on the confidence and offset of each cell corresponding to the historical road image;
[0032] The historical bird's-eye view feature map is determined based on the historical lane line representation.
[0033] In some embodiments, converting the perspective of the current road image to a bird's-eye view perspective to obtain a current bird's-eye view feature map includes:
[0034] Based on the current road image and a preset image coding network, a two-dimensional viewing angle feature map is obtained;
[0035] Based on the two-dimensional viewing angle feature map and a preset bird's-eye view feature conversion network, the current bird's-eye view feature map is obtained.
[0036] In a second aspect, an embodiment of the present application further provides a lane line detection device, including:
[0037] An image acquisition module, used to acquire the current road image;
[0038] A perspective conversion module, used to convert the perspective of the current road image to a bird's-eye view to obtain a current bird's-eye view feature map;
[0039] A feature acquisition module, used to acquire a historical bird's-eye view feature map of at least one historical road image;
[0040] Wherein, the at least one historical road image and the current road image are adjacent road images obtained by photographing the road on which the vehicle is traveling;
[0041] A feature fusion module, used for fusing the features of the current bird's-eye view feature map and the historical bird's-eye view feature map to obtain a fused bird's-eye view feature map;
[0042] The detection module is used to perform lane line detection based on the fused bird's-eye view feature map to obtain a lane line detection result of the current road image.
[0043] In a third aspect, an embodiment of the present application further provides an electronic device, comprising a processor and a memory, wherein the memory is used to store instructions, and the processor is used to call the instructions in the memory so that the electronic device executes the lane line detection method described in the first aspect.
[0044] An embodiment of the present application also provides a computer-readable storage medium, which stores computer instructions. When the computer instructions are executed on an electronic device, the electronic device executes the lane line detection method described in the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS
[0045] Figure 1 A schematic diagram of an implementation environment of a lane detection method provided according to an embodiment of the present application.
[0046] Figure 2 The present invention provides a flowchart of the steps of a lane detection method according to an embodiment of the present application.
[0047] Figure 3 A schematic diagram of the model structure of a lane detection model provided according to an embodiment of the present application.
[0048] Figure 4 The flowchart is a sub-step flowchart of step 203 provided according to an embodiment of the present application.
[0049] Figure 5 A schematic diagram of a model structure of a historical bird's-eye view feature generation network provided according to an embodiment of the present application.
[0050] Figure 6 A schematic diagram of a heat map of three-dimensional lane information provided according to an embodiment of the present application.
[0051] Figure 7 A schematic diagram of the training phase of a lane detection model provided according to an embodiment of the present application.
[0052] Figure 8 The present invention is a schematic diagram of the structure of a lane detection device according to an embodiment of the present application.
[0053] Fig. 9 The figure is a schematic diagram of the structure of an electronic device provided according to an embodiment of the present application. DETAILED DESCRIPTION
[0054] In order to more clearly understand the above-mentioned purposes, features and advantages of the present application, the present application is described in detail below in conjunction with the accompanying drawings and specific implementation methods. It should be noted that the implementation methods of the present application and the features in the implementation methods can be combined with each other without conflict.
[0055] In the following description, many specific details are set forth to facilitate a full understanding of the present application. The described implementations are only part of the implementations of the present application, rather than all the implementations.
[0056] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein in the specification of this application are only for the purpose of describing specific embodiments and are not intended to limit this application.
[0057] It should be further noted that, in this article, the terms "comprises", "includes" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the presence of other identical elements in the process, method, article or device including the element.
[0058] In this application, "at least one" means one or more, and "more than one" means two or more than two. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone, where A and B can be singular or plural. The terms "first", "second", "third", "fourth", etc. (if any) in the specification, claims and drawings of this application are used to distinguish similar objects, rather than to describe a specific order or sequence.
[0059] In the embodiments of the present application, words such as "exemplary" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a specific way.
[0060] Lane detection refers to the process of automatically detecting and identifying lane lines on the road. Lane lines are markings on the road that indicate the direction and position of the vehicle. Lane detection can provide prior information for downstream control functions of the vehicle, such as automatic navigation, lane keeping, and lane departure warning. Currently, lane lines can be detected from road images captured by cameras.
[0061] However, there may be jitters, flickers, etc. between the continuous road images taken by the camera, which may cause the lane lines recognized based on the continuous road images to jump, resulting in unstable recognized lane lines.
[0062] In view of the above, embodiments of the present application provide a lane line detection method, device, electronic device, and computer-readable storage medium.
[0063] refer to Figure 1 As shown, Figure 1 A schematic diagram of an implementation environment of a lane line detection method provided in an embodiment of the present application. The implementation environment may include an electronic device and a camera, and the electronic device and the camera may be configured in a vehicle.
[0064] The camera can be used to shoot the road, obtain a road image video, and transmit each frame of the road image in the road image video to an electronic device.
[0065] The electronic device can be used to obtain a current road image; convert the perspective of the current road image to a bird's-eye perspective to obtain a current bird's-eye feature map, and obtain a historical bird's-eye feature map of at least one historical road image, where the at least one historical road image and the current road image are adjacent road images obtained by photographing the road on which the vehicle is traveling; then, feature-fuse the current bird's-eye feature map and the historical bird's-eye feature map to obtain a fused bird's-eye feature map; perform lane line detection based on the fused bird's-eye feature map to obtain a lane line detection result of the current road image.
[0066] The electronic device is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to processors, microprogrammed control units (MCUs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), digital signal processors (DSPs), embedded devices, etc. The electronic device can be a vehicle controller, such as a central controller or a regional controller of a vehicle, but is not limited thereto, and the embodiments of the present application do not limit this.
[0067] Figure 2 This is a flowchart of the steps of an embodiment of the lane line detection method of the present application. According to different requirements, the order of the steps in the flowchart can be changed, and some steps can be omitted.
[0068] See also Figure 2 As shown, the lane line detection method may include the following steps.
[0069] Step 201, obtaining a current road image.
[0070] The current road image refers to the image currently used for lane line detection.
[0071] For example, the vehicle is equipped with a camera, which can be used to shoot the road during driving to obtain a road video. The road video includes multiple frames of road images, and the multiple frames of road images are arranged according to the acquisition sequence of the road images. The electronic device can select road images from the road video in sequence according to the acquisition sequence to perform lane line detection, and use the currently selected road image as the current road image.
[0072] For another example, a camera configured in a lane is used to photograph the road during driving at preset intervals, that is, the camera can capture a road image at preset intervals, and the camera can transmit the road image to an electronic device. The electronic device can perform lane line detection based on the currently received road image, and the currently received road image is the current road image.
[0073] Step 202: convert the perspective of the current road image to a bird's-eye view to obtain a current bird's-eye view feature map.
[0074] The current bird's-eye view feature map is a vectorized representation of the current road displayed by the current road image from a bird's-eye view perspective.
[0075] In some embodiments, the embodiments of the present application may use a lane detection model to convert the perspective of the current road image to a bird's-eye view to obtain the current bird's-eye view feature map. The lane detection model may use the BEV-LaneDet framework.
[0076] For further reference, Figure 3 As shown, the lane line detection model may include: an image coding network and a bird's-eye view feature conversion network. Step 202 may be implemented as follows: the electronic device obtains a two-dimensional viewing feature map based on the current road image and a preset image coding network, and then obtains the current bird's-eye view feature map based on the two-dimensional viewing feature map and a preset bird's-eye view feature conversion network.
[0077] For example, the camera in the vehicle is a monocular camera, and the current road image captured by the camera is a road image from a forward perspective. The preset image encoder can perform feature extraction based on the current road image to obtain a two-dimensional perspective feature map, and the two-dimensional perspective feature map is used to describe the features of the current road displayed by the current road image in the forward perspective; then, based on the preset bird's-eye view feature conversion network, the two-dimensional perspective feature map is converted to a bird's-eye view to obtain the current bird's-eye view feature map.
[0078] The preset image coding network can be created based on a residual neural network, but is not limited thereto. For example, the preset image coding network can be created based on ResNet18.
[0079] The residual neural network is trained on a large-scale detection data set and can construct features of different objects in the image well. The residual neural network can extract features based on the input current road image to obtain a shallow feature map, and then perform secondary deep feature extraction on the shallow feature map through downsampling to obtain a deep feature map; the shallow feature map and the deep feature map are feature fused to obtain a two-dimensional perspective feature map.
[0080] The bird's-eye view feature conversion network can perform spatial transformation, and the bird's-eye view feature conversion network can learn the pixel mapping relationship between the front view feature and the bird's-eye view feature, so as to convert the two-dimensional perspective feature map to the bird's-eye view perspective feature map. The bird's-eye view feature conversion network can be created based on a multi-layer convolutional network, an MLP network, or a transformer, etc., and the embodiments of the present application are not limited to this.
[0081] Step 203: Acquire a historical bird's-eye view feature map of at least one historical road image.
[0082] The at least one historical road image and the current road image are adjacent road images captured by photographing the road on which the vehicle is currently traveling.
[0083] That is, assuming that in step 203 the electronic device obtains historical bird's-eye view feature maps of K historical road images, and the current road image is the t-th road image that performs lane line detection, then the K historical road images are respectively the t-1-th road image that performs lane line detection, the t-2-th road image that performs lane line detection, ..., the tK-th road image that performs lane line detection. The K value can be configured according to actual application requirements, for example, it can be configured to 2 or 3, and the embodiments of the present application are not limited to this.
[0084] The historical bird's-eye view feature map includes the bird's-eye view features of lane lines detected based on the historical road image. The historical bird's-eye view feature map and the current bird's-eye view feature map are located in the same feature space, which is convenient for subsequent feature fusion.
[0085] In some embodiments, step 203 may be implemented in the following manner: a historical bird's-eye view feature map set is created in the electronic device, and the electronic device may retrieve at least one historical bird's-eye view feature map from the historical bird's-eye view feature map set.
[0086] For example, after receiving a frame of road image for lane line detection, the electronic device can convert the perspective of the frame of road image to a bird's-eye view to obtain a bird's-eye feature map, and store the bird's-eye feature map of the road image in a historical bird's-eye feature map set. When the historical bird's-eye feature map is needed, the historical bird's-eye feature can be retrieved from the historical bird's-eye feature map set.
[0087] In some other embodiments, step 203 may also be implemented in the following manner: the electronic device may obtain a historical lane line representation of the at least one historical road image, and then determine the historical bird's-eye view feature map based on the historical lane line representation of the historical road image.
[0088] The embodiment of the present application uses historical lane lines to represent and determine a historical bird's-eye view feature map, so that the historical bird's-eye view feature map can fully reflect the characteristics of the historical lane lines and eliminate redundant information.
[0089] Specifically, refer to Figure 4 As shown, step 203 may include:
[0090] Step 401, obtaining a historical lane line representation of a historical road image.
[0091] The historical lane line representation is used to describe a vector representation of the lane line detected based on the historical road image.
[0092] That is, the historical lane line representation can be determined based on the lane line detection result of the historical road image. The lane line detection result of the historical road image can represent the lane line position of each lane line detected based on the historical road image, and the historical lane line representation can be obtained based on the lane line position.
[0093] For example, refer to Figure 5 As shown, assuming that the current road image is the t-th frame road image, the ti-th frame road image is the historical road image, i∈(1, k), the historical lane line representation of the ti-th frame road image is recorded as Ht-i, and the historical lane line representation of the K historical road images obtained by the electronic device can be recorded as:
[0094]
[0095] After acquiring the historical lane line representation of the historical road image, the electronic device may generate a network based on the historical lane line representation and the historical bird's-eye view feature to obtain the historical bird's-eye view feature of the historical road image.
[0096] For example, the electronic device may convert the historical lane line representation into a bird's-eye view feature to obtain the historical bird's-eye view feature of the historical road image.
[0097] For another example, the electronic device may execute the following steps 402 to 403 to obtain the historical bird's-eye view features of the historical road image.
[0098] Step 402: Extend the lane lines based on the historical lane line representations to obtain a predicted lane line representation group.
[0099] In some embodiments, the historical bird's-eye view feature generation network includes a trajectory prediction model. Step 402 may include: extending the lane line based on the preset trajectory prediction model and the historical lane line representation to obtain a predicted lane line representation.
[0100] Among them, the predicted lane line representation group includes at least one predicted lane line representation.
[0101] The trajectory prediction model can use the lane line represented by the historical lane line representation as a trajectory, and predict various possible extensions of the lane line to obtain a predicted lane line representation group.
[0102] The trajectory prediction model can be configured according to actual application requirements, and the embodiments of the present application are not limited to this.
[0103] Assume that the predicted lane line representation group includes N+1 different extension situations. The j-th predicted lane line representation of the historical lane line representation Ht-i is recorded as, j∈(0,N), then the predicted lane line representation group of the historical lane line representation Ht-i can be recorded as
[0104] For example, in Figure 5 In the above example, the predicted lane line representation group of each historical lane line representation includes three predicted lane line representations: the historical lane line representation Ht- 1The predicted lane line representation group may include:
[0105] Step 403: determine a historical bird's-eye view feature map based on the predicted lane line representation group.
[0106] In some embodiments, when the predicted lane line representation group includes a predicted lane line representation, the electronic device may convert the predicted lane line representation to a bird's-eye view to obtain a historical bird's-eye view feature map.
[0107] In the case where the predicted lane line representation group includes multiple predicted lane line representations, the electronic device may perform feature fusion on the multiple predicted lane line representations to obtain the historical bird's-eye view feature map.
[0108] Specifically, the electronic device can input the multiple predicted lane line representations into a convolutional layer, and the convolutional layer can perform feature fusion on the multiple predicted lane line representations to form a historical bird's-eye view feature map.
[0109] For example, continue to refer to Figure 5 As shown, the electronic device represents the historical lane line H t-1 The predicted lane line represents the group Input the convolutional layer to obtain the historical bird's-eye view feature map F of the t-1th frame road image t-1 .
[0110] The embodiment of the present application can integrate various extension situations represented by historical lane lines to obtain a historical bird's-eye view feature map, thereby improving the accuracy of historical lane line prediction.
[0111] In other embodiments, when the predicted lane line representation group includes multiple predicted lane line representations, one predicted lane line representation is selected from the multiple predicted lane line representations, and a historical bird's-eye view feature map is obtained based on the selected predicted lane line representation.
[0112] This embodiment extends the lane lines based on the historical lane line representation, and can predict and supplement the lane lines that are blocked or lost, thereby improving the detection accuracy of the lane lines in the blocked state.
[0113] For example, during the commuting conditions during rush hour in the city, the lane lines in the road images captured by the camera are severely blocked, or the lane lines are incomplete due to long-term use. The embodiment of the present application takes the above steps 401 to 403 to obtain historical bird's-eye view features, which can output more complete lane lines and provide more accurate and comprehensive prior information for downstream control functions.
[0114] After acquiring the current bird's-eye view feature map and the historical bird's-eye view feature map, the electronic device may execute step 204 .
[0115] Step 204 , feature fusion is performed on the current bird's-eye view feature map and the historical bird's-eye view feature map to obtain a fused bird's-eye view feature map.
[0116] Specifically, the electronic device may perform channel alignment and size alignment on the current bird's-eye view feature map and each historical bird's-eye view feature map, and then perform feature fusion on the current bird's-eye view feature map and the historical bird's-eye view feature map by channel splicing to obtain a fused bird's-eye view feature map.
[0117] For example, continue to refer to Figure 5 As shown, assuming that the historical lane line represents the historical bird's-eye view feature map of Ht-i, it is recorded as F t-i , i∈(0, k), the historical bird's-eye view feature map represented by K historical lane lines can be recorded as {F t-i} i∈(0,k) ; The current bird's-eye view feature map is denoted as F t ; Electronic equipment can {F t-i} i∈(0,k) The historical bird's-eye view feature maps and the current bird's-eye view feature maps F t Channel splicing is performed to obtain a fused bird's-eye view feature map.
[0118] Step 205 , determining a lane line detection result of the current road image based on the fused bird's-eye view feature map.
[0119] The lane line detection result is used to describe the lane line position, such as the three-dimensional coordinates of the lane line.
[0120] In some embodiments, step 205 can be implemented as follows: an electronic device extracts three-dimensional lane information from the fused bird's-eye view feature map, such as extracting information from the fused bird's-eye view feature map based on a three-dimensional lane information extraction network to obtain three-dimensional lane information, and then determining the lane line detection result based on the three-dimensional lane information.
[0121] The three-dimensional lane information extraction network may use a multi-layer convolutional network, such as an upsampling encoder, but is not limited thereto. That is, the electronic device may extract the three-dimensional lane information in the fused bird's-eye view feature map based on the upsampling encoder.
[0122] The fused bird's-eye view feature map includes a plurality of cells, each of which corresponds to at least one pixel point, and the pixel point is used to display a certain position on the driving road from a bird's-eye view.
[0123] The three-dimensional lane information may include: lane line instances, confidence levels of each cell, offsets of each cell, and average heights of each cell.
[0124] The lane line instance is used to distinguish different lane lines in the fused bird's-eye view feature map. For example, the upsampling encoder can predict an embedding vector for each cell, which is used for clustering, and different categories correspond to different lane lines (i.e., lane line instances).
[0125] Continue to refer Figure 6 As shown, lane line instances may be presented in the form of an instantiated heat map 61 .
[0126] The confidence of a cell is used to characterize whether there is a lane line in the cell. That is, the confidence of a cell is used to characterize whether there is a lane line passing through the cell. The confidence can be determined by binary classification. For example, when the confidence is 1, it indicates that there is a lane line in the cell, and when the confidence is 0, it indicates that there is no lane line in the cell. The confidence of each cell can be presented in the form of a key point heat map. That is, the key point heat map includes the confidence of each cell.
[0127] The offset of a cell is used to represent the offset between the center of the cell and the lane line in a preset direction. For example, the preset direction is Figure 6 In the y-axis direction shown in FIG. 1 , a straight line is drawn through the center of the cell toward the y-axis. The straight line intersects the lane line. The relative position of the intersection point and the center of the cell is recorded as Δy, and Δy is the offset. The offset of each cell can be presented in the form of an offset heat map, that is, the offset heat map includes the offset of each cell.
[0128] Continue to refer Figure 6 As shown, Figure 6 The comprehensive heat map 62 shown reflects the confidence and offset of each cell. Figure 6 The shaded cells in the table represent that the confidence level of the cells is 1. Figure 6 The Δy in represents the offset.
[0129] The average height of each cell represents the average height of the driving road corresponding to the cell.
[0130] The electronic device can distinguish cells with lane lines based on the confidence of the cells, distinguish the lane line instances to which cells with lane lines belong based on the lane line instances, determine the position of the lane line passing through the cells in the horizontal plane based on the offset of the cells, and determine the height of the lane line compared to the horizontal plane based on the average height of the cells. Therefore, the horizontal position and height of each lane line (i.e., the lane line detection result) can be obtained through the above-mentioned three-dimensional lane information.
[0131] In some embodiments, after extracting the three-dimensional lane information in the fused bird's-eye view feature map, the electronic device stores the confidence of each cell and the offset in a preset storage space.
[0132] Among them, the confidence of each cell of the storage space (such as a key point heat map) and the offset of each cell (such as an offset heat map) can be used to calculate the lane line representation of the road image.
[0133] That is, in some embodiments, the above step 401 may include: obtaining the confidence and offset of each cell corresponding to the historical road image from the storage space; determining the historical lane line representation of the historical road image based on the confidence and offset of each cell corresponding to the historical road image, for example, performing point fitting based on the confidence and offset of each cell to obtain the historical lane line representation.
[0134] The embodiment of the present application obtains the historical lane line representation based on the confidence and offset of the cell, which can reduce the complexity of the historical lane line representation and reduce the amount of calculation.
[0135] In the embodiment of the present application, each time the confidence and offset of a cell corresponding to a road image are obtained, they are stored in the storage space to facilitate lane line detection based on subsequently obtained road images.
[0136] In an embodiment of the present application, the electronic device can fuse the features of the current bird's-eye view feature map and the historical bird's-eye view feature map to obtain a fused bird's-eye view feature map. The fused bird's-eye view feature map can integrate the features of adjacent road images, so that the lane lines identified based on the adjacent road images are smoother, which can effectively reduce the lane line jumps between adjacent road images and improve the stability of the identified lane lines.
[0137] In addition, this embodiment extends the lane lines for the historical lane line representation to obtain a historical bird's-eye view feature map, which can predict and supplement the lane lines that are blocked or lost, thereby improving the detection accuracy of the lane lines in the blocked state.
[0138] The lane detection method of the embodiment of the present application can be implemented based on a lane line detection model, and the lane detection model can adopt a BEV-LaneDet framework.
[0139] In some embodiments, the training step of the lane line detection model may include: a model training device obtains a three-dimensional lane line loss based on a road video sample and the lane line detection model; and updates a model parameter of the lane line detection model based on the three-dimensional lane line loss.
[0140] Among them, the three-dimensional lane line loss is the sum of confidence loss, offset loss, first lane line instance loss (i.e., embedding vector loss) and lane line height loss.
[0141] Confidence loss can be calculated using the Binary Cross Entropy loss function.
[0142] The offset loss can be calculated using mean square error loss.
[0143] The lane line instance loss can use push-pull loss to reduce the point set loss between the same lane and improve the point set loss between different lanes. For example, minimize the mean of the embedding vectors of the grids belonging to the same lane line and maximize the variance of the embedding vectors of the grids of different lane lines.
[0144] The lane height loss is calculated based on the average height of the detected grid and the height label of the grid.
[0145] The above-mentioned calculation methods of the loss values are only examples. In actual application, other methods can also be used for calculation, and the embodiments of the present application are not limited to this.
[0146] For further reference, Figure 7 As shown, during the training phase of the lane detection model, the lane detection model may also be configured with an auxiliary training branch, which includes a two-dimensional lane information extraction network. The two-dimensional lane information extraction network may use an upsampling decoder, but is not limited thereto.
[0147] After the model training device obtains a fused bird's-eye view feature map based on the road video sample and the lane line detection model, the fused bird's-eye view feature map can be input into the two-dimensional lane information extraction network to obtain a lane line segmentation map and a lane line instance map; the two-dimensional perspective lane line loss is calculated based on the lane line segmentation map and the lane line instance map, and the network parameters of the lane line detection model are updated based on the two-dimensional perspective lane line loss and the three-dimensional lane line loss. For example, the sum of the two-dimensional perspective lane line loss and the three-dimensional lane line loss is used as the comprehensive loss of the lane detection model, and the network parameters of the lane line detection model are updated based on the comprehensive loss.
[0148] The 2D lane line loss includes the sum of the lane line segmentation loss and the second lane line instance loss.
[0149] The lane segmentation loss is obtained based on the detected two-dimensional lane segmentation map and the lane segmentation true value.
[0150] The second lane line instance loss is obtained based on the detected lane line instance image from a two-dimensional perspective and the true value of the lane line instance.
[0151] That is, the total loss of the lane line detection model = confidence loss + offset loss + first lane line instance loss + lane line height loss + lane line segmentation loss + second lane line instance loss.
[0152] The embodiment of the present application adds an auxiliary training structure to train the lane detection model, which can improve the lane detection model's ability to extract lane line features from a two-dimensional perspective.
[0153] Based on the same idea as the lane line detection method in the above embodiment, the present application also provides a lane line detection device, which can be used to execute the above lane line detection method. For ease of explanation, the structural diagram of the lane line detection device embodiment only shows the parts related to the embodiment of the present application. Those skilled in the art can understand that the illustrated structure does not constitute a limitation on the device, and may include more or fewer components than shown in the diagram, or combine certain components, or arrange the components differently.
[0154] As shown in the figure, the lane line detection device includes an image acquisition module 801, a perspective conversion module 802, a feature acquisition module 803, a feature fusion module 804 and a detection module 805. In some embodiments, the above modules can be programmable software instructions stored in a memory and can be called and executed by a processor. It is understood that in other embodiments, the above modules can also be program instructions or firmware solidified in the processor.
[0155] An image acquisition module 801 is used to acquire a current road image;
[0156] A perspective conversion module 802 is used to convert the perspective of the current road image to a bird's-eye view to obtain a current bird's-eye view feature map;
[0157] The feature acquisition module 803 is used to acquire a historical bird's-eye view feature map of at least one historical road image; wherein the at least one historical road image and the current road image are adjacent road images obtained by photographing the road on which the vehicle is traveling;
[0158] A feature fusion module 804 is used to fuse the features of the current bird's-eye view feature map and the historical bird's-eye view feature map to obtain a fused bird's-eye view feature map;
[0159] The detection module 805 is used to perform lane line detection based on the fused bird's-eye view feature image to obtain a lane line detection result of the current road image.
[0160] Fig. 9 This is a schematic diagram of an embodiment of an electronic device of the present application.
[0161] The electronic device 100 includes a memory 20, a processor 30, and a computer program 40 stored in the memory 20 and executable on the processor 30. When the processor 30 executes the computer program 40, the steps in the above lane line detection method embodiment are implemented, for example Figure 2 Steps 201 to 205 are shown.
[0162] Exemplarily, the computer program 40 can also be divided into one or more modules / units, which are stored in the memory 20 and executed by the processor 30. The one or more modules / units can be a series of computer program instruction segments that can perform specific functions, and the instruction segments are used to describe the execution process of the computer program 40 in the electronic device 100. For example, it can be divided into Figure 8 The image acquisition module 801, the perspective conversion module 802, the feature acquisition module 803, the feature fusion module 804 and the detection module 805 are shown.
[0163] Those skilled in the art will understand that the schematic diagram is merely an example of the electronic device 100 and does not constitute a limitation of the electronic device 100. The electronic device 100 may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, the electronic device 100 may also include input and output devices, network access devices, buses, etc.
[0164] The processor 30 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor, a single-chip microcomputer, or the processor 30 may also be any conventional processor, etc.
[0165] The memory 20 can be used to store the computer program 40 and / or the module / unit. The processor 30 realizes various functions of the electronic device 100 by running or executing the computer program and / or the module / unit stored in the memory 20 and calling the data stored in the memory 20. The memory 20 can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area can store data (such as audio data) created according to the use of the electronic device 100, etc. In addition, the memory 20 can include a high-speed random access memory, and can also include a non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), at least one disk storage device, a flash memory device, or other non-volatile solid-state storage devices.
[0166] If the module / unit integrated in the electronic device 100 is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electric carrier signal, telecommunication signal and software distribution medium. It should be noted that the content contained in the computer-readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electric carrier signals and telecommunication signals.
[0167] In the several embodiments provided in this application, it should be understood that the disclosed electronic device and method can be implemented in other ways. For example, the electronic device embodiments described above are only schematic, for example, the division of the units is only a logical function division, and there may be other division methods in actual implementation.
[0168] In addition, each functional unit in each embodiment of the present application may be integrated into the same processing unit, each unit may exist physically separately, or two or more units may be integrated into the same unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of hardware plus software functional modules.
[0169] It is obvious to those skilled in the art that the present application is not limited to the details of the above exemplary embodiments, and that the present application can be implemented in other specific forms without departing from the spirit or basic features of the present application. Therefore, no matter from which point of view, the embodiments should be regarded as exemplary and non-restrictive. In addition, it is obvious that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. Multiple units or electronic devices stated in the electronic device claim can also be implemented by the same unit or electronic device through software or hardware. The words first, second, etc. are used to indicate names, and do not indicate any particular order.
[0170] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present application and are not intended to limit it. Although the present application has been described in detail with reference to the above embodiments, a person of ordinary skill in the art should understand that the technical solution of the present application may be modified or replaced by equivalents without departing from the spirit and scope of the technical solution of the present application.
Claims
1. A lane line detection method, characterized in that: include: Get the current road image; Converting the perspective of the current road image to a bird's-eye view to obtain a current bird's-eye view feature map; Obtaining a historical bird's-eye view feature map of at least one historical road image; Wherein, the at least one historical road image and the current road image are adjacent road images collected by photographing the road on which the vehicle is traveling; Performing feature fusion on the current bird's-eye view feature map and the historical bird's-eye view feature map to obtain a fused bird's-eye view feature map; Lane line detection is performed based on the fused bird's-eye view feature map to obtain a lane line detection result of the current road image.
2. The lane line detection method according to claim 1, characterized in that: The step of obtaining a historical bird's-eye view feature map of at least one historical road image comprises: Obtaining a historical lane line representation of the historical road image; Extending the lane line based on the historical lane line representation to obtain a predicted lane line representation group, wherein the predicted lane line representation group includes at least one predicted lane line representation; The historical bird's-eye view feature map is determined based on the predicted lane line representation group.
3. The lane line detection method according to claim 2, characterized in that: The determining the historical bird's-eye view feature map based on the predicted lane line representation group includes: In the case where the predicted lane line representation group includes multiple predicted lane line representations, the multiple predicted lane line representations are feature fused to obtain the historical bird's-eye view feature map.
4. The lane line detection method according to claim 2, characterized in that: The extending the lane line based on the historical lane line representation to obtain a predicted lane line representation group includes: The lane line is extended based on the preset trajectory prediction model and the historical lane line representation to obtain a predicted lane line representation.
5. The lane line detection method according to any one of claims 1 to 4, characterized in that: The fused bird's-eye view feature map includes a plurality of cells, each of which includes at least one pixel point, and the lane line detection is performed based on the fused bird's-eye view feature map to obtain the lane line detection result of the current road image, including: Extracting three-dimensional lane information from the fused bird's-eye view feature image; The three-dimensional lane information includes: lane line instances, confidence of each cell, offset of each cell and average height of each cell; the confidence of the cell is used to indicate whether there is a lane line in the cell; the offset of the cell is used to indicate the offset between the center of the cell and the lane line in a preset direction; The lane line detection result is determined based on the three-dimensional lane information.
6. The lane line detection method according to claim 5, characterized in that: After extracting the three-dimensional lane information in the fused bird's-eye view feature image, the method further includes: Storing the confidence of each cell and the offset in a preset storage space; The step of obtaining a historical bird's-eye view feature map of at least one historical road image comprises: Acquire the confidence and offset of each cell corresponding to the historical road image from the storage space; Determining a historical lane line representation of the historical road image based on the confidence and offset of each cell corresponding to the historical road image; The historical bird's-eye view feature map is determined based on the historical lane line representation.
7. The lane line detection method according to any one of claims 1 to 4, characterized in that: The step of converting the perspective of the current road image to a bird's-eye view perspective to obtain a current bird's-eye view feature map includes: Based on the current road image and a preset image coding network, a two-dimensional viewing angle feature map is obtained; Based on the two-dimensional viewing angle feature map and a preset bird's-eye view feature conversion network, the current bird's-eye view feature map is obtained.
8. A lane line detection device, characterized in that: include: An image acquisition module, used to acquire the current road image; A perspective conversion module, used to convert the perspective of the current road image to a bird's-eye view to obtain a current bird's-eye view feature map; A feature acquisition module, used to acquire a historical bird's-eye view feature map of at least one historical road image; Wherein, the at least one historical road image and the current road image are adjacent road images obtained by photographing the road on which the vehicle is traveling; A feature fusion module, used for fusing the features of the current bird's-eye view feature map and the historical bird's-eye view feature map to obtain a fused bird's-eye view feature map; The detection module is used to perform lane line detection based on the fused bird's-eye view feature map to obtain a lane line detection result of the current road image.
9. An electronic device, comprising a processor and a memory, characterized in that: The memory is used to store instructions, and the processor is used to call the instructions in the memory, so that the electronic device executes the lane line detection method according to any one of claims 1 to claim 7.
10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and when the computer instructions are executed on an electronic device, the electronic device executes the lane line detection method according to any one of claims 1 to 7.
Citation Information
Patent Citations
Lane line recognition method and device, storage medium and electronic equipment
CN111797658A
Lane line detection method, distance measurement method and corresponding device
CN112949398A
Lane line detection method and system, and computer readable storage medium
CN115049994A
Lane line end-to-end real-time detection method for automatic driving
CN116580369A
