Lane detection system and method for a vehicle

Abstract

The present application relates to a lane detection system (100) for a vehicle for determining a current lane on which a vehicle is driving on a multi-lane road. The lane detection system (100) comprises processing circuitry configured to: obtain a current image of the multi-lane road (i.e. a driving scene); determine a first candidate current lane, i.e. a first estimate of the current lane, by taking a leftmost lane in the current image as a first reference lane; determine a second candidate current lane, i.e. a second estimate of the current lane, by taking a rightmost lane in the current image of the multi-lane road as a second reference lane; and determine the current lane of the vehicle in the current image of the multi-lane road based on the first candidate current lane and the second candidate current lane. Furthermore, the present application also relates to a corresponding lane detection method.

Description

Technical Field

[0001] The present invention relates to image processing and, more particularly, to a lane detection system and method, for example, as a component in an advanced driver assistance system for a vehicle. Background Art

[0002] Advanced driver assistance systems (ADAS), which can alert drivers to dangerous situations or actively participate in driving, are gradually being incorporated into vehicles. These systems, also known as autonomous driving systems, are expected to become increasingly complex and sophisticated in the future, ultimately enabling fully autonomous driving.

[0003] One challenge facing autonomous driving systems is determining the lane the car is traveling in (i.e., lane ID) and the total number of lanes on the road (i.e., lane count), as this information is crucial for several related technologies, including localization and mapping, safe path planning, and many other tasks. In the context of mapping, for example, lane ID and lane count represent additional valuable features that can be added and stored as other map features. Similarly, information about lane ID and lane count improves the accuracy of the localization step. For safe path planning, accurately determining the number of lanes on the road is a crucial prerequisite for accurately estimating one or more safe paths.

[0004] A known method for accurately estimating the current lane of travel has been published in "LaneQuest: An Accurate and Energy-Efficient Lane Detection System" by Heba Aly, Anas Basalamah, and Moustafa Youssef at IEEE PerCom 2015. This paper relies on inertial sensors commonly used in commercial smartphones to provide an accurate estimate of the car's current lane based on information about the car's surroundings without any visual input.

[0005] Knoop, Victor L., et al., in "Lane determination with GPS precise point positioning," published in IEEE Transactions on Intelligent Transportation Systems 18.9 (2017), pages 2503–2513, introduced a GPS-based method called GPS-PPP. This method can accurately locate vehicles on multi-lane highways in real time with sub-meter accuracy.

[0006] Dao, Thanh-Son et al., “Markov-based lane positioning using intervehicle communication,” IEEE Transactions on Intelligent Transportation Systems 8.4 (2007), pp. 641–650, disclose an alternative Markov-based lane-level positioning scheme. This scheme exploits the connectivity between neighboring vehicles participating in traffic and exchanging information within a certain range to achieve mutual precise positioning.

[0007] In addition, there are some proposals to use visual cues to perform lane-level localization of autonomous vehicles. For example, Dixiao Cui, Jianru Xue, and Nanning Zheng published "Real-time global localization of robotic cars in lane level via lane marking detection and shape registration" on pages 1039-1050 in IEEE Transactions on Intelligent Transportation Systems 17.4 (2015). They proposed an accurate real-time localization method for robotic cars in urban environments. This method uses a robust lane marking detection algorithm and an efficient shape registration algorithm between the detected lane markings and the GPS-based prior road shape to improve the robustness and accuracy of the global localization of the robotic car.

[0008] Nedevschi, Sergiu, et al., in their paper “3D lane detection system based on stereovision,” published in the 2004 IEEE Proceedings of the Seventh International Conference on Intelligent Transportation Systems (IEEE Catalog No. 04TH8749), introduced a 3D lane detection method based on stereovision. The availability of 3D information allows for the separation of road and obstacle features. Lanes are then modeled as 3D surfaces, whose current parameters are predicted based on past information and vehicle dynamics.

[0009] Although the above conventional methods have provided some advantages, there is still room for improvement. Therefore, an improved lane detection system and corresponding method are needed. Summary of the Invention

[0010] The object of the present invention is to provide an improved lane detection system and a corresponding method.

[0011] The above and other objects are achieved by the subject matter claimed in the independent claims. Other implementations are apparent from the dependent claims, the detailed description and the drawings.

[0012] The following abbreviations and definitions will be used herein to describe the present invention in more detail:

[0013] DNN Deep Neural Network

[0014] CNN Convolutional Neural Network

[0015] RNN Recurrent Neural Network

[0016] Lane ID The lane identifier or number of the lane the car is currently traveling in

[0017] Lane count: The total number of lanes on a road.

[0018] LSTM is a long-short term memory unit that is responsible for gating past information in a neural network.

[0019] ConvLSTM is an LSTM unit that can process 2D tensors (such as images).

[0020] Image: A digital camera's visual representation of a real-world or synthetic scene, also known as a picture.

[0021] Pixel: The smallest addressable image / graphic element.

[0022] Scene The surrounding environment relative to a reference, for example, a camera's scene is the part of the environment visible from the camera.

[0023] Semantic Segmentation is a method that divides an image into different regions based on semantic attribution. For example, pixels depicting cars are all red, pixels depicting roads are all blue, and so on.

[0024] Machine Learning is a field of study that focuses on analyzing and learning from input data to build models that can make predictions.

[0025] Deep Learning is a field of study that is part of machine learning and is used when the model is a neural network with a deep architecture and a high-cardinality parameter set.

[0026] Artificial Neural Networks (ANNs) are a subfield of machine learning within the context of deep learning, driven by biological neural networks. The goal of an artificial network is to estimate a function with a large number of inputs by adaptively learning a set of connection weights.

[0027] Convolution is a mathematical operation that uses an integral operation to calculate the overlap of two functions, where one function is inverted and shifted.

[0028] Convolutional Neural Network: An artificial neural network whose architecture includes at least one convolutional layer.

[0029] Autonomous driving occurs when the driving task is partially achieved to a certain level or is fully achieved by the vehicle itself without driver intervention.

[0030] Convention: The left or right convention corresponding to the side of the road where the lane counting operation starts. From left to right is "left convention", and from right to left is "right convention".

[0031] Confidence value is a scalar value between 0 and 1 that indicates the model's confidence in the correctness of the estimate (1 means high confidence, 0 means no confidence).

[0032] In general, embodiments of the present invention relate to an end-to-end deep learning based approach for estimating lane ID and lane number. Embodiments of the present invention are based on a visual representation (image) of the driving scene and exploit the time dimension by deploying LSTM units in a convolutional neural network. Embodiments of the present invention can understand which lane the car is driving in, especially in extreme cases where there is not enough visual content. Embodiments of the present invention are based on the idea of ​​approaching the task from two different perspectives (i.e. the left and right side of the road) and exploiting other information to improve the estimate using the more reliable side with more visual information and better features for estimating the current lane.

[0033] More specifically, according to a first aspect, the present invention relates to a lane detection system for a vehicle, for determining a current lane in which the vehicle is traveling on a multi-lane road. The lane detection system includes processing circuitry configured to: obtain a current image of the multi-lane road (i.e., a driving scene); determine a first candidate current lane, i.e., a first estimate of the current lane, by using the leftmost lane in the current image as a first reference lane; determine a second candidate current lane, i.e., a second estimate of the current lane, by using the rightmost lane in the current image of the multi-lane road as a second reference lane; and determine the current lane of the vehicle in the current image of the multi-lane road based on the first candidate current lane and the second candidate current lane.

[0034] Advantageously, the lane detection system according to the first aspect provides a holistic approach that considers all available visual features / cues in the input image by relying on a first lane ID estimate based on the leftmost lane in which the vehicle is traveling on the multi-lane road, and a second lane ID estimate based on the rightmost lane. This more robust approach can handle challenging situations, such as obstructions caused by nearby cars / trucks, changing weather conditions, daytime or nighttime scenes, varying lighting conditions, highway or urban scenes, and several other scenarios that can impact the accuracy of the method.

[0035] In another possible implementation of the first aspect, the processing circuit is further configured to determine the total number of lanes of the multi-lane road based on the current image of the multi-lane road.

[0036] In another possible implementation of the first aspect, the processing circuit is used to determine the first candidate current lane as a first set of likelihood measurement values ​​(e.g., a first likelihood vector) and to determine the second candidate current lane as a second set of likelihood measurement values ​​(e.g., a second likelihood vector), wherein the first set of likelihood measurement values ​​includes corresponding likelihood measurement values ​​for each lane in which the car is traveling on the multi-lane road, and the second set of likelihood measurement values ​​includes corresponding likelihood measurement values ​​for each lane in which the car is traveling on the multi-lane road.

[0037] In another possible implementation of the first aspect, the processing circuit for determining the current lane of the vehicle in the current image of the multi-lane road based on the first candidate current lane and the second candidate current lane is used to: determine the first candidate current lane as the lane associated with the maximum likelihood metric value in the first group of likelihood metric values; determine the second candidate current lane as the lane associated with the maximum likelihood metric value in the second group of likelihood metric values; and determine the first candidate current lane or the second candidate current lane as the current lane of the vehicle in the current image by comparing the difference between the maximum likelihood metric value and the average likelihood metric value in the first group of likelihood metric values ​​with the difference between the maximum likelihood metric value and the average likelihood metric value in the first group of likelihood metric values.

[0038] In another possible implementation of the first aspect, the processing circuit is used to acquire multiple time-continuous images of the multi-lane road, wherein the multiple time-continuous images include the current image and one or more previous images; and the processing circuit is used to determine the current lane of the vehicle in the current image of the multi-lane road based on the first candidate current lane, the second candidate current lane, and one or more previously determined lanes of the vehicle in the one or more previous images of the multi-lane road.

[0039] In another possible implementation of the first aspect, the processing circuit is further used to: determine whether the vehicle is moving; and if the vehicle is not moving, determine the current lane of the vehicle in the current image of the multi-lane road based on the one or more previously determined lanes of the vehicle in the previous one or more images.

[0040] In another possible implementation of the first aspect, the processing circuit is configured to determine whether the vehicle is moving based on the one or more previously determined lanes of the vehicle in the previous one or more images.

[0041] In another possible implementation of the first aspect, the processing circuit is configured to implement a convolutional neural network configured to determine the first candidate current lane, the second candidate current lane, and the current lane using one or more convolutional neural network weights. In one implementation, the convolutional neural network includes one or more LSTM units.

[0042] In another possible implementation of the first aspect, the convolutional neural network is used to train the convolutional neural network weights using a cost function, wherein the cost function is based on a cross-entropy loss function, and the cost function also depends on the distance between a lane and the leftmost lane and / or the rightmost lane in the current image of the multi-lane road, and / or the cost function implements a constraint condition, namely that the first candidate current lane and the second candidate current lane are related to the total number of lanes on the multi-lane road.

[0043] In another possible implementation of the first aspect, the processing circuit is further used to implement a semantic segmentation network, which is used to semantically segment the current image of the multi-lane road, and the processing circuit is used to determine the current lane of the vehicle in the current image of the multi-lane road based on the first candidate current lane, the second candidate current lane and the current image after semantic segmentation.

[0044] In another possible implementation of the first aspect, the processing circuit is further configured to generate a first confidence value for the first candidate current lane and a second confidence value for the second candidate current lane.

[0045] In another possible implementation of the first aspect, the processing circuit is used to: flip the image of the multi-lane road along the longitudinal axis to obtain a flipped image of the multi-lane road; determine a third candidate current lane, i.e., a third estimated value of the current lane, by using the leftmost lane in the flipped image of the multi-lane road as the first reference lane; determine a fourth candidate current lane, i.e., a fourth estimated value of the current lane, by using the rightmost lane in the flipped image of the multi-lane road as the second reference lane; and determine the current lane of the vehicle based on the first candidate current lane, the second candidate current lane, and the constraints that the first candidate current lane is equal to the fourth candidate current lane and the second candidate current lane is equal to the third candidate current lane.

[0046] In another possible implementation of the first aspect, the processing circuit is used to obtain multiple time-continuous images of the multi-lane road, wherein the multiple time-continuous images include the current image; and the processing circuit is used to adjust the brightness of the current image based on the average brightness of the multiple time-continuous images of the multi-lane road.

[0047] In another possible implementation of the first aspect, the lane detection system further includes a camera for capturing the image of the multi-lane road.

[0048] According to a second aspect, the present invention relates to a vehicle comprising the lane detection system provided by the first aspect of the present invention.

[0049] According to a third aspect, the present invention relates to a method for determining the current lane in which a vehicle is traveling on a multi-lane road. The method comprises the following steps:

[0050] Acquire a current image of the multi-lane road (i.e., the driving scene);

[0051] determining a first candidate current lane, i.e., a first estimate of the current lane, by using a leftmost lane in the current image of the multi-lane road as a first reference lane;

[0052] determining a second candidate current lane, i.e., a second estimate of the current lane, by using a rightmost lane in the current image of the multi-lane road as a second reference lane;

[0053] The current lane of the vehicle in the current image of the multi-lane road is determined based on the first candidate current lane and the second candidate current lane.

[0054] The lane detection method provided in the second aspect of the present invention can be performed by the lane detection system provided in the first aspect of the present invention. Other features of the lane detection method provided in the third aspect of the present invention are directly derived from the functionality of the lane detection system provided in the first aspect of the present invention and the different implementations described in the context thereof.

[0055] According to a fourth aspect, the present invention relates to a computer program product. The computer program product includes a non-transitory computer-readable storage medium carrying program code. When the program code is executed by a computer or a processor, the computer or the processor performs the lane detection method provided in the third aspect.

[0056] The following drawings and description set forth in detail one or more embodiments. Other features, objects, and advantages are apparent from the description, drawings, and claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] Embodiments of the present invention are described in more detail below with reference to the accompanying drawings and schematic diagrams.

[0058] Figure 1 A schematic diagram of processing blocks implemented by a lane detection system for a vehicle during a training phase is provided for one embodiment of the present invention.

[0059] Figure 2 A schematic diagram of different aspects implemented in a lane detection system according to an embodiment of the present invention.

[0060] Figure 3 A schematic diagram of the architecture of a neural network implemented by a lane detection system for a vehicle according to one embodiment of the present invention.

[0061] Figure 4 A schematic diagram of different aspects implemented in a lane detection system according to an embodiment of the present invention.

[0062] Figure 5 A schematic diagram of the architecture of a neural network implemented by a lane detection system for a vehicle according to one embodiment of the present invention.

[0063] Figure 6 A schematic diagram of the architecture of a neural network implemented by a lane detection system for a vehicle according to one embodiment of the present invention.

[0064] Figure 7 A schematic diagram of the architecture of a neural network implemented by a lane detection system for a vehicle according to one embodiment of the present invention.

[0065] Figure 8 A schematic diagram of processing blocks implemented by a lane detection system for a vehicle during an application phase according to an embodiment of the present invention is provided.

[0066] Figure 9 A schematic diagram of different aspects implemented in a lane detection system according to an embodiment of the present invention.

[0067] Figure 10 A schematic diagram of processing blocks implemented by a lane detection system for a vehicle during an application phase according to an embodiment of the present invention is provided.

[0068] Figure 11 A schematic diagram of different aspects implemented in a lane detection system according to an embodiment of the present invention.

[0069] Figure 12 A schematic diagram of different aspects implemented in a lane detection system according to an embodiment of the present invention.

[0070] Figures 13a to 13d An exemplary image of a multi-lane road and lane estimation values ​​provided by a lane detection system according to an embodiment of the present invention are shown.

[0071] Figure 14 A flowchart of a lane detection method provided by one embodiment of the present invention.

[0072] In the following, identical reference numerals denote identical features or at least functionally equivalent features. DETAILED DESCRIPTION

[0073] In the following description, reference is made to the accompanying drawings that form a part hereof and that illustrate, by way of illustration, specific aspects of embodiments of the invention or in which embodiments of the invention may be used. It should be understood that embodiments of the invention may be used in other aspects and may include structural or logical variations not depicted in the accompanying drawings. Therefore, the following detailed description should not be construed in a limiting sense, and the scope of the invention is defined by the appended claims.

[0074] For example, it should be understood that the disclosure relating to describing a method may also apply to a corresponding device or system for performing the method, and vice versa. For example, if one or more specific method steps are described, the corresponding device may include one or more units (e.g., functional units) to perform the one or more method steps described (e.g., one unit performs one or more steps, or multiple units perform one or more of the multiple steps respectively), even if such one or more units are not explicitly described or illustrated in the accompanying drawings. On the other hand, for example, if a specific device is described in terms of one or more units (e.g., functional units), the corresponding method may include a step to implement the function of the one or more units (e.g., one step to implement the function of the one or more units, or multiple steps to implement the function of one or more of the multiple units respectively), even if the one or more steps are not explicitly described or illustrated in the accompanying drawings. In addition, it should be understood that, unless otherwise expressly stated, the features of the various exemplary embodiments and / or aspects described herein may be combined with each other.

[0075] An embodiment of a lane detection system 100 in a vehicle is described in more detail below with reference to the accompanying drawings. Lane detection system 100 is used to determine the current lane in which the vehicle is traveling on a multi-lane road. Lane detection system 100 may be part of an advanced driver assistance system (ADAS) implemented in the vehicle.

[0076] Typically, as described in detail below, lane detection system 100 includes processing circuitry configured to: acquire a current image of a multi-lane road (i.e., a driving scene); determine a first candidate current lane, i.e., a first estimate of the current lane, by using the leftmost lane in the current image as a first reference lane; determine a second candidate current lane, i.e., a second estimate of the current lane, by using the rightmost lane in the current image of the multi-lane road as a second reference lane; and determine the current lane of the vehicle in the current image of the multi-lane road based on the first candidate current lane and the second candidate current lane. The processing circuitry may include hardware and software. The hardware may include digital circuitry or both analog and digital circuitry. The digital circuitry may include components such as an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), a digital signal processor (DSP), or a general-purpose processor. In one embodiment, the processing circuitry includes one or more processors and non-volatile memory coupled to the one or more processors. The non-volatile memory may carry executable program code. When the executable program code is executed by the one or more processors, it causes the apparatus to perform the operations or methods described herein.

[0077] In one embodiment, the processing circuit in the lane detection system 100 is used to implement convolutional neural networks 109, 110, which are used to determine the first candidate current lane, the second candidate current lane, and the current lane using one or more convolutional neural network weights. Figure 1 FIG. 1 shows a processing block implemented by a processing circuit in the lane detection system 100 during the training phase of the convolutional neural networks 107 and 109 according to an embodiment. Figure 8 and Figure 10 1 shows processing blocks implemented by processing circuitry in lane detection system 100 within the application or inference phase of convolutional neural network 109 , according to various embodiments.

[0078] exist Figure 1 In processing block 103, the processing circuitry in lane detection system 100 is configured to obtain or capture a visual representation of the driving scene. To this end, in one embodiment, lane detection system 100 may include or be connected to a camera employed in a vehicle for capturing images of a multi-lane road (i.e., the driving scene). Figure 1In addition to the input image obtained in the processing block 103 of the lane detection system 100, the ground truth label of the correct lane ID (i.e., the current lane number) and the lane number is also obtained by the processing circuit in the lane detection system 100. Figure 1 For example, in the processing block 101 Figure 1 In the illustrated processing block 101 , the correct left side ID is obtained by counting from the leftmost lane on a multi-lane road. Using this "left side convention," the left side ID is lane number 1. However, it will be appreciated that the processing block 101 may also obtain the correct right side ID based on the rightmost lane on a multi-lane road.

[0079] The "left side convention" and "right side convention" used here are Figure 2 This example is illustrated in Figure 1. In this example, there are five lanes on a multi-lane road. According to the "left side convention", the car is currently traveling in lane number 4 (i.e., the left lane ID), while according to the "right side convention", the car is currently traveling in lane number 2 (i.e., the right lane ID). It can be understood that there is a simple relationship between the left lane ID, the right lane ID, and the total number of lanes on the multi-lane road. This relationship is used to Figure 1 In the processing block 105, the right lane ID is determined based on the left lane ID and the number of lanes (ie, the total number of lanes) provided by the processing block 101.

[0080] Will be in Figure 1 The training input image of the driving scene obtained in processing block 103 is provided to the convolutional neural network 109 to estimate the left lane ID (lane counted from the leftmost lane on the road), the right lane ID (lane counted from the rightmost lane on the road), and the total number of lanes. According to one embodiment, processing block 103 can be configured to use an image sequence extracted from a video of a moving car. However, according to one embodiment, only a single image is processed during each forward pass.

[0081] Figure 3 One embodiment of the architecture of the neural network 109 implemented by the processing circuitry in the lane detection system 100 is shown. Figure 3 In the exemplary embodiment shown, the neural network 109 is based on a "Moka" architecture. It is understood that the Moka architecture consists of an encoder part and a decoder part, wherein the encoder part gradually extracts high-level features by downsampling the image / feature map, while the decoder part is able to restore the full resolution. In addition, the architecture can be designed so that the corresponding parts in the encoder and decoder are connected together using long-range links. Finally, in Figure 3In the exemplary architecture shown, three fully connected blocks can be linked to the network to generate the three required scalars corresponding to the left / right lane ID estimates and the lane number estimates. According to one embodiment, the neural network 109 is equipped with convolutional LSTM (ConvLSTM) units for each resolution. These units are capable of capturing the temporal dimension inherent in the input image sequence. More details about convolutional LSTM (conLSTM) are disclosed in "Convolutional LSTM Network: A Machine Learning Approach for Precipitation Nowcasting" published by Xingjian Shi, Zhourong Chen, Hao Wang, Dit-Yan Yeung, Wai-kin Wong, and Wang-chun Woo in the IEEE Advances in Neural Information Processing Systems, 2015, pages 802 to 810, etc., the entire content of which is incorporated by reference into this application.

[0082] According to one embodiment, the neural network 109 is used to compress a tensor of 1×Num_Classes×Height×Width into a vector of 1×Num_Classes, such as Figure 4 This provides the convolutional layer from the last one to Figure 3 Smooth transitions of fully connected blocks in the example neural network architecture shown.

[0083] Back to Figure 1 In processing block 111, the estimated left / right lane IDs and lane numbers provided by the neural network 109 are compared with the ground truth left / right lane IDs and lane numbers provided by processing blocks 101 and 105 using a properly designed cost function 111. The above comparison generates a "loss" term, which is fed back to the neural network 109 to adjust the corresponding weights and biases through backpropagation.

[0084] In one embodiment, the cost function 111 is based on a traditional cross-entropy loss function.

[0085] In one embodiment, the cost function 111 may also depend on the distance between a lane and the leftmost lane and / or the rightmost lane in the current image of a multi-lane road. This adaptive penalty term gives more weight to the right estimate, which is the smallest, so that the network 109 learns to update its weights and biases (partially) based on the smallest output, which is likely to be more accurate.

[0086] In one embodiment, the cost function 111 implements a constraint that the first candidate current lane and the second candidate current lane are related to the total number of lanes on a multi-lane road. This constraint can be expressed as a triangle regularization term that implements the following relationship:

[0087] L+R=C,

[0088] Wherein, L represents the left lane ID, R represents the right lane ID, and C represents the total number of lanes. In this way, the three outputs of the neural network 109 (i.e., the left lane ID, the right lane ID, and the number of lanes) are linearly related.

[0089] In one embodiment, the cost function 111 may be the sum of all the above terms, ie, the cross entropy term, the adaptability penalty term, and the triangle regularization term.

[0090] According to another embodiment, the processing circuit in the lane detection system 100, in particular the neural network 109 implemented as a single-path processing circuit, is configured to generate a first confidence value for the left lane ID (i.e., the first candidate current lane) and a second confidence value for the right lane ID (i.e., the second candidate current lane). Such an embodiment, for example, Figure 5 As shown, in which, in addition to the output estimates of the left and right IDs and lane numbers, the neural network 109 also generates a confidence value between 0 and 1 for each estimate that describes the reliability, where a confidence value of 0 indicates that the estimate is expected to be wrong and a confidence value of 1 indicates that the network 109 has 100% confidence and the estimate is correct.

[0091] According to another embodiment, the processing circuit in the lane detection system 100 is used to: flip (i.e., mirror) the image of the multi-lane road along the longitudinal axis to obtain a flipped (i.e., mirrored) image of the multi-lane road; determine the third candidate current lane, i.e., the third estimated value of the current lane, by using the leftmost lane in the flipped image of the multi-lane road as the first reference lane; determine the fourth candidate current lane, i.e., the fourth estimated value of the current lane, by using the rightmost lane in the flipped image of the multi-lane road as the second reference lane; determine the current lane of the vehicle based on the first candidate current lane, the second candidate current lane, and the constraints that the first candidate current lane is equal to the fourth candidate current lane and the second candidate current lane is equal to the third candidate current lane. Such an implementation is, for example, Figure 6 As shown, the following concept is used: neural network 109 processes the mirrored image to obtain three other outputs using the constraints that the left lane ID in the original image is equal to the right lane ID in the mirrored image, the right lane ID in the original image is equal to the left lane ID in the mirrored image, and the number of lanes in the two images is equal. It can be understood that the estimation redundancy implemented in this way improves the overall accuracy of the original model.

[0092] according to Figure 7 In another embodiment shown, the processing circuitry in lane detection system 100 is used to implement a semantic segmentation network 116, which is used to semantically segment a current image of a multi-lane road. Neural network 109 is used to determine the vehicle's current lane in the current image of the multi-lane road based on the left lane ID (i.e., the first candidate current lane), the right lane ID (i.e., the second candidate current lane), and the semantically segmented current image. By understanding the semantic content of the current image, this embodiment can improve the accuracy of the estimated value due to improved detection of semantically related objects such as roads, cars, and lane markings. According to one embodiment, semantic segmentation network 116 can be pre-trained or trained simultaneously with the original model 109.

[0093] As mentioned above, Figure 8 and Figure 10 1 shows processing blocks implemented by the processing circuitry in the lane detection system 100 during the application or inference phase of the convolutional neural network 109 according to various embodiments. Figure 8 and Figure 10 include Figure 1 Several of the processing blocks shown and described above. For example, Figure 8 and Figure 10 Both include a processing block 103 for obtaining a visual representation, i.e., an image of the current driving scene. According to one embodiment, in this inference phase (and in Figure 1 The one or more images used in the training phase (described in

[0045] ) may be grayscale images or color images, such as RGB images. According to one embodiment, the resolution of the input images used in the inference phase may depend on the resolution selected in the training phase.

[0094] exist Figure 8 and Figure 10 In the processing block 106 of , a pre-processing step can be applied to the input image provided by the processing block 103. In one embodiment, the pre-processing step 106 includes a brightness adjustment based on the following considerations. Within the inference phase, the average perceived brightness of the images in a certain driving sequence is determined. If the perceived brightness of the current image is less than the average brightness (according to a certain threshold found empirically), the brightness of the image being processed can be adjusted. The brightness adjustment can be achieved by a linear transformation of the pixel intensities or by using a gamma correction. According to another option, a new layer of the neural network 109 can also be considered to learn the optimal parameters to be used for the gamma correction.

[0095] like Figure 8 and Figure 10As shown, the pre-processed image is provided by the processing block 106 to the neural network 109 to determine a first candidate current lane (i.e., a left lane ID) by taking the leftmost lane in the current image of the multi-lane road as a first reference lane, and to determine a second candidate current lane (i.e., a right lane ID) by taking the rightmost lane in the current image of the multi-lane road as a second reference lane.

[0096] If training is done using Figure 6 and Figure 7 If one of the alternative neural network architectures shown is executed, the input settings can be adjusted accordingly. Figure 6 The neural network architecture shown forwards the processed image to the mirroring step, which produces a flipped version (along the vertical axis) of the processed input image. Thus, the processed input image is presented to the first network component ( Figure 6 ), forwarding its flipped version to the second network component ( Figure 6 "MokaConvLSTM Block 2" in

[15] ).

[0097] for Figure 7 The neural network architecture shown performs pixel-level image segmentation on the processed input image before lane ID / lane number estimation. The segmentation result is presented to the network as a concatenated tensor along with the processed input image ( Figure 7 "MokaConvLSTM Block 1" in

[15] ).

[0098] Figure 8 The processing block 110 (referred to as the "final output decider") is used to determine the current lane of the vehicle in the current image of the multi-lane road based on the first candidate current lane (i.e., the left lane ID) and the second candidate current lane (i.e., the right lane ID) provided by the neural network 109. In one embodiment, the processing block 110 is used to decide whether to use the left lane ID or the right lane ID as the final output, i.e., the current lane. In one embodiment, the decision to be made by the processing block 110 is defined as an entropy-based decision, which will be further referred to below. Figure 9 Provide a description.

[0099] like Figure 9 As shown, according to one embodiment, the neural network 109 implemented by the processing circuit in the lane detection system 100 is used to determine the first candidate current lane as a first set of likelihood metric values ​​(i.e., a first likelihood vector (in Figure 9 The second candidate current lane is determined as the second set of likelihood measures (i.e., the second likelihood vector (in Figure 9The first set of likelihood measures comprises a corresponding likelihood measure for each lane in which the car is traveling on the multi-lane road, and the second set of likelihood measures comprises a corresponding likelihood measure for each lane in which the car is traveling on the multi-lane road.

[0100] according to Figure 9 In the illustrated embodiment, processing block 110 is configured to make an entropy-based decision by determining a first candidate current lane (i.e., the left lane ID) as the lane associated with the maximum likelihood metric value in the first set of likelihood metrics; determining a second candidate current lane (i.e., the right lane ID) as the lane associated with the maximum likelihood metric value in the second set of likelihood metrics; and determining that either the first candidate current lane (i.e., the left lane ID) or the second candidate current lane (i.e., the right lane ID) is the vehicle's current lane in the current image by comparing the difference between the maximum likelihood metric value and the average likelihood metric value in the first set of likelihood metrics with the difference between the maximum likelihood metric value and the average likelihood metric value in the first set of likelihood metrics. In other words, given the output vectors of lane ID estimates for each lane ID, processing block 110 is configured to select the maximum activation value from both the left estimate vector and the right estimate vector. The average activation value of the corresponding vector is then subtracted from these maximum values. A decision regarding the final output lane ID is made by comparing these resulting values. The ID with the higher value is used as the final estimate, i.e., the current lane.

[0101] Figure 10 The embodiment shown is Figure 8 The embodiment shown differs primarily in that an additional penalty term is implemented for fluctuations between consecutive estimates of temporally adjacent input images, which allows for smooth estimation of consecutive IDs. According to one embodiment, this is achieved by weighting all previously discussed calculated entropy values ​​to make a decision by calculating the term P as follows:

[0102]

[0103] According to another embodiment, the lane detection system 100 is configured to take into account the vehicle's motion state, for example, whether the vehicle is stopped due to traffic congestion. Traffic congestion may occur due to reasons such as traffic lights or spontaneous congestion. To this end, Figure 10 The processing circuitry in the lane detection system 100 shown implements a modified processing block 110 (in Figure 10 The modified final output decision maker is referred to as the "modified final output decision maker" in the figure, which takes into account the vehicle's motion state when making decisions, that is, whether the car is moving (dynamic) or stationary (static). More details about the modified processing block 110 can be found in Figure 11 If the motion state is "dynamic" (given by Figure 11If the motion state is "static," the final lane ID and lane number are read from buffer 104. Buffer 104 contains decisions for a certain N previous dynamic frames (this means that in the static case, the buffer is not updated).

[0104] When the vehicle ( Figure 11 When a vehicle (referred to as the ego vehicle) is waiting at a traffic light or encountering a traffic jam, the estimation value provided by the neural network 109 may become less accurate due to the limited space of features that the model 109 can extract from the current image. To handle such situations, one embodiment can detect when the vehicle is stopped. This is done by comparing the variance between the left and / or right outputs with the previous N frames / images stored in another buffer 102, using Figure 11 The processing block 108 completes the process. When the vehicle stops, since the scene contents in these N frames are similar, the variance of the entropy values ​​of the last N images will also decrease. Figure 12 This illustrates the point.

[0105] Therefore, according to one embodiment, the processing circuit in the lane detection system 100 is used to: obtain multiple time-continuous images of the multi-lane road, wherein the multiple time-continuous images include a current image and one or more previous images; determine the current lane of the vehicle in the current image of the multi-lane road based on the first candidate current lane, the second candidate current lane, and one or more previously determined lanes in the one or more previous images of the vehicle on the multi-lane road that can be retrieved from the buffer 104.

[0106] Figures 13a to 13d Some exemplary images of driving scenes are shown, along with left and right ID estimates.

[0107] Figure 14 14 is a flow chart of the steps of a lane detection method 1400, which can be performed by the lane detection system 100. The lane detection method 1400 includes the following steps: acquiring 1401 a current image of a multi-lane road (i.e., a driving scene); determining 1403 a first candidate current lane, i.e., a first estimate of the current lane, by using the leftmost lane in the current image of the multi-lane road as a first reference lane; determining 1405 a second candidate current lane, i.e., a second estimate of the current lane, by using the rightmost lane in the current image of the multi-lane road as a second reference lane; and determining 1407 a current lane of a vehicle in the current image of the multi-lane road based on the first candidate current lane and the second candidate current lane.

[0108] Embodiments of the present invention can estimate the total number of lanes inherent in the input texture image and the ID of the current lane in which the vehicle is traveling. Embodiments of the present invention can identify the current lane and the corresponding lane number in extreme scenarios based on information aggregated from the current image and past images. According to embodiments of the present invention, the temporal dimension inherent in the structure of the input data (continuous stream of images / videos) processed by the Long-Short Term Memory (LSTM) unit in the neural network 109 can guarantee an additional source of information, which helps to deal with strong occlusions because the previous information extracted from the previous images / frames in the sequence helps to solve this problem. Due to the nature of the input signal (2D image), an LSTM variant called convolutional LSTM (conLSTM) can be used to be able to capture the temporal information inherent in a two-dimensional input signal such as an image.

[0109] Those skilled in the art will understand that the "blocks" ("units") in the various figures (methods and devices) represent or describe the functions of an embodiment of the present invention (and not necessarily separate "units" in hardware or software), and therefore equally describe the functions or features of the device embodiments and the method embodiments (units are equivalent steps).

[0110] In the several embodiments provided in this application, it should be understood that the disclosed systems, devices, and methods can be implemented in other ways. For example, the above-mentioned device embodiments are merely exemplary. For example, unit division is merely a logical function division, and there may be other division methods in actual implementation. For example, multiple units or components can be merged or integrated into another system, or some features can be ignored or not performed. In addition, the mutual coupling or direct coupling or communication connection shown or described can be implemented through some interfaces. The direct coupling or communication connection between devices or units can be implemented through electronic, mechanical or other forms.

[0111] Units described as separate components may or may not be physically separate, and components shown as units may or may not be physical units, may be located in one location, or may be distributed across multiple network units. Some or all of the units may be selected as needed to achieve the objectives of the technical solutions of the embodiments.

[0112] In addition, the functional units in the embodiments of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

Claims

1. A lane detection system (100) for a vehicle, for determining a current lane in which a vehicle is traveling on a multi-lane road, characterized in that: The system (100) comprises: Processing circuitry for: acquiring a current image of the multi-lane road; determining a first candidate current lane based on a first reference lane, the first reference lane being the leftmost lane in the current image of the multi-lane road, and the first candidate lane being a first estimated value of the current lane; determining a second candidate current lane based on a second reference lane, the second reference lane being the rightmost lane in the current image of the multi-lane road, and the second candidate lane being a second estimated value of the current lane; determining the first candidate current lane as a lane associated with a maximum likelihood measure value in a first set of likelihood measures, wherein the first set of likelihood measures includes a corresponding likelihood measure value for each lane on the multi-lane road in which the vehicle is traveling; determining the second candidate current lane as the lane associated with the maximum likelihood measure value in a second set of likelihood measures, wherein the second set of likelihood measures includes a corresponding likelihood measure value for each lane on the multi-lane road in which the vehicle is traveling; According to the first candidate current lane and the second candidate current lane, the first candidate current lane or the second candidate current lane is determined as the current lane of the vehicle in the current image of the multi-lane road.

2. The lane detection system (100) according to claim 1, characterized in that The processing circuit is further configured to determine the number of lanes on the multi-lane road based on the current image of the multi-lane road.

3. The lane detection system (100) according to claim 1, characterized in that The processing circuit is used for: By comparing the difference between the maximum likelihood metric value and the average likelihood metric value in the first group of likelihood metric values ​​with the difference between the maximum likelihood metric value and the average likelihood metric value in the second group of likelihood metric values, the first candidate current lane or the second candidate current lane is determined to be the current lane of the vehicle in the current image of the multi-lane road.

4. The lane detection system (100) according to any one of claims 1-3, characterized in that The processing circuit is used to: acquire a plurality of time-continuous images of the multi-lane road, wherein the plurality of time-continuous images include the current image and one or more previous images; and determine the current lane of the vehicle in the current image of the multi-lane road based on the first candidate current lane, the second candidate current lane, and one or more previously determined lanes of the vehicle in the one or more previous images of the multi-lane road.

5. The lane detection system (100) according to claim 4, characterized in that The processing circuit is also used to: determine whether the vehicle is moving; and if the vehicle is not moving, determine the current lane of the vehicle in the current image of the multi-lane road based on the one or more previously determined lanes of the vehicle in the previous one or more images.

6. The lane detection system (100) according to claim 5, characterized in that The processing circuit is configured to determine whether the vehicle is moving based on the one or more previously determined lanes of the vehicle in the previous one or more images.

7. The lane detection system (100) according to any one of claims 1-3, 5, and 6, characterized in that: The processing circuit is used to implement a convolutional neural network (107, 109) for determining the first candidate current lane, the second candidate current lane, the current lane and / or the total number of lanes on the multi-lane road using one or more convolutional neural network weights.

8. The lane detection system (100) according to claim 7, characterized in that The convolutional neural network (107, 109) is used to train the convolutional neural network weights using a cost function (111), wherein the cost function (111) is based on a cross entropy loss function, the cost function (111) also depends on the distance between a lane and the leftmost lane and / or the rightmost lane in the current image of the multi-lane road, and / or the cost function (111) implements a constraint that the first candidate current lane and the second candidate current lane are related to the total number of lanes on the multi-lane road.

9. The lane detection system (100) according to claim 7, characterized in that The processing circuit is further configured to implement a semantic segmentation network (116), wherein the semantic segmentation network (116) is configured to semantically segment the current image of the multi-lane road, and the processing circuit is configured to determine the current lane of the vehicle in the current image of the multi-lane road based on the first candidate current lane, the second candidate current lane, and the semantically segmented current image.

10. The lane detection system (100) according to any one of claims 1-3, 5, 6, 8, and 9, characterized in that: The processing circuit is further configured to generate a first confidence value for the first candidate current lane and a second confidence value for the second candidate current lane.

11. The lane detection system (100) according to any one of claims 1-3, 5, 6, 8, and 9, characterized in that: The processing circuit is used for: flipping the current image of the multi-lane road along a longitudinal axis to obtain a flipped image of the multi-lane road; determining a third candidate current lane by using the leftmost lane in the flipped image of the multi-lane road as the first reference lane; determining a fourth candidate current lane by using the rightmost lane in the flipped image of the multi-lane road as the second reference lane; The current lane of the vehicle is determined according to the first candidate current lane, the second candidate current lane, and the constraints that the first candidate current lane is equal to the fourth candidate current lane and the second candidate current lane is equal to the third candidate current lane.

12. The lane detection system (100) according to any one of claims 1-3, 5, 6, 8, and 9, characterized in that: The processing circuit is used to obtain multiple time-continuous images of the multi-lane road, wherein the multiple time-continuous images include the current image; the processing circuit is used to adjust the brightness of the current image according to the average brightness of the multiple time-continuous images of the multi-lane road.

13. The lane detection system (100) according to any one of claims 1-3, 5, 6, 8, and 9, characterized in that: The lane detection system (100) further comprises a camera for capturing the image of the multi-lane road.

14. A vehicle, characterized in that: The vehicle comprises a lane detection system (100) according to any one of claims 1-13.

15. A method (1400) for determining a current lane in which a vehicle is traveling on a multi-lane road, characterized in that: The method (1400) comprises: Acquiring (1401) a current image of the multi-lane road; determining a first candidate current lane based on a first reference lane, the first reference lane being the leftmost lane in the current image of the multi-lane road, and the first candidate lane being a first estimated value of the current lane; determining a second candidate current lane based on a second reference lane, the second reference lane being the rightmost lane in the current image of the multi-lane road, and the second candidate lane being a second estimated value of the current lane; determining the first candidate current lane as a first set of likelihood measures, wherein the first set of likelihood measures includes a corresponding likelihood measure value for each lane in which the vehicle is traveling on the multi-lane road; determining the second candidate current lane as a second set of likelihood measures, wherein the second set of likelihood measures includes a corresponding likelihood measure value for each lane on the multi-lane road in which the vehicle is traveling; Based on the first candidate current lane and the second candidate current lane, it is determined ( 1407 ) that the first candidate current lane or the second candidate current lane is the current lane of the vehicle in the current image of the multi-lane road.

16. The method according to claim 15, characterized in that Also includes: The number of lanes on the multi-lane road is determined based on the current image of the multi-lane road.

17. The method according to claim 15, characterized in that The determining (1407) that the first candidate current lane or the second candidate current lane is the current lane of the vehicle in the current image of the multi-lane road based on the first candidate current lane and the second candidate current lane includes: By comparing the difference between the maximum likelihood metric value and the average likelihood metric value in the first group of likelihood metric values ​​with the difference between the maximum likelihood metric value and the average likelihood metric value in the second group of likelihood metric values, the first candidate current lane or the second candidate current lane is determined to be the current lane of the vehicle in the current image of the multi-lane road.

18. The method according to any one of claims 15 to 17, characterized in that Also includes: Acquiring a plurality of time-continuous images of the multi-lane road, wherein the plurality of time-continuous images include the current image and one or more previous images; The current lane of the vehicle in the current image of the multi-lane road is determined based on the first candidate current lane, the second candidate current lane, and one or more previously determined lanes of the vehicle in the previous one or more images of the multi-lane road.

19. The method according to claim 18, characterized in that Also includes: determining whether the vehicle is moving; If the vehicle is not moving, the current lane of the vehicle in the current image of the multi-lane road is determined based on the one or more previously determined lanes of the vehicle in the previous one or more images.

20. The method according to claim 19, characterized in that Also includes: A determination is made as to whether the vehicle is moving based on the one or more previously determined lanes of the vehicle in the previous one or more images.

21. The method according to any one of claims 15 to 17, 19 and 20, characterized in that: Also includes: The first candidate current lane, the second candidate current lane, the current lane, and / or the total number of lanes on the multi-lane road are determined using one or more convolutional neural network weights.

22. The method according to claim 21, characterized in that Also includes: The convolutional neural network weights are trained using a cost function (111), wherein the cost function (111) is based on a cross entropy loss function, the cost function (111) also depends on the distance between a lane and the leftmost lane and / or the rightmost lane in the current image of the multi-lane road, and / or the cost function (111) implements a constraint that the first candidate current lane and the second candidate current lane are related to the total number of lanes on the multi-lane road.

23. The method according to claim 21, characterized in that Also includes: semantically segmenting the current image of the multi-lane road; The current lane of the vehicle in the current image of the multi-lane road is determined according to the first candidate current lane, the second candidate current lane, and the current image after semantic segmentation.

24. The method according to any one of claims 15 to 17, 19, 20, 22 and 23, characterized in that Also includes: A first confidence value is generated for the first candidate current lane and a second confidence value is generated for the second candidate current lane.

25. The method according to any one of claims 15 to 17, 19, 20, 22 and 23, characterized in that Also includes: flipping the current image of the multi-lane road along a longitudinal axis to obtain a flipped image of the multi-lane road; determining a third candidate current lane by using the leftmost lane in the flipped image of the multi-lane road as the first reference lane; determining a fourth candidate current lane by using the rightmost lane in the flipped image of the multi-lane road as the second reference lane; The current lane of the vehicle is determined according to the first candidate current lane, the second candidate current lane, and the constraints that the first candidate current lane is equal to the fourth candidate current lane and the second candidate current lane is equal to the third candidate current lane.

26. The method according to any one of claims 15-17, 19, 20, 22, and 23, characterized in that Also includes: acquiring a plurality of time-continuous images of the multi-lane road, wherein the plurality of time-continuous images include the current image; The brightness of the current image is adjusted according to an average brightness of the plurality of temporally consecutive images of the multi-lane road.

27. The method according to any one of claims 15 to 17, 19, 20, 22 and 23, wherein: Also includes: The image of the multi-lane road is captured by a camera.

28. A computer program product comprising program code, characterized in that When the program code is executed by a computer, the computer is caused to perform the method according to any one of claims 15 to 27.

Images