Lane marking detection

Abstract

Certain aspects of the present disclosure provide a method for lane mark detection, comprising: receiving an input image; providing the input image to a lane mark detection model; processing the input image using a shared lane mark portion of the lane mark detection model; processing an output of the shared lane mark portion of the lane mark detection model using a plurality of lane mark specific representation layers of the lane mark detection model to generate a plurality of lane mark representations; and outputting a plurality of lane marks based on the plurality of lane mark representations.

Description

[0001] Cross-reference to related applications

[0002] This application claims priority to U.S. Patent Application No. 17 / 200,592, filed March 12, 2021, and U.S. Patent Application No. 62 / 988,795, filed March 12, 2020, the entire contents of each of which are incorporated herein by reference.

[0003] introduction

[0004] Various aspects of this disclosure relate to systems and methods for rapid and robust lane detection.

[0005] Modern vehicles are increasingly equipped with advanced driver assistance systems (ADAS), which may include lane detection, in particular, for assisted and autonomous driving functions. Existing technologies for lane detection are slow, require significant manual configuration, and lack robustness across many driving scenarios. Therefore, existing lane detection systems are unsuitable for modern vehicles.

[0006] Accordingly, what is needed are systems and methods that provide fast and robust lane detection.

[0007] Brief Overview

[0008] Some aspects provide a method for lane marking detection, the method comprising: receiving an input image; providing the input image to a lane marking detection model; processing the input image using a shared lane marking portion of the lane marking detection model; processing the output of the shared lane marking portion of the lane marking detection model using a plurality of lane marking-specific representation layers of the lane marking detection model to generate a plurality of lane marking representations; and outputting a plurality of lane markings based on the plurality of lane marking representations.

[0009] Other aspects include: a processing system configured to perform the foregoing methods and those methods described herein; a non-transient computer-readable medium including instructions that, when executed by one or more processors of the processing system, cause the processing system to perform the foregoing methods and those methods described herein; a computer program product implemented on a computer-readable storage medium including code for performing the foregoing methods and those methods further described herein; and a processing system including means for performing the foregoing methods and those methods further described herein. Brief description of the attached diagram

[0011] The accompanying drawings depict some of these aspects and are therefore not intended to limit the scope of this disclosure.

[0012] Figure 1 An example depicting lane markings on a road.

[0013] Figure 2 It describes various aspects of an example advanced driver assistance system or autonomous driving system.

[0014] Figure 3 The various aspects of the example lane marking detection model architecture are described.

[0015] Figure 4 An example horizontal reduction module for compression (or "squeezing") layers that can be used in lane marking detection models is depicted.

[0016] Figure 5 An example method for detecting lane markings is described.

[0017] Figure 6 An example processing system that can be configured to detect lane markings as described herein is depicted.

[0018] To facilitate understanding, the same reference numerals are used wherever possible to designate common elements shared by all figures. Elements and features conceived of one aspect can be beneficially incorporated into other aspects without further elaboration.

[0019] Detailed description

[0020] The systems and methods disclosed herein enable fast and robust lane detection in a variety of applications. For example, the systems and methods disclosed herein can achieve accurate and robust localization of lane markings in real time in vehicles with assisted and autonomous driving capabilities.

[0021] Generally, lane markings are devices or materials on the road surface that convey information such as where lanes are located on the road. Examples of lane markings include painted traffic lines, painted pedestrian crossings, painted parking spaces, disabled parking spaces, reflective markings, curbs, drains, Botts' dots, and raised strips, to name just a few.

[0022] Lane markings can be used for navigation in autonomous vehicles. For example, vehicles may include Advanced Driver Assistance Systems (ADAS) or Advanced Automated Driving Systems (SDS). Such systems are widely adopted, largely based on concurrent improvements in computer vision technology.

[0023] Although several components exist associated with ADAS and SDS systems (such as lane marking detection, vehicle detection and tracking, obstacle detection, scene understanding, and semantic segmentation), lane detection is a critical component for camera perception and localization. For example, lane detection is essential for keeping vehicles within their own lane (the lane in which the vehicle is located, also known as the main lane) and for assisting vehicles in changing lanes to the left or right of their own lane.

[0024] Most conventional lane marking detection methods are based on a two-stage semantic segmentation approach. In the first stage of such methods, the network is designed to perform pixel-level classification, assigning each pixel in the image to a binary label: lane or non-lane. However, in each pixel classification, dependencies or structures between pixels are not considered, thus requiring additional post-processing in the second stage to explicitly impose constraints, such as the uniqueness and straightness of detected lines. Post-processing can be applied using, for example, conditional random fields, additional networks, or sophisticated computer vision techniques such as Random Sample Consensus (RANSAC), but these post-processing techniques require high computational complexity and typically need to be carefully and manually integrated into the lane marking detection system. Thus, two-stage semantic segmentation methods are difficult to deploy in a wide range of applications.

[0025] Another issue with current lane marking detection methods is power consumption. Processing each image in an image stream can require significant computational power, which is not always available or feasible in automotive environments. Specifically, for electric vehicles, the power consumption of the onboard system is a crucial consideration.

[0026] In contrast, the aspects described in this paper involve an efficient end-to-end architecture that directly predicts lane marker vertices without any costly post-processing steps. More specifically, in the aspects described in this paper, the lane marker detection problem is treated as a line-by-line classification task, where the structural shape of lane markers extending from left to right in an image is modeled as two subsets of layers in a single model. The first subset of model layers is configured to compress and model the horizontal component of a shared representation for all lanes, while the second subset of model layers is configured to model each lane based on this shared representation to directly output the lane marker vertices. In some aspects, the target lane marker location is efficiently obtained at runtime via the argmax function.

[0027] Therefore, the aspects described in this paper beneficially improve upon conventional methods for lane detection, eliminating the need for complex post-processing and reducing computational power, computation time, and power consumption. Consequently, the aspects described in this paper can be effectively deployed in a wider range of applications and environments.

[0028] Example lane markings on the road

[0029] Figure 1 An example of lane marking 102 on road 104 is depicted. In this example, lane marking 102 is a strip drawn on road 104, but in other examples, lane markings can be any kind of recognizable lane division. Lane marking 102 can be detected by a lane marking detection system and used to determine lanes for advanced driver assistance systems, as well as semi-autonomous and autonomous driving systems in vehicles.

[0030] Example Electronic Lane Marking Detection System

[0031] Figure 2 Various aspects of an example advanced driver assistance system or autonomous driving system 200 are depicted. System 200 can be configured to work with local sensors and edge processing capabilities, such as those provided by mapping algorithm 218.

[0032] In this respect, system 200 includes one or more cameras 202, a Global Positioning System (GPS) 204, an Inertial Measurement Unit (IMU) 206, a sensor synchronization board 208, and one or more processors 220. In this example, processor 220 includes a sensor driver module 210, a positioning engine 212, a perception engine 214, and a data aggregation and connectivity module 216. Processor 220 can typically communicate with one or more devices over a wireless or wired network.

[0033] In some aspects, the front-end sensors (such as camera 202, GPS receiver 204, and IMU 206) can be consumer-grade sensors. The sensor synchronization board 208 may include an embedded microcontroller that controls the timestamps of all sensors. In some aspects, the sensor synchronization board 208 can generate timestamps with timing errors of less than 10 microseconds.

[0034] The outputs of camera 202, GPS 204, and IMU 206 can be fed to both positioning engine 212 and perception engine 214. Perception engine 214 is configured to detect key landmarks (e.g., lane markings) in the incoming video stream and accurately locate them in image frames. Positioning engine 212 is configured to provide an accurate estimate of the camera's attitude (e.g., in six degrees of freedom) by correlating GPS signals, inertial sensor readings, and camera video input.

[0035] The outputs of the localization engine 212 and the perception engine 214 can be aggregated and sent to edge processing services, such as cloud processing systems, via the data aggregation and connectivity module 216. In some aspects, the mapping algorithm 218 can generate location estimates of landmarks in a global frame, where the global frame includes the entire map or a large portion of the map.

[0036] Lane Marking Detection Model Architecture

[0037] Unlike conventional segmentation-based lane detection techniques, the aspects described in this paper directly identify the location of lane markers in the image input data. More specifically, given an input image X∈R h×w×c Where h is the height of the image data (e.g., in pixels), w is the width of the image data (e.g., in pixels), and c is the number of channels in the image, the goal being to find lane markings l i (i = 1, ..., N) 车道 Vertex set {vl ij}={(x ij ,y ij )}(j=1,…,K), where N 车道 is the number of lanes in the image (usually predefined), while K is the total number of vertices.

[0038] The method described in this paper reduces the complexity of the aforementioned objective by finding the set of horizontal locations of lane markings in an image. Specifically, the row-by-row representation of lane markings can be determined by dividing the image into rows and using a convolutional neural network to solve the lane marking detection task as either a row-by-row location classification problem or a regression problem, depending on the chosen loss function. Note that for the row-by-row representation, the total number of possible vertices K is limited to the image height h. In some aspects, the network can perform row-by-row regression to identify lane locations, as described in more detail below.

[0039] The lane marking detection model (e.g., a convolutional neural network model) discussed in this paper outputs three predictions: (1) the horizontal position x of the lane vertices. ij (2) Existence confidence per vertex vc ij ; and (3) the confidence level of the presence of lane-by-lane markings ci Based on the output of these models, each lane marking l i It can be obtained through the following expression:

[0040]

[0041] vl ij It is the set of vertices associated with lane markings {(x ij ,y ij )}, and T vc and T lc These are the existence confidence thresholds for each vertex and each lane, respectively.

[0042] Figure 3 The aspects of the example lane marking detection model architecture are described, such as its ability to be generated by... Figure 2The lane marking detection model is implemented using a perception engine 214. The lane marking detection model can be configured as a deep neural network that is trained to predict the location of lane markings in input image data (e.g., image 301), which can then be used to predict lane representations (e.g., 322 in output image 321).

[0043] In the following discussion, lane representation may generally refer to the boundaries of a driving lane on a road. For example, the current or self-defined lane in which a vehicle is traveling may include two representational boundaries that define that lane, on the left and right. Similarly, the lane to the left of the current lane may include its own two representational boundaries, and in some cases, these two representational boundaries may include a shared boundary representation. For example, the left boundary representation of the current lane may represent the right boundary representation of the lane to the left of the current lane.

[0044] In this example, the model architecture 300 includes three in-model stages 302A-C. The first stage 302A includes an encoder-decoder segmentation network 304 configured to encode information about lane markings in the image. Unlike conventional image segmentation, in this example, the encoder-decoder segmentation network 304 is configured to recover half the spatial resolution of the input image 301 to advantageously reduce computational complexity. In other words, in this example, the input image 301 has an input resolution of 128 high x 512 wide x 3 channels (e.g., red, green, blue), while the output of the encoder-decoder segmentation network 304 is 128 high x 256 wide x 3 channels. It is worth noting that these resolutions and... Figure 3 The other resolutions depicted are merely examples, and other resolutions are possible. Furthermore, the encoder-decoder network can be omitted in other aspects to support other image data compression techniques, including other types of neural network blocks.

[0045] Phase 2, 302B, comprises a shared layer represented by all potential lane markings, and may be referred to as the shared reduction layer. In this example, it includes three layers: 306, 308, and 310. In Phase 2, 302B, the horizontal (h) dimension represented by the shared lane markings is progressively compressed (or “squeezed”) in each successive layer using a Horizontal Reduction Module (HRM), without altering the vertical dimension. Example HRM reference. Figure 4 This form of compression operation naturally results in a line-by-line representation because the HRM continuously squeezes the spatial width components so that, for example, the final feature of each lane has only one lane vertex in each column (each width).

[0046] Phase 302C includes lane-specific reduction layers, which in this example include layers 314, 316, 318, and 320. In Phase 303C, the lane-specific reduction layers further use lane-specific HRM to compress the lane marking representation to generate a single vector lane marking representation (such as 316).

[0047] It is worth noting that the shared representation in Phase 302B and the lane-specific representation in Phase 302C allow for tuning the performance of the entire model. For computational efficiency, a first subset of reduction layers is shared across lanes in Phase 302B, followed by a second subset of lane-specific reduction layers in Phase 302C. Using a dedicated reduction layer for each lane label representation in Phase 302C improves the overall model performance because each lane label representation has different inherent spatial and shape characteristics. Using more shared layers can save additional computational costs, but the accuracy of each lane may be degraded. Thus, performance can be tuned for the application.

[0048] Phase 302C includes lane-specific representations. i The two branches (where as an example in) Figure 3 The diagram shows six lane-specific representations: a row-by-row vertex position branch 320 and a vertex-by-vertex confidence branch 318, both configured to perform classification and confidence regression on the final compression layer 316. In this respect, the final (and fully) compression layer 316 has spatial resolution only in the vertical dimension and several channels according to the target horizontal resolution h′ (e.g., h′ = h / 2). The classification branch 320 predicts lane markings l. i Horizontal position x ij It can set the predicted vertex as (x ij ,y ij ),y ij ∈h′. Confidence 318, branch prediction vertex confidence vc ij This confidence level is related to each vertex vl of the lane marking. ij Is there a correlation?

[0049] Finally, lane markings have a confidence level of 312, which generates a confidence level l after sharing the HRM. ci The presence of lane markings is related to whether semantic lane markings (e.g., left marking of the current lane, right marking of the current lane, left marking of the left lane, right marking of the left lane, left marking of the right lane, right marking of the right lane, etc.) are detected in the image data.

[0050] Figure 4 It describes what can be used as Figure 3 The exemplary horizontal reduction module 400 of the compression (or "squeeze") layer discussed herein.

[0051] To efficiently compress the horizontal lane marker representation, a residual layer is used in the horizontal reduction model 400. Specifically, a horizontal average pooling layer with 1×1 convolutions is added to skip connections 402 to downsample the horizontal components. Although pooling operations allow deeper layers to collect more spatial context (to improve classification) and reduce computational complexity, they still have the disadvantage of reducing pixel precision. Therefore, to efficiently preserve and enhance the horizontal representation of C×H×W elements, the input tensor (X) is rearranged into a tensor of shape rC×H×W / r in the residual branch 404. This operation can be referred to as a horizontal pixel anti-shift layer. By rearranging the representation of the input tensor, spatial information can be efficiently shifted to the channel dimension.

[0052] After rearranging the input tensors, a convolution (Conv) operation at box 406 can be applied to reduce the increased channel dimension rC back to the original channel dimension C. This not only reduces the computational burden but also helps to efficiently compress the lane marking spatial information from the pixel anti-shift layer.

[0053] Then at 410, the output of the residual branch 404 is combined with the jump connection 402, and the combined result is then provided to an activation function (such as ReLU(412) in this example).

[0054] To further improve the distinction between lane markings, an attention mechanism can be added via a squeeze and excitation (SE) block 408. The SE block helps include global information in the decision-making process by aggregating information from the entire receptive field and recalibrating the channel-wise feature responses with spatial information encoded by the pixel anti-shift layer.

[0055] Thus, the horizontal reduction module 400 advantageously compresses the spatial lane marking information to generate a compressed horizontal representation output.

[0056] Example training methods for lane marking detection models

[0057] In one example, the training objective is to optimize the total loss L, which is given by the following formula:

[0058] L = L vl +λ1L vc +λ2L lc ,

[0059] Among them, L vl L vc and L lc These are the losses for lane marker vertex position, lane marker vertex confidence, and lane marker confidence, respectively, and λ1 and λ2 are the weights of the latter two losses, which allows for fine-tuning of the training.

[0060] Regarding the lane marking vertex position loss (L) vl Since lane marking detection is programmed to classify the horizontal positions of lane markings line by line, any loss function used for classification can be used to train the lane marking vertex branches, including cross-entropy, Kullback–Leibler (KL) divergence, and piecewise linear probability (PL) loss, to name just a few examples.

[0061] Lane markings l i At the vertical position y ij Cross-entropy loss at point Use truth position And the predicted logarithm f with W = 2 channels ij To calculate.

[0062] To use KL divergence loss To train the lane marker vertex position branch, the spike target distribution of the lane marker position can be used as a Laplace distribution. gt (μ,b), where And b = 1, and subsequently with the estimated distribution Laplace 预测 (μ,b) are compared, where:

[0063]

[0064] and

[0065] In the case of PL loss, the lane marker location probability can be modeled as a piecewise linear probability distribution. For the input image, the total lane marker vertex location loss is given by the following equation: Where e ij The existence of a truth value can be expressed as follows: if a truth value exists with y... ij Lane markings at the apex of the lane i If it does not exist, set it to 1; otherwise, set it to 0.

[0066] For the lane marker vertex confidence loss, the existence of lane marker vertices is a binary classification problem, hence the use of lane marker l i Each y ij The prediction of a single scalar value at a location is equal to the true value. ij Binary cross-entropy loss between Training is then performed. The loss for the entire image is calculated as follows:

[0067]

[0068] For lane marking loss, a binary cross-entropy loss can be added. To train lane marker-level presence prediction. The loss uses the predicted N 车道 dimensional vector and each lane l i The existence of truth value is used to calculate.

[0069] The total loss was then calculated as

[0070] Inference using lane marking detection models

[0071] During inference, the estimated vertex locations can be predicted based on a chosen loss function (e.g., those selected as described above). Specifically, if cross-entropy or PL loss is used, lane marker vertices can be selected using the argmax function. Instead, if KL divergence loss is used, the estimated vertices can be extracted using softargmax. To account for the presence of each lane marker vertex, low-confidence locations can be rejected using the sigmoid output of the vertex and lane-by-lane presence branches, according to Equation 1 above.

[0072] Example methods for detecting lane markings

[0073] Figure 5 An example method 500 for detecting lane markings is described.

[0074] Method 500 begins in step 502 by receiving an input image. In some examples, the input image includes h vertical pixels, w horizontal pixels, and c channels, such as, for example, regarding... Figure 3 As described. The input image can typically be a still image, an image from a video stream, or a frame, etc.

[0075] Method 500 then proceeds to step 504, where the input image is provided to a lane marking detection model, such as those described above. Figure 3 The model described.

[0076] Method 500 then proceeds to step 506, where the input image is processed using the shared lane marking portion of the lane marking detection model, as described above. Figure 3 As described in 302B. In some cases, processing the input image using the shared lane marking portion of the lane marking detection model includes processing the input image through multiple shared lane marking representation layers of the lane marking detection model.

[0077] Method 500 then proceeds to step 508, where multiple lane-marking-specific representation layers of the lane marking detection model are used to process the output of the shared lane marking portion of the lane marking detection model to generate multiple lane marking representations, such as those described above. Figure 3As described in 302C. In some aspects, each lane marking-specific representation layer is associated with one of a plurality of lane marking representations, and each lane marking representation may be associated with a subset of the plurality of lane marking-specific representation layers.

[0078] Method 500 then proceeds to step 510, where multiple lane markings are output based on the multiple lane marking representations.

[0079] In some aspects of method 500, the multiple lane markings are output (e.g., l i This includes, for each of the multiple lane markings: using the first output layer of the lane marking detection model to predict the lane vertex (e.g., x). ij The horizontal position of the lane marking detection model is used to predict the per-vertex presence confidence (e.g., vc). ij ); and using the third output layer of the lane marking detection model to predict the presence confidence of lane-by-lane markings (e.g., l ci ).

[0080] In some aspects of aspect 500, the output of these multiple lane markings further includes: using the following expression to predict each lane marking l i :

[0081]

[0082] vl ij It is with each lane marking l i Associated vertex set {(x ij ,y ij )}.

[0083] In some respects, method 500 further includes: using an encoder-decoder segmentation network to compress the input image.

[0084] In some aspects of method 500, the final lane mark-specific representation layer for each corresponding lane mark-specific representation comprises a size of h vertical pixels, 1 horizontal pixel, and c channels.

[0085] In some aspects, method 500 further includes: using one or more level reduction modules to compress the input data in the shared lane marking portion of the lane marking detection model, such as the above regarding... Figure 3 and 4 As described.

[0086] In some aspects, method 500 further includes: using one or more additional level reduction modules to compress the input data in the plurality of lane-marking-specific representation layers of the lane marking detection model, such as those mentioned above. Figure 3 and4 As described.

[0087] In some aspects, method 500 further includes: displaying multiple lane representations on the output image, such as in Figure 3 As depicted in (e.g., 322).

[0088] Note that method 500 is an example method, and other methods are also possible. Specifically, other examples may include fewer, additional, and / or alternative steps compared to method 500, which is consistent with the aspects described in this article.

[0089] Example processing system

[0090] Figure 6 It describes what can be configured as described in this article. Figure 5 The example processing system 600 for detecting lane markings is as described in method 500.

[0091] The processing system 600 includes a central processing unit (CPU) 602, which in some examples may be a multi-core CPU. Instructions executed at the CPU 602 may be loaded, for example, from program memory associated with the CPU 602 or from memory partition 624.

[0092] The processing system 600 also includes additional processing components tailored for specific functions, such as a graphics processing unit (GPU) 604, a digital signal processor (DSP) 606, a neural processing unit (NPU) 608, a multimedia processing unit 610, and a wireless connectivity component 612. In some examples, one or more of the CPU 602, GPU 604, DSP 606, and NPU 608 may act as... Figure 2 The processor 202.

[0093] NPUs (such as the 608) are generally dedicated circuits configured to implement all the necessary control and arithmetic logic for executing machine learning algorithms, such as those for processing artificial neural networks (ANNs), deep neural networks (DNNs), random forests (RFs), etc. NPUs are sometimes alternatively referred to as neural signal processors (NSPs), tensor processing units (TPUs), neural network processors (NNPs), intelligent processing units (IPUs), vision processing units (VPUs), or graphics processing units.

[0094] NPUs (such as the 608) are configured to accelerate the execution of common machine learning tasks, such as image classification, machine translation, object detection, and various other predictive models. In some examples, multiple NPUs may be instantiated on a single chip (such as a system-on-a-chip (SoC)), while in other examples, multiple NPUs may be part of a dedicated neural network accelerator.

[0095] An NPU can be optimized for either training or inference, or in some cases, configured to balance performance between training and inference. For an NPU capable of performing both training and inference, these two tasks may typically still be performed independently.

[0096] NPUs designed to accelerate training are typically configured to speed up the optimization of new models. This involves highly computationally intensive operations, including taking an existing dataset (often labeled or sublabeled), iterating over the dataset, and then tuning model parameters (such as weights and biases) to improve model performance. Generally, optimization based on incorrect predictions involves backtracking through the layers of the model and determining gradients to reduce prediction errors. In some embodiments, the NPU 608 can be configured to train... Figure 3 The model architecture of 300 encompasses various aspects.

[0097] NPUs designed to accelerate inference are typically configured to operate on the full model. Such NPUs can thus be configured to take new data segments as input and rapidly process those segments using an already trained model to generate model outputs (e.g., inference). In some embodiments, NPU 608 can be configured to process... Figure 3 The model architecture of 300 encompasses various aspects.

[0098] In one implementation, the NPU 608 is part of one or more of the CPU 602, GPU 604, and / or DSP 606.

[0099] In some examples, the wireless connectivity component 612 may include sub-components for, for example, third-generation (3G) connectivity, fourth-generation (4G) connectivity (e.g., 4G LTE), fifth-generation connectivity (e.g., 5G or NR), Wi-Fi connectivity, Bluetooth connectivity, and other wireless data transmission standards. The wireless connectivity processing component 612 is further connected to one or more antennas 614.

[0100] The processing system 600 may also include one or more sensor processing units 616 associated with any type of sensor, one or more image signal processors (ISPs) 618 associated with any type of image sensor, and / or may include a navigation processor 620 that includes satellite-based positioning system components (e.g., GPS or GLONASS) and inertial positioning system components. In some examples, the sensor 616 may include Figure 2 The camera 202 and IMU 206. In some examples, the navigation processor 620 may include... Figure 2 GPS 204 in the middle.

[0101] The processing system 600 may also include one or more input and / or output devices 622, such as screens, touch-sensitive surfaces (including touch-sensitive displays), physical buttons, speakers, microphones, etc.

[0102] In some examples, one or more processors of the processing system 600 may be based on the ARM or RISC-V instruction set.

[0103] The processing system 600 also includes a memory 624, which represents one or more static and / or dynamic memories, such as dynamic random access memory, flash-based static memory, etc. In this example, the memory 624 includes computer-executable components that can be executed by one or more of the aforementioned processors of the processing system 600.

[0104] In this example, memory 624 includes a receiving component 624A (e.g., for receiving input data to be processed by the lane marking detection model), a processing component 624B (e.g., for processing various aspects of the lane marking detection model described herein), a compression component 624C (e.g., for compressing data, such as through a horizontal reduction module), a prediction (or output) component 624D (e.g., various outputs of the lane marking detection model described herein, for example), a training component 624E (e.g., for training the lane marking detection model), an inference component 624F, an encoding component 624G (e.g., for encoding the input data), a decoding component 624H (e.g., for decoding the encoded data), a display component 624I (e.g., for displaying lane markings and other information), and model parameters 624J (e.g., including parameters for a lane marking detection model such as that described herein). The depicted components, as well as other components not depicted, can be configured to perform aspects of the methods described herein.

[0105] For example, the receiving component 624A can be configured to receive input data, such as input image data.

[0106] It is worth noting that in other embodiments, aspects of the processing system 600 may be omitted, such as when the processing system 600 is a server computer. For example, in other embodiments, the multimedia component 610, wireless connectivity 612, sensor 616, ISP 618, and / or navigation component 620 may be omitted. Furthermore, aspects of the processing system 600 may be distributed.

[0107] Note Figure 6 This is just one example, and alternative processing systems with fewer, additional, and / or alternative components may be used in other examples.

[0108] Example Terms

[0109] Examples of implementations are described in the following numbered clauses.

[0110] Clause 1: A method comprising: receiving an input image; providing the input image to a lane marking detection model; processing the input image using a shared lane marking portion of the lane marking detection model; processing the output of the shared lane marking portion of the lane marking detection model using a plurality of lane marking-specific representation layers of the lane marking detection model to generate a plurality of lane marking representations; and outputting a plurality of lane markings based on the plurality of lane marking representations.

[0111] Clause 2: The method of Clause 1, wherein processing the input image using the shared lane marking portion of the lane marking detection model comprises: processing the input image through multiple shared lane marking representation layers of the lane marking detection model.

[0112] Clause 3: The method of any of Clauses 1-2, wherein outputting the plurality of lane markers comprises, for each of the plurality of lane markers: using a first output layer of the lane marker detection model to predict the horizontal position of a lane vertex; using a second output layer of the lane marker detection model to predict a vertex-by-vertex presence confidence; and using a third output layer of the lane marker detection model to predict a lane-by-lane marker presence confidence.

[0113] Clause 4: As in Clause 3, wherein the multiple lane markings are predicted. i Further includes: predicting each lane marking l according to the following formula i : vl ij It is with each lane marking l i Associated vertex set {(x ij ,y ij )}, T vc It is the existence confidence per vertex, while T lc It is the confidence threshold for the existence of each lane.

[0114] Clause 5: The method of any of Clauses 1-4 further includes: using an encoder-decoder segmentation network to compress the input image.

[0115] Clause 6: The method of any of Clauses 1-5, wherein the final lane mark-specific representation layer for each corresponding lane mark-specific representation comprises a size of h vertical pixels, 1 horizontal pixel, and c channels.

[0116] Clause 7: The method of any of Clauses 1-6 further includes: using one or more horizontal reduction modules to compress the input data in the shared lane marking portion of the lane marking detection model.

[0117] Clause 8: The method of Clause 7 further includes: using one or more additional level reduction modules to compress the input data in the plurality of lane mark-specific representation layers of the lane mark detection model.

[0118] Clause 9: The method of any of Clauses 1-7 further includes: displaying multiple lane representations on the output image.

[0119] Item 10: A method for training a lane marking detection model, comprising minimizing the lane marking detection parameters defined by L = L vl +λ1L vc +λ2L lc The total loss is given as L, where L vl It is the lane marker vertex position loss component, L vc It is the lane marker vertex confidence loss component, L lc λ1 is the confidence loss component for lane-by-lane marking, λ2 is the first loss tuning parameter, and λ2 is the second loss tuning parameter.

[0120] Clause 11: A processing system comprising: a memory including computer-executable instructions; and one or more processors configured to execute the computer-executable instructions and cause the processing system to perform a method according to any one of Clauses 1-10.

[0121] Clause 12: A processing system comprising means for performing a method according to any one of Clauses 1-10.

[0122] Clause 13: A non-transient computer-readable medium comprising computer-executable instructions that, when executed by one or more processors of a processing system, cause the processing system to perform a method according to any one of Clauses 1-10.

[0123] Clause 14: A computer program product implemented on a computer-readable storage medium, the computer-readable storage medium including code for performing a method according to any one of Clauses 1-10.

[0124] Additional considerations

[0125] The foregoing description is provided to enable any person skilled in the art to practice the various aspects described herein. The examples discussed herein are not intended to limit the scope, applicability, or aspects set forth in the claims. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects. For example, the function and arrangement of the elements in discussion may be altered without departing from the scope of this disclosure. Various procedures or components may be appropriately omitted, substituted, or added to the various examples. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Moreover, features described with reference to some examples may be combined in others. For example, any number of aspects set forth herein may be used to implement an apparatus or practice a method. Furthermore, the scope of this disclosure is intended to cover such apparatuses or methods practiced using other structures, functionalities, or structures and functionalities that complement or differ from the aspects of this disclosure set forth herein. It should be understood that any aspect of this disclosure disclosed herein may be implemented by one or more elements of the claims.

[0126] As used herein, the term “exemplary” means “serving as an example, instance, or illustration.” Any aspect described herein as “exemplary” should not be construed as superior to or better than the others.

[0127] As used herein, the phrase “at least one of” refers to any combination of these items, including a single member. As an example, “at least one of a, b, or c” is intended to cover: a, b, c, ab, ac, bc, and abc, as well as any combination with multiple identical elements (e.g., aa, aaa, aab, aac, abb, acc, bb, bbb, bbc, cc, and ccc, or any other ordering of a, b, and c).

[0128] As used herein, the term "determine" encompasses a wide variety of actions. For example, "determine" can include calculation, computation, processing, derivation, research, searching (e.g., looking in a table, database, or other data structure), ascertaining, and the like. Furthermore, "determine" can include receiving (e.g., receiving information), accessing (e.g., accessing data in memory), and the like. Moreover, "determine" can include parsing, selecting, choosing, building, and the like.

[0129] The methods disclosed herein include one or more steps or actions for implementing the method. These method steps and / or actions may be interchanged without departing from the scope of the claims. In other words, unless a specific order of steps or actions is specified, the order and / or use of specific steps and / or actions may be modified without departing from the scope of the claims. Furthermore, the various operations of the above methods can be performed by any suitable means capable of performing the corresponding functions. These means may include various hardware and / or software components and / or modules, including but not limited to circuits, application-specific integrated circuits (ASICs), or processors. Generally, where operations illustrated in the drawings are present, these operations may have corresponding paired means with similar numbers plus functional components.

[0130] The following claims are not intended to be limited to the aspects shown herein, but should be granted the full scope consistent with the language of the claims. Within the claims, references to singular elements are not intended to mean “one and only one” (unless specifically stated so), but rather “one or more.” Unless specifically stated otherwise, the term “some / a” refers to one or more. No element of the claims should be interpreted in accordance with the provisions of 35 USC §112(f) unless the element is expressly stated using the phrase “means for…” or, in the case of a method claim, the element is stated using the phrase “steps for…”. Elements of all aspects described throughout this disclosure that are now or hereafter known to a person skilled in the art are expressly incorporated herein by reference and are intended to be covered by the claims. Furthermore, nothing disclosed herein is intended to be donated to the public, whether or not such disclosure is expressly stated in the claims.

Claims

1. A method comprising: Receive input image; The input image is provided to the lane marking detection model; The input image is processed using the shared lane marking portion of the lane marking detection model. The output of the shared lane marking portion of the lane marking detection model is processed using multiple lane marking-specific representation layers of the lane marking detection model to generate multiple lane marking representations; as well as Multiple lane markings are output based on the multiple lane marking representations. The output of the plurality of lane markings includes, for each of the plurality of lane markings: The first output layer of the lane marking detection model is used to predict the horizontal position of the lane vertices; The second output layer of the lane marking detection model is used to predict the existence confidence per vertex. as well as The third output layer of the lane marking detection model is used to predict the confidence level of the presence of lane markings.

2. The method of claim 1, wherein processing the input image using the shared lane marking portion of the lane marking detection model comprises: The input image is processed by multiple shared lane marking representation layers of the lane marking detection model.

3. The method of claim 1, wherein outputting the plurality of lane markings further comprises: Predict each lane marking using the following formula. l i : ,in vl ij It is with each lane marking l i Associated vertex set {( x ij , y ij )}, x ij It is the horizontal position of the apex of the lane. y ij It is the vertical position of the apex of the lane. vc ij It is the existence confidence level for each vertex. T vc It is the existence confidence threshold for each vertex. l ci It is the confidence level of existence for each lane, and T lc It is the confidence threshold for the existence of each lane.

4. The method of claim 1, further comprising: The input image is compressed using an encoder-decoder segmentation network.

5. The method of claim 1, wherein the final lane mark-specific representation layer for each corresponding lane mark-specific representation includes h The size consists of one vertical pixel, one horizontal pixel, and c channels, where h is the height of the image data, and c is the number of channels in the image.

6. The method of claim 1, further comprising: One or more horizontal reduction modules are used to compress input data in the shared lane marking portion of the lane marking detection model, wherein the horizontal reduction modules can be used as compression or "squeezing" layers in the lane marking detection model.

7. The method of claim 6, further comprising: One or more additional level reduction modules are used to compress the input data in the multiple lane mark-specific representation layers of the lane mark detection model.

8. The method of claim 1, further comprising: The output image displays multiple lane representations.

9. A processing system, comprising: Memory including computer-executable instructions; as well as One or more processors, the one or more processors being configured to execute the computer-executable instructions and enable the processing system to: Receive input image; The input image is provided to the lane marking detection model; The input image is processed using the shared lane marking portion of the lane marking detection model. The output of the shared lane marking portion of the lane marking detection model is processed using multiple lane marking-specific representation layers of the lane marking detection model to generate multiple lane marking representations; as well as Multiple lane markings are output based on the multiple lane marking representations. In order to output the plurality of lane markings, the one or more processors are further configured to make the processing system, for each of the plurality of lane markings: The first output layer of the lane marking detection model is used to predict the horizontal position of the lane vertices; The second output layer of the lane marking detection model is used to predict the existence confidence per vertex. as well as The third output layer of the lane marking detection model is used to predict the confidence level of the presence of lane markings.

10. The processing system of claim 9, wherein, in order to process the input image using the shared lane marking portion of the lane marking detection model, the one or more processors are further configured to cause the processing system to process the input image through a plurality of shared lane marking representation layers of the lane marking detection model.

11. The processing system of claim 9, wherein outputting the plurality of lane markings further comprises: Predict each lane marking using the following formula. l i : ,in vl ij It is with each lane marking l i Associated vertex set {( x ij , y ij )}, x ij It is the horizontal position of the apex of the lane. y ij It is the vertical position of the apex of the lane. vc ij It is the existence confidence level for each vertex. T vc It is the existence confidence threshold for each vertex. l ci It is the confidence level of existence for each lane, and T lc It is the confidence threshold for the existence of each lane.

12. The processing system of claim 9, wherein the one or more processors are further configured to cause the processing system to: compress the input image using an encoder-decoder segmentation network.

13. The processing system of claim 9, wherein the final lane mark-specific representation layer for each corresponding lane mark-specific representation includes h The size consists of one vertical pixel, one horizontal pixel, and c channels, where h is the height of the image data, and c is the number of channels in the image.

14. The processing system of claim 9, wherein the one or more processors are further configured to cause the processing system to: use one or more horizontal reduction modules to compress input data in the shared lane marking portion of the lane marking detection model, wherein the horizontal reduction modules can be used for compression or "squeezing" layers in the lane marking detection model.

15. The processing system of claim 14, wherein the one or more processors are further configured to cause the processing system to: use one or more additional level reduction modules to compress input data in the plurality of lane mark-specific representation layers of the lane mark detection model.

16. The processing system of claim 9, wherein the one or more processors are further configured to cause the processing system to display a plurality of lane representations on an output image.

17. A non-transitory computer-readable medium including instructions that, when executed by one or more processors of a processing system, cause the processing system to perform a method, the method comprising: Receive input image; The input image is provided to the lane marking detection model; The input image is processed using the shared lane marking portion of the lane marking detection model. The output of the shared lane marking portion of the lane marking detection model is processed using multiple lane marking-specific representation layers of the lane marking detection model to generate multiple lane marking representations; as well as Multiple lane markings are output based on the multiple lane marking representations. The output of the plurality of lane markings includes, for each of the plurality of lane markings: The first output layer of the lane marking detection model is used to predict the horizontal position of the lane vertices; The second output layer of the lane marking detection model is used to predict the existence confidence per vertex. as well as The third output layer of the lane marking detection model is used to predict the confidence level of the presence of lane markings.

18. The non-transient computer-readable medium of claim 17, wherein processing the input image using the shared lane marking portion of the lane marking detection model comprises: The input image is processed by multiple shared lane marking representation layers of the lane marking detection model.

19. The non-transient computer-readable medium of claim 17, wherein outputting the plurality of lane markings further comprises: Predict each lane marking using the following formula. l i : ,in vl ij It is with each lane marking l i Associated vertex set {( x ij , y ij )}, x ij It is the horizontal position of the apex of the lane. y ij It is the vertical position of the apex of the lane. vc ij It is the existence confidence level for each vertex. T vc It is the existence confidence threshold for each vertex. l ci It is the confidence level of existence for each lane, and T lc It is the confidence threshold for the existence of each lane.

20. The non-transient computer-readable medium of claim 17, wherein the method further comprises: The input image is compressed using an encoder-decoder segmentation network.

21. The non-transient computer-readable medium of claim 17, wherein the final lane mark-specific representation layer for each corresponding lane mark-specific representation includes h The size consists of one vertical pixel, one horizontal pixel, and c channels, where h is the height of the image data, and c is the number of channels in the image.

22. The non-transient computer-readable medium of claim 17, wherein the method further comprises: One or more horizontal reduction modules are used to compress input data in the shared lane marking portion of the lane marking detection model, wherein the horizontal reduction modules can be used as compression or "squeezing" layers in the lane marking detection model.

23. The non-transient computer-readable medium of claim 22, wherein the method further comprises: One or more additional level reduction modules are used to compress the input data in the multiple lane mark-specific representation layers of the lane mark detection model.

24. The non-transient computer-readable medium of claim 17, wherein the method further comprises: The output image displays multiple lane representations.

25. A processing system, comprising: A device for receiving input images; A means for providing the input image to a lane marking detection model; A means for processing the input image using the shared lane marking portion of the lane marking detection model; A means for processing the output of the shared lane marking portion of the lane marking detection model using multiple lane marking-specific representation layers of the lane marking detection model to generate multiple lane marking representations; as well as A means for outputting multiple lane markings based on the multiple lane marking representations. The means for outputting the plurality of lane markings includes means for performing the following operations for each of the plurality of lane markings: The first output layer of the lane marking detection model is used to predict the horizontal position of the lane vertices; The second output layer of the lane marking detection model is used to predict the existence confidence per vertex. as well as The third output layer of the lane marking detection model is used to predict the confidence level of the presence of lane markings.

Images