Lane Marker Detection

Abstract

Some aspects of the present disclosure provide a method for lane marker detection that includes receiving an input image, providing the input image to a lane marker detection model, processing the input image with a shared lane marker portion of the lane marker detection model, processing an output of the shared lane marker portion of the lane marker detection model with multiple lane marker-specific representation layers of the lane marker detection model to generate multiple lane marker representations, and outputting multiple lane markers based on the multiple lane marker representations.

Description

[Technical field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims priority to U.S. Patent Application No. 17 / 200,592, filed March 12, 2021, which claims the benefit of and priority to Provisional Patent Application No. 62 / 988,795, filed March 12, 2020, the entire contents of each of which are incorporated herein by reference.

[0002] Aspects of the present disclosure relate to systems and methods for fast and robust lane detection. [Background technology]

[0003] Modern vehicles are increasingly equipped with advanced driver assistance systems, which may include lane detection, among other things, for assisted autonomous driving functions. Existing techniques for lane detection are slow, require significant manual configuration, and lack robustness across many driving scenarios. Thus, existing systems for lane detection are not suitable for modern vehicles.

[0004] Therefore, what is needed is a system and method that provides fast and robust lane detection. Summary of the Invention [Means for solving the problem]

[0005] Some aspects provide a method for lane marker detection that includes receiving an input image, providing the input image to a lane marker detection model, processing the input image with a shared lane marker portion of the lane marker detection model, processing an output of the shared lane marker portion of the lane marker detection model with multiple lane marker-specific representation layers of the lane marker detection model to generate a plurality of lane marker representations, and outputting a plurality of lane markers based on the plurality of lane marker representations.

[0006] Other aspects provide a processing system configured to perform the methods described above and further described herein, a non-transitory computer readable medium comprising instructions that, when executed by one or more processors of the processing system, cause the processing system to perform the methods described above and further described herein, a computer program product embodied on a computer readable storage medium comprising code for performing the methods described above and further described herein, and a processing system comprising means for performing the methods described above and further described herein.

[0007] The accompanying drawings illustrate some aspects of the one or more aspects and therefore should not be considered limiting of the scope of the disclosure. [Brief description of the drawings]

[0008] [Figure 1] FIG. 2 is a diagram showing an example of lane markers on a road. [Diagram 2] FIG. 1 illustrates aspects of an exemplary advanced driver assistance system or automated driving system. [Diagram 3] 1 illustrates aspects of an example lane marker detection model architecture. [Figure 4] FIG. 1 illustrates an example horizontal shrinking module that may be used to compress (or “squeeze”) layers in a lane marker detection model. [Diagram 5] FIG. 1 illustrates an exemplary method for detecting lane markers. [Figure 6] FIG. 2 illustrates an example processing system that may be configured to detect lane markers as described herein. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] For ease of understanding, wherever possible, like reference numbers have been used to designate like elements common to the figures. It is contemplated that elements and features of one embodiment may be beneficially incorporated into other embodiments without further recitation.

[0010] The systems and methods disclosed herein enable fast and robust lane detection in a variety of applications, for example, the systems and methods disclosed herein may enable accurate and robust localization of lane markers in real time in vehicles with assisted autonomous driving capabilities.

[0011] Generally, lane markers are devices or materials on a road surface that convey information such as where lanes exist on a roadway. Examples of lane markers include painted traffic lines, painted crosswalks, painted parking spaces, handicapped parking spaces, reflective markers, curbs, gutters, Botts' dots, and speed reduction pavements, to name a few.

[0012] Lane markers may be used in autonomous vehicle navigation. For example, a vehicle may include an advanced driver assistance system (ADAS) or a high level of self-driving system (SDS). Such systems are becoming widely adopted, based in large part on contemporaneous improvements in computer vision technology.

[0013] Although there are several components involved in ADAS and SDS systems, such as lane marker detection, vehicle detection and tracking, obstacle detection, scene understanding, and semantic segmentation, lane detection is a key component for camera perception and positioning. For example, lane detection is necessary to keep the vehicle within its own lane (sometimes called the host lane, the lane in which it is located) and to assist the vehicle in changing lanes to the left or right of its own lane.

[0014] Most conventional lane marker detection methods are based on a two-stage semantic segmentation approach. In the first stage of such an approach, a network is designed to perform pixel-level classification, assigning each pixel in an image to a binary label, i.e., lane or non-lane. However, in each pixel classification, dependencies or structures between pixels are not considered, and therefore additional post-processing is performed in the second stage to explicitly impose constraints such as uniqueness or straightness of the detected line. Post-processing can be applied using sophisticated computer vision techniques, such as conditional random fields, additional networks, or random sample consensus (RANSAC), but these post-processing techniques are computationally intensive and generally require careful manual integration into the lane marker detection system. Therefore, two-stage semantic segmentation approaches are difficult to deploy in various applications.

[0015] Another issue with current lane marker detection methods is power consumption. Processing each image in a stream of images can require a large amount of computational power that is not always available or practical in an automotive context. Particularly for electric vehicles, power consumption of on-board systems is an important consideration.

[0016] In contrast, aspects described herein relate to an efficient end-to-end architecture that directly predicts lane marker vertices without any costly post-processing steps. More specifically, in aspects described herein, the lane marker detection problem is viewed as a column-wise classification task where the structural shapes of lane markers that extend from left to right in an image are modeled in two subsets of layers in a single model. A first subset of model layers is configured to compress and model horizontal components for a shared representation of all lanes, and a second subset of model layers is configured to model each lane based on the shared representation to directly output lane marker vertices. In some aspects, the target lane marker positions are efficiently obtained at run time by an argmax function.

[0017] Thus, the aspects described herein beneficially improve upon conventional approaches for lane detection without complex post-processing, which reduces computational power, computational time, and power usage when in use. Thus, the aspects described herein may be beneficially deployed in a greater number of applications and contexts.

[0018] Exemplary lane markers on a road 1 shows an example of lane markers 102 on a road 104. In this example, the lane markers 102 are painted stripes on the road 104, but in other examples, the lane markers may be any type of recognizable division of lanes. The lane markers 102 may be detected by a lane marker detection system and may be used to determine lanes for advanced driver assistance systems as well as semi-autonomous and autonomous driving systems in vehicles.

[0019] Exemplary Electronic Lane Marker Detection System 2 illustrates aspects of an example advanced driver assistance or automated driving system 200. The system 200 can be configured to operate with local sensors and edge processing capabilities, such as those provided by a mapping algorithm 218.

[0020] In this embodiment, the system 200 includes one or more cameras 202, a global positioning system (GPS) 204, an inertial measurement unit (IMU) 206, a sensor synchronization board 208, and one or more processors 220. In this example, the processor 220 includes a sensor driver module 210, a positioning engine 212, a perception engine 214, and a data aggregation and connectivity module 216. The processor 220 may generally communicate with one or more devices via a wireless or wired network.

[0021] In some embodiments, the front-end sensors, such as the camera 202, GPS receiver 204, and IMU 206, may be consumer-grade sensors. The sensor synchronization board 208 may include an embedded microcontroller that controls the timestamps of all the sensors. In some embodiments, the sensor synchronization board 208 may generate timestamps with a timing error of less than 10 microseconds.

[0022] The outputs of the camera 202, GPS 204, and IMU 206 may be fed into both a positioning engine 212 and a perception engine 214. The perception engine 214 is configured to detect important landmarks (e.g., lane markers) in the incoming video stream and precisely locate them in the image frames. The positioning engine 212 is configured to provide an accurate estimate of the camera pose, e.g., in six degrees of freedom, by correlating the GPS signal, inertial sensor readings, and camera video input.

[0023] The output of the positioning engine 212 and the perception engine 214 may be aggregated and sent to an edge processing service, such as a cloud processing system, via a data aggregation and connectivity module 216. In some aspects, the mapping algorithm 218 may generate location estimates of landmarks in a global frame, where the global frame includes the entire map or a large portion of the map.

[0024] Lane Marker Detection Model Architecture Unlike conventional segmentation-based lane detection techniques, the embodiments described herein directly recognize lane marker locations in the image input data. More specifically, given an input image X∈R, where h is the height of the image data (e.g., in pixels), w is the width of the image data (e.g., in pixels), and c is the number of channels in the image: h×w×c Given, the target is the lane marker l i (i=1,...,N lane ) vertex set {vl ij}={(x ij ,y ij )}(j=1,...,K) where N lane is the number of lanes in the image, which is typically predefined, and K is the total number of vertices.

[0025] The methods described herein reduce the complexity of the above-mentioned goal by finding a set of horizontal locations of lane markers in an image. In particular, a column-wise representation for lane markers is determined by splitting the image into columns and using a convolutional neural network to solve the lane marker detection task as a column-wise location classification problem or a regression problem depending on the loss function selected. Note that with column-wise representation, the total possible number of vertices K is limited to the image height h. In some aspects, the network may perform column-wise regression to identify lane locations, as described in more detail below.

[0026] The lane marker detection model described herein (e.g., a convolutional neural network model) makes three predictions: (1) the horizontal location x of the lane vertex; ij (2) The existence reliability of each vertex, vc ij , and (3) the presence reliability of each lane marker. ci From these model outputs, the following equation is obtained:

[0027]

number

[0028] Each lane marker i can be obtained, where vl ij is the set of vertices associated with the lane markers {(x ij ,y ij )}, and T vc and T lc are the vertex-wise and lane-wise presence confidence thresholds, respectively.

[0029] Figure 3 shows aspects of an example lane marker detection model architecture as may be implemented by the perception engine 214 in Figure 2. The lane marker detection model may be configured as a deep neural network trained to predict the location of lane markers in input image data (e.g., image 301), which may then be used to predict lane delineations (e.g., 322 in output image 321).

[0030] In the following description, lane delineation may generally refer to the boundary of a driving lane on a road. For example, the current lane in which a vehicle is driving, i.e., the own lane, may include two delineation boundaries that define the lane on the left and right sides. Similarly, the lane to the left of the current lane may include its own two delineation boundaries, which may in some cases include a shared boundary delineation. For example, the left boundary delineation of the current lane may represent the right boundary delineation of the lane to the left of the current lane.

[0031] In this example, the model architecture 300 consists of three in-model stages 302A-C. The first stage 302A includes an encoder-decoder type segmentation network 304 configured to encode information of lane markers in an image. Unlike conventional image segmentation, in this example, the encoder-decoder type segmentation network 304 is configured to restore half the spatial resolution of the input image 301 to beneficially reduce the amount of computation. In other words, in this example, the input image 301 includes an input resolution of 128 height x 512 width x 3 channels (e.g., red, green, blue), and the output of the encoder-decoder type segmentation network 304 is 128 height x 256 width x 3 channels. Notably, these resolutions and other resolutions shown in FIG. 3 are only examples, and other resolutions are possible. Furthermore, other embodiments may omit the encoder-decoder network in favor of other image data compression techniques, including other types of neural network blocks.

[0032] The second stage 302B includes layers that are shared by all of the potential lane marker representations and may be referred to as shared reduction layers, which in this example include three layers 306, 308, and 310. In the second stage 302B, the horizontal (h) dimensions of the shared lane marker representations are progressively compressed (or "squeezed") in each successive layer using a horizontal reduction module (HRM) without changing the vertical dimensions. An exemplary HRM is described with respect to FIG. 4. As the HRM successively squeezes the spatial width components, this form of compression operation naturally leads to a row-by-row representation, so that, for example, there is only one lane vertex in each row (width) in the final feature of each lane.

[0033] The third stage 302C includes a lane-specific reduction layer, which in this example includes layers 314, 316, 318, and 320. In the third stage 302C, the lane-specific reduction layer further compresses the lane marker representations using the lane-specific HRM to generate a single-vector lane marker representation, such as 316.

[0034] In particular, the shared representation in stage 302B and the lane-specific representation in stage 302C allow for tuning of the overall model performance. For computational efficiency, a first subset of reduction layers is shared across lanes in stage 302B, followed by a second subset of reduction layers per lane in stage 302C. Because each lane marker representation has a priori different spatial and shape properties, using a dedicated reduction layer for each lane marker representation in stage 302C improves the overall model performance. With more shared layers, additional computational costs may be saved, but the accuracy of each lane may be degraded. Thus, the performance may be tuned for the application.

[0035] Stage 302C is lane-specific depiction i The compression layer 316 includes two branches per column (six of which are depicted in FIG. 3 as an example), namely, a column-wise vertex location branch 320 and a vertex-wise confidence branch 318, configured to perform classification and confidence regression on the final (and fully) compressed layer 316. In this embodiment, the final (and fully) compressed layer 316 has a spatial resolution in the vertical dimension only, and a number of channels according to a target horizontal resolution h′ (e.g., h′=h / 2). The classification branch 320 classifies the predicted vertices as (x ij ,y ij ), y ij ∈h′, where l is the lane marker i Horizontal position x ij The reliability branch 318 predicts each vertex vl of the lane marker. ij The vertex reliability vc related to whether it exists or not ij Predict.

[0036] Finally, a lane marker presence confidence layer 312 calculates the confidence after the shared HRM. ciLane marker presence relates to whether semantic lane markers (e.g., left marker of current lane, right marker of current lane, left marker of lane one left, right marker of lane one left, left marker of lane one right, right marker of lane one right, etc.) are detected in the image data.

[0037] FIG. 4 illustrates an example horizontal reduction module 400 that may be used to compress (or "squeeze") layers as described in FIG.

[0038] To effectively compress the horizontal lane marker representation, a residual layer is utilized in the horizontal reduction module 400. In particular, a horizontal average pooling layer with 1×1 convolutions is added to the skip connection 402 to the downsampled horizontal components. The pooling operation allows deeper layers to gather more spatial context (to improve classification) and reduce computational complexity, but it also has the drawback of reducing pixel precision. Therefore, to effectively preserve and expand the horizontal representation of C×H×W elements, the input tensor (X) is rearranged into a tensor of shape rC×H×W / r in the residual branch 404. This operation is sometimes referred to as a horizontal pixel unshuffle layer. By rearranging the representation of the input tensor, spatial information can be efficiently moved to the channel dimension.

[0039] After rearranging the input tensors, a convolution operation in block 406 may be applied to reduce the increased channel dimensionality r back to the original channel dimensionality C, which not only reduces the computational burden but also helps to effectively compress the lane marker spatial information from the pixel unshuffle layer.

[0040] The output of the residual branch 404 is then combined with the skip connection 402 at 410, where the combined result is then provided to an activation function, such as ReLU (412) in this example.

[0041] To further improve the discrimination between lane markers, an attention mechanism may be added by the Squeeze and Excitation (SE) block 408. The SE block helps to include global information in the decision process by aggregating information into the overall receptive field and recalibrates the per-channel feature responses with the spatial information encoded by the pixel unshuffle layer.

[0042] Thus, the horizontal reduction module 400 produces a compressed horizontal representation output

[0043]

number

[0044] To generate, the spatial lane marker information is usefully compressed.

[0045] Exemplary Training Method for a Lane Marker Detection Model In one example, the training goal may be: L=L vl +λ1L vc +λ2L lc The goal is to optimize the total loss L given by vl , L vc , and L lc are the losses for lane marker vertex location, lane marker vertex confidence, and lane marker-wise confidence, respectively, and λ1 and λ2 are weights for the last two losses that allow fine-tuning the training.

[0046] Lane marker vertex location loss (L vl), since lane marker detection is formulated as a column-wise classification on the horizontal positions of the lane markers, any loss function for classification can be used to train the lane marker vertex bifurcation, including cross-entropy loss, Kullback-Leibler (KL) divergence loss, and piecewise linear probability (PL) loss, to name a few.

[0047] vertical position y ij Lane markers in i Cross entropy loss for

[0048]

number

[0049] is the ground truth location

[0050]

number

[0051] , and the predicted logit f with W=2 channels ij It can be calculated using:

[0052] KL divergence loss

[0053]

number

[0054] To train the lane marker vertex location branch using, we set a sharp-peaked target distribution of lane marker locations,

[0055]

number

[0056] and Laplace distribution with b=1 gt (μ, b), then

[0057]

number

[0058] and

[0059]

number

[0060] The estimated distribution Laplace pred It can be compared with (μ,b).

[0061] In the case of PL loss, the lane marker position probability may be modeled as a piecewise linear probability distribution. For an input image, the overall lane marker vertex location loss is

[0062]

number

[0063] where e ij is y ij Lane marker l with vertex at i We indicate whether ground truth is present or not by setting 1 if it is present and 0 otherwise.

[0064] For lane marker vertex confidence loss, lane marker vertex presence is a binary classification problem, and therefore lane marker l i Each y ij Single scalar value prediction and ground truth presence in location ij The binary cross-entropy loss between

[0065]

number

[0066] The loss for the entire image is then trained using

[0067]

number

[0068] It is calculated as:

[0069] For the lane marker label loss, we use binary cross-entropy loss to train lane marker-level presence prediction.

[0070]

number

[0071] may be added. The loss is calculated based on the predicted N lane dimensional vector, and each lane in the ground truth i The total loss is then calculated using the existence of

[0072]

number

[0073] It is calculated as:

[0074] Inference using lane marker detection model At inference time, estimated vertex locations may be predicted based on a selected loss function (e.g., such as from the loss functions described above). In particular, if cross-entropy loss or PL loss is used, lane marker vertices may be selected by using an argmax function. If KL divergence loss is used instead, estimated vertices may be extracted by using softargmax. To consider the presence of each lane marker vertex, the sigmoid output of the vertices and the presence branch per lane may be used to reject low confidence locations according to Equation 1 above.

[0075] Exemplary Methods for Detecting Lane Markers FIG. 5 illustrates an example method 500 for detecting lane markers.

[0076] Method 500 begins at step 502 with receiving an input image. In some examples, the input image comprises h vertical pixels, w horizontal pixels, and c channels, for example, as described with respect to Figure 3. The input image may generally be a still image, an image or frame of a stream of images in a video stream, etc.

[0077] The method 500 then proceeds to step 504, which involves providing the input image to a lane marker detection model, such as the model described above with respect to FIG.

[0078] The method 500 then proceeds to step 506, which involves processing the input image with the shared lane marker portion of the lane marker detection model, as described above with respect to 302B in Figure 3. In some cases, processing the input image with the shared lane marker portion of the lane marker detection model comprises processing the input image through multiple shared lane marker representation layers of the lane marker detection model.

[0079] The method 500 then proceeds to step 508, which involves processing the output of the shared lane marker portion of the lane marker detection model with multiple lane marker-specific representation layers of the lane marker detection model to generate multiple lane marker representations, such as described above with respect to 302C in Figure 3. In some aspects, each lane marker-specific representation layer is associated with one lane marker representation of the multiple lane marker representations, and each lane marker representation may be associated with a subset of the multiple lane marker-specific representation layers.

[0080] The method 500 then proceeds to step 510 with outputting a plurality of lane markers based on the plurality of lane marker representations.

[0081] In some embodiments of the method 500, a plurality of lane markers (e.g., lane i ) may include, for each lane marker of the plurality of lane markers, detecting the horizontal location of the lane vertex (e.g., x ij ) and the second output layer of the lane marker detection model is used to predict the presence confidence per vertex (e.g., vc ij ) and the third output layer of the lane marker detection model is used to predict the presence confidence of each lane marker (e.g., l ci ) and

[0082] In some aspects of the method 500, outputting the plurality of lane markers is performed according to the following formula:

[0083]

number

[0084] Use each lane marker i , further comprising predicting vl ij Each lane marker i The set of vertices associated with x ij ,y ij )}.

[0085] In some aspects, the method 500 further comprises compressing the input image using an encoder-decoder segmentation network.

[0086] In some aspects of method 500, the final lane marker specific representation layer for each respective lane marker specific representation has a size of h vertical pixels, 1 horizontal pixel, and c channels.

[0087] In some aspects, the method 500 further comprises using one or more horizontal reduction modules to compress the input data within the shared lane marker portion of the lane marker detection model, such as described above with respect to Figures 3 and 4.

[0088] In some aspects, the method 500 further comprises using one or more additional horizontal reduction modules to compress the input data within multiple lane marker specific representation layers of the lane marker detection model, such as described above with respect to Figures 3 and 4.

[0089] In some aspects, the method 500 further comprises displaying a plurality of lane depictions over the output image, such as shown in FIG. 3 (eg, 322).

[0090] It should be noted that method 500 is one exemplary method and that other methods are possible, in particular, other examples may include fewer, additional, and / or alternative steps as compared to method 500, consistent with various aspects described herein.

[0091] Exemplary Processing System FIG. 6 illustrates an example processing system 600 that may be configured to detect lane markers, for example, as described herein with respect to method 500 of FIG.

[0092] The processing system 600 includes a central processing unit (CPU) 602, which in some examples may be a multi-core CPU. Instructions executed in the CPU 602 may be loaded from a program memory associated with the CPU 602 or may be loaded from a memory partition 624, for example.

[0093] The processing system 600 also includes additional processing components tailored to specific functions, such as a graphics processing unit (GPU) 604, a digital signal processor (DSP) 606, a neural processing unit (NPU) 608, a multimedia processing unit 610, and wireless connectivity components 612. In some examples, one or more of the CPU 602, GPU 604, DSP 606, and NPU 608 may serve as the processor 220 of FIG.

[0094] An NPU, such as 608, is generally a specialized circuit configured to implement all the necessary control and computational logic to execute machine learning algorithms, such as algorithms for processing artificial neural networks (ANN), deep neural networks (DNN), random forests (RF), etc. An NPU may alternatively be referred to as a neural signal processor (NSP), tensor processing unit (TPU), neural network processor (NNP), intelligence processing unit (IPU), vision processing unit (VPU), or graph processing unit.

[0095] An NPU, such as 608, is configured to accelerate the execution of common machine learning tasks, such as image classification, machine translation, object detection, and various other predictive models. In some examples, multiple NPUs may be instantiated on a single chip, such as a system-on-chip (SoC), and in other examples, may be part of a dedicated neural network accelerator.

[0096] An NPU may be optimized for training or inference, or in some cases may be configured to balance performance between both. For NPUs capable of performing both training and inference, the two tasks may still be performed generally independently.

[0097] NPUs designed to accelerate training are generally configured to accelerate the optimization of new models, which is a highly computationally intensive operation that involves inputting an existing dataset (often labeled or tagged), iterating over the dataset, and then adjusting model parameters such as weights and biases to improve model performance. In general, optimization based on erroneous predictions involves propagating backwards through layers of the model and determining gradients to reduce prediction errors. In some embodiments, the NPU 608 may be configured to train various aspects of the model architecture 300 of FIG. 3.

[0098] NPUs designed to accelerate inference are generally configured to operate on complete models. Thus, such NPUs may be configured to input new data and rapidly process the data through already trained models to generate model outputs (e.g., inferences). In some embodiments, the NPU 608 may be configured to process various aspects of the model architecture 300 of FIG. 3.

[0099] In one implementation, the NPU 608 is part of one or more of the CPU 602, the GPU 604, and / or the DSP 606.

[0100] In some examples, the wireless connectivity component 612 may include subcomponents for, for example, third generation (3G) connectivity, fourth generation (4G) connectivity (e.g., 4G LTE), fifth generation connectivity (e.g., 5G or NR), Wi-Fi connectivity, Bluetooth connectivity, and other wireless data transmission standards. The wireless connectivity processing component 612 is further connected to one or more antennas 614.

[0101] The processing system 600 may also include one or more sensor processing units 616 associated with any type of sensor, one or more image signal processors (ISPs) 618 associated with any type of image sensor, and / or a navigation processor 620, which may include satellite-based positioning system components (e.g., GPS or GLONASS), as well as inertial positioning system components. In some examples, the sensors 616 may include the camera 202 and the IMU 206 of FIG. 2. In some examples, the navigation processor 620 may include the GPS 204 of FIG. 2.

[0102] The processing system 600 may also include one or more input and / or output devices 622, such as a screen, a touch-sensitive surface (including a touch-sensitive display), physical buttons, a speaker, a microphone, etc.

[0103] In some examples, one or more of the processors of processing system 600 may be based on the ARM or RISC-V instruction set.

[0104] Processing system 600 also includes memory 624, which represents one or more static and / or dynamic memories, such as dynamic random access memory, flash-based static memory, etc. In this example, memory 624 includes computer-executable components that may be executed by one or more of the above-mentioned processors of processing system 600.

[0105] In this example, memory 624 includes a receiving component 624A (e.g., for receiving input data to be processed by the lane marker detection model), a processing component 624B (e.g., for processing various aspects of the lane marker detection model described herein), a compression component 624C (e.g., for compressing data such as by a horizontal reduction module), a prediction (or output) component 624D (e.g., such as various outputs of the lane marker detection model described herein), a training component 624E (e.g., for training the lane marker detection model), an inference component 624F, an encoding component 624G (e.g., for encoding input data), a decoding component 624H (e.g., for decoding the encoded data), a display component 624I (e.g., for displaying lane markers and other information), and a model parameters 624J (e.g., comprising parameters for the lane marker detection model as described herein). The illustrated components and other components not shown may be configured to perform various aspects of the methods described herein.

[0106] For example, the receiving component 624A may be configured to receive input data, such as input image data.

[0107] Notably, in other embodiments, aspects of the processing system 600 may be omitted, such as when the processing system 600 is a server computer, etc. For example, the multimedia components 610, the wireless connectivity 612, the sensors 616, the ISP 618, and / or the navigation components 620 may be omitted in other embodiments. Additionally, aspects of the processing system 600 may be distributed.

[0108] It should be noted that FIG. 6 is just one example, and that in other examples, alternative processing systems having fewer, additional, and / or alternative components may be used.

[0109] Example clauses Example implementations are described in the numbered clauses below.

[0110] Clause 1: A method comprising: receiving an input image; providing the input image to a lane marker detection model; processing the input image with a shared lane marker portion of the lane marker detection model; processing an output of the shared lane marker portion of the lane marker detection model with a plurality of lane marker specific representation layers of the lane marker detection model to generate a plurality of lane marker representations; and outputting a plurality of lane markers based on the plurality of lane marker representations.

[0111] Clause 2: The method of clause 1, wherein processing the input image with the shared lane marker portion of the lane marker detection model comprises processing the input image through a plurality of shared lane marker representation layers of the lane marker detection model.

[0112] Clause 3: The method of any one of clauses 1 to 2, wherein outputting the plurality of lane markers comprises, for each lane marker of the plurality of lane markers, predicting a horizontal location of a lane vertex using a first output layer of the lane marker detection model, predicting a presence confidence for each vertex using a second output layer of the lane marker detection model, and predicting a presence confidence for each lane marker using a third output layer of the lane marker detection model.

[0113] Clause 4: The method of clause 3, further comprising: i To predict

[0114]

number

[0115] According to each lane marker i Further comprising predicting vl ij Each lane marker i The set of vertices associated with x ij ,y ij )}, and Tvc is the presence confidence threshold for each vertex, T lc is the presence confidence threshold for each lane.

[0116] Clause 5: The method of any one of clauses 1 to 4, further comprising compressing the input image using an encoder-decoder segmentation network.

[0117] Clause 6: The method of any one of clauses 1 to 5, wherein the final lane marker specific representation layer for each respective lane marker specific representation has a size of h vertical pixels, 1 horizontal pixel, and c channels.

[0118] Clause 7: The method of any one of clauses 1-6, further comprising using one or more horizontal reduction modules to compress the input data within the shared lane marker portion of the lane marker detection model.

[0119] Clause 8. The method of clause 7, further comprising using one or more additional horizontal reduction modules to compress the input data within a plurality of lane marker specific representation layers of the lane marker detection model.

[0120] Clause 9: The method of any one of clauses 1-7, further comprising displaying a plurality of lane depictions on the output image.

[0121] Clause 10: L=L vl +λ1L vc +λ2L lc A method of training a lane marker detection model comprising minimizing a total loss L given by: vl is the lane marker vertex location loss component, L vc is the lane marker vertex reliability loss component, L lc is the reliability loss component for each lane marker, λ1 is the first loss adjustment parameter, and λ2 is the second loss adjustment parameter.

[0122] Clause 11: A processing system comprising a memory comprising computer-executable instructions and one or more processors configured to execute the computer-executable instructions to cause the processing system to perform a method according to any one of clauses 1 to 10.

[0123] Clause 12: A processing system comprising means for carrying out the method according to any one of clauses 1 to 10.

[0124] Clause 13: A non-transitory computer-readable medium comprising computer-executable instructions that, when executed by one or more processors of a processing system, cause the processing system to perform a method according to any one of clauses 1 to 10.

[0125] Clause 14: A computer program product embodied on a computer-readable storage medium comprising code for performing the method according to any one of clauses 1 to 10.

[0126] Additional Considerations The preceding description is provided to enable any person skilled in the art to practice the various aspects described herein. The examples described herein are not intended to limit the scope, applicability, or aspects described in the claims. Various modifications of these aspects will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects. For example, changes may be made in the function and arrangement of the elements described without departing from the scope of the disclosure. Various examples may omit, substitute, or add various procedures or components, as appropriate. For example, the methods described may be performed in an order different from that described, and various steps may be added, omitted, or combined. Also, features described with respect to some examples may be combined in some other examples. For example, an apparatus may be implemented or a method may be practiced using any number of the aspects described herein. Additionally, the scope of the disclosure is intended to cover such apparatus or methods practiced using other structures, functionality, or structures and functionality in addition to or other than the various aspects of the disclosure described herein. It should be understood that any aspect of the disclosure disclosed herein may be embodied by one or more elements of a claim.

[0127] The word "exemplary" is used herein to mean "serving as an example, instance, or illustration." Any aspect described herein as "exemplary" is not necessarily to be construed as preferred or advantageous over other aspects.

[0128] As used herein, a phrase referring to "at least one of" a list of items refers to any combination of those items, including single members. As an example, "at least one of a, b, or c" is intended to cover a, b, c, ab, ac, bc, and abc, as well as any combination having multiples of the same element (e.g., aa, aaa, aab, aac, abb, acc, bb, bbb, bbc, cc, and ccc, or any other permutation of a, b, and c).

[0129] As used herein, the term "determining" encompasses a wide variety of actions. For example, "determining" may include calculating, computing, processing, deriving, investigating, looking up (e.g., looking up in a table, database, or another data structure), ascertaining, and the like. Also, "determining" may include receiving (e.g., receiving information), accessing (e.g., accessing data in a memory), and the like. Also, "determining" may include resolving, selecting, choosing, establishing, and the like.

[0130] The methods disclosed herein comprise one or more steps or actions for achieving the method. The steps and / or actions of the methods may be interchanged with one another without departing from the scope of the claims. In other words, unless a specific order of steps or actions is specified, the order and / or use of specific steps and / or actions may be modified without departing from the scope of the claims. Furthermore, various operations of the methods described above may be performed by any suitable means capable of performing the corresponding functions. The means may include various hardware and / or software components and / or modules, including but not limited to circuits, application specific integrated circuits (ASICs), or processors. In general, where there are operations illustrated in figures, those operations may have corresponding counterpart means-plus-function components with similar numbering.

[0131] The following claims are not limited to the embodiments set forth herein, but are to be accorded the full scope consistent with the language of the claims. Within the claims, reference to an element in the singular does not mean "the one and only" unless expressly stated as such, but means "one or more." Unless otherwise expressly stated, the term "several" refers to one or more. No element of a claim is to be construed under the provisions of 35 U.S.C. 112(f) unless the element is expressly recited using the phrase "means for" or, in the case of a method claim, unless the element is recited using the phrase "step for." All structural and functional equivalents of the elements of the various embodiments described throughout this disclosure that are known or later become known to those skilled in the art are expressly incorporated herein by reference and are encompassed by the claims. Moreover, nothing disclosed herein is made public, regardless of whether such disclosure is expressly recited in the claims. [Explanation of symbols]

[0132] 102 Lane Markers 104 Road 200 Advanced Driver Assistance Systems or Automated Driving Systems 202 Camera 204 Global Positioning System (GPS), GPS receiver 206 Inertial Measurement Unit (IMU) 208 Sensor Synchronization Board 210 Sensor Driver Module 212 Positioning Engine 214 Perception Engine 216 Data Aggregation and Connectivity Module 218 Mapping Algorithms 220 Processor 301 Input images 302 Stage 304 Encoder-Decoder Segmentation Network 312 Lane Marker Presence Reliability Layer 316 Final compacted layer 318 Vertex-wise Reliability Branching 320 Column-wise vertex location split, classification split 321 Output Image 322 Lane drawing 400 Horizontal Shrink Module 402 Skip Connection 404 Residual Branch 408 Squeezing and Excitation (SE) Block 412 ReLU 600 Processing System 602 Central Processing Unit (CPU) 604 Graphics Processing Unit (GPU) 606 Digital Signal Processor (DSP) 608 Neural Processing Unit (NPU) 610 Multimedia Processing Units, Multimedia Components 612 Wireless connectivity components, wireless connectivity processing components 614 Antenna 616 Sensor Processing Unit, Sensor 618 Image Signal Processor (ISP) 620 Navigation processor, navigation component 622 Input and / or Output Devices 624 Memory partition, memory 624A Receiver Components 624B Processing Components 624C Compression Components 624D Predictive Components 624E Training Components 624F Inference Components 624G Encoding Components 624H Decryption Component 624I Display component 624J Model Parameter

Claims

1. receiving an input image; providing the input image to a lane marker detection model; processing the input image with a shared lane marker portion of the lane marker detection model; processing an output of the shared lane marker portion of the lane marker detection model with a plurality of lane marker specific reduction layers of the lane marker detection model to generate a plurality of lane marker representations; outputting a plurality of lane markers based on the plurality of lane marker representations; A method comprising: The step of outputting a plurality of lane markers includes, for each lane marker of the plurality of lane markers, predicting horizontal locations of lane vertices using a first output layer of the lane marker detection model; predicting a presence confidence for each vertex using a second output layer of the lane marker detection model; and predicting a presence confidence for each lane marker using a third output layer of the lane marker detection model. method.

2. 2. The method of claim 1 , wherein processing the input image with the shared lane marker portion of the lane marker detection model comprises processing the input image through a plurality of shared lane marker reduction layers of the lane marker detection model.

3. outputting the plurality of lane markers, [0010] According to each lane marker i and further comprising the step of predicting vl ij However, each lane marker i The set of vertices associated with x ij ,y ij )}, and T vc is the existence confidence threshold for each vertex, and T lc The method of claim 1 , wherein x is a per-lane presence confidence threshold.

4. The method of claim 1 , further comprising compressing the input image using an encoder-decoder segmentation network.

5. The method of claim 1 , wherein a final lane marker-specific reduced layer for each respective lane marker-specific representation has a size of h vertical pixels, 1 horizontal pixel, and c channels.

6. using one or more horizontal reduction modules to compress input data within the shared lane marker portion of the lane marker detection model; using one or more additional horizontal reduction modules to compress input data within the plurality of lane marker specific reduction layers of the lane marker detection model. The method of claim 1.

7. The method of claim 1 , further comprising displaying a plurality of lane depictions over the output image.

8. 1. A processing system comprising: a memory having computer-executable instructions; Executing the computer-executable instructions, the processing system Receiving an input image; providing the input image to a lane marker detection model; processing the input image with a shared lane marker portion of the lane marker detection model; processing an output of the shared lane marker portion of the lane marker detection model with a plurality of lane marker specific reduction layers of the lane marker detection model to generate a plurality of lane marker representations; outputting a plurality of lane markers based on the plurality of lane marker representations. One or more processors Equipped with To output the plurality of lane markers, the one or more processors may cause the processing system to, for each lane marker of the plurality of lane markers: predicting horizontal locations of lane vertices using a first output layer of the lane marker detection model; predicting a presence confidence for each vertex using a second output layer of the lane marker detection model; and predicting a presence confidence for each lane marker using a third output layer of the lane marker detection model. Processing system.

9. 9. The processing system of claim 8, wherein the one or more processors are further configured to cause the processing system to process the input image through a plurality of shared lane marker reduction layers of the lane marker detection model to process the input image with the shared lane marker portion of the lane marker detection model.

10. outputting the plurality of lane markers, [0025] According to each lane marker i Further comprising predicting vl ij However, each lane marker i The set of vertices associated with x ij ,y ij )}, and T vc is the existence confidence threshold for each vertex, and T lc 9. The processing system of claim 8, wherein x is a per lane presence confidence threshold.

11. 10. The processing system of claim 8, wherein the one or more processors are further configured to cause the processing system to compress the input image using an encoder-decoder segmentation network.

12. 10. The processing system of claim 8, wherein the final lane marker-specific reduced layer for each respective lane marker-specific representation has a size of h vertical pixels, 1 horizontal pixel, and c channels.

13. The one or more processors may be configured to configure the processing system: using one or more horizontal shrinking modules to compress input data within the shared lane marker portion of the lane marker detection model; causing one or more additional horizontal downscaling modules to be used to compress input data within the plurality of lane marker specific downscaling layers of the lane marker detection model. The processing system of claim 10 further comprising:

14. 10. The processing system of claim 8, wherein the one or more processors are further configured to cause the processing system to display a plurality of lane delineations over an output image.

15. A non-transitory computer readable medium comprising instructions that, when executed by one or more processors of a processing system, cause the processing system to perform the method of any one of claims 1 to 7.

Images