SYSTEM AND METHOD FOR END-TO-END PREDICTION OF LANE DETECTION UNCERTAINTY

Abstract

An end-to-end lane detection uncertainty prediction system that includes the following: a sensor device for a carrier vehicle that can be operated on a road surface within a driving environment including lane markings, wherein the sensor device is operable to generate data relating to the road surface within the driving environment; a navigation control system that includes a computer-based processor capable of being operated for: Monitoring the data, including an input image from the sensor device; Using a convolutional neural network to analyze the input image and output a lane prediction and a lane uncertainty prediction; and Generating a commanded navigation display based on lane prediction and lane uncertainty prediction; and a vehicle control system that controls the carrier vehicle based on the commanded navigation display; the neural convolutional network is initially configured using an association and The training focuses on error calculation per point, with the association and error calculation per point including the following: First, mapping a selected basic truth track to a selected set of data points that are related to a predicted track, and Secondly, mapping at least one point of the selected basic truth track to a corresponding data point from the selected set of data points relating to the predicted track.

Description

INTRODUCTION

[0001] The disclosure generally relates to a system and a method for end-to-end prediction of lane detection uncertainty.

[0002] Navigation systems and methods for autonomous and semi-autonomous vehicles use computer algorithms to determine a navigation path for the vehicle being controlled. Digital maps and sensor inputs are useful for establishing the vehicle's navigation path. Sensor inputs can include image recognition of lane markings and road features. Sensor inputs can also include image, radar, light detection, and rangefinder (LiDAR) or other similar sensor detection methods to monitor the positions of other vehicles relative to the controlled vehicle, for example, to prevent the controlled vehicle from getting too close to another vehicle in traffic.

[0003] The processing of sensor measurements for lane geometry detection includes image recognition programming. Image recognition programming analyzes shapes and pixel patterns in images to predict the features depicted. Image recognition inherently includes errors in its predictions.

[0004] DE 11 2019 000 048 T5 describes a method in which sensor data from a vehicle is acquired and applied to a machine learning model to identify boundary points that separate drivable from non-drivable areas. Based on these boundary points and associated class designations, locations in the surroundings are determined, enabling the vehicle to be controlled within the drivable area.

[0005] DE 10 2019 113 114 A1 describes a method in which a machine learning model, for example a deep neural network (DNN), is trained to use image and / or sensor data as input to generate two-dimensional or three-dimensional trajectory points, vehicle orientations, and / or vehicle states. Sensor data such as orientation, steering information, or vehicle speed can be collected to automatically create trajectories as ground-truth data for training the DNN. Subsequently, the generated trajectory points, vehicle orientations, and states can be used by a control component to steer the vehicle through a physical environment by determining a vehicle-specific control profile for steering, deceleration, and acceleration. DESCRIPTION

[0006] The object of the invention is to improve image recognition. This object is achieved by the subject matter according to claim 1. Further developments are described in the dependent claims.

[0007] An end-to-end lane detection uncertainty prediction system is provided. The system comprises a sensor device for a carrier vehicle that can operate on a road surface within a driving environment with lane markings. The sensor device can be operated to generate data relating to the road surface within the driving environment. The system further comprises a navigation controller containing a computer processor that can be operated to monitor the data, including an input image from the sensor device, use a convolutional neural network to analyze the input image and output a lane prediction and lane uncertainty prediction, and generate a commanded navigation display based on the lane prediction and lane uncertainty prediction.The system also includes a vehicle control unit that steers the carrier vehicle based on the commanded navigation display. The convolutional neural network is initially trained using per-point association and error calculations. These per-point calculations comprise, firstly, mapping a selected baseline truth track to a selected set of data points relating to a predicted track, and secondly, mapping at least one point of the selected baseline truth track to a corresponding data point from the selected set of data points relating to the predicted track.

[0008] In some embodiments, the convolutional neural network includes a trained deep tile prediction network that can perform image recognition on the input image, and a head error prediction network that can be trained using per-point association and error calculation.

[0009] In some embodiments, the use of the convolutional neural network involves the use of a trained lane and uncertainty network, wherein the trained lane and uncertainty network includes a combination of the trained deep tile prediction network and the head error prediction network.

[0010] In some embodiments, the per-point mapping and error calculation further includes determining an error with respect to the at least one point of the selected basic truth track and the corresponding data point from the selected set of data points with respect to the predicted track.

[0011] In some embodiments, the convolutional neural network uses a tile matrix to analyze the input image. The per-point mapping and error calculation further include identifying a corresponding tile of the tile matrix for each point of the selected set of data points relating to the predicted lane, and identifying a tile center for each of the identified tiles. Determining the error with respect to the at least one point of the selected baseline lane and the corresponding data point from the selected set of data points relating to the predicted lane involves determining a difference between the at least one point of the selected baseline lane and a selected tile center.

[0012] In some embodiments, determining the difference between the at least one point of the selected basic truth track and the selected tile center point involves identifying the selected tile center point from the tile centers based on the distances of each tile center point to the at least one point of the selected basic truth track.

[0013] In some embodiments, the identification of the selected tile center from the tile centers continues to be based on incremental progression along the selected basic truth line and the predicted trace.

[0014] In some embodiments, the computer-aided processor is also able to selectively locate the carrier vehicle with respect to a digital map based on the prediction of lane uncertainty.

[0015] In some embodiments, the sensor device includes a camera device.

[0016] In some embodiments, the sensor device includes a LiDAR device.

[0017] According to an alternative embodiment, a system for end-to-end prediction of lane detection uncertainty is provided within a carrier vehicle. The system comprises the carrier vehicle. The carrier vehicle includes a sensor device that can operate on a road surface within a driving environment containing lane markings, wherein the sensor device is operational to generate data relating to the road surface within the driving environment, and wherein the sensor device comprises either a camera device or a LiDAR device.The carrier vehicle also includes a navigation controller with a computer-based processor that can be operated to monitor data, including an input image from the sensor device, use a convolutional neural network to analyze the input image, and output a lane prediction and a lane uncertainty prediction. The convolutional neural network includes a trained deep tile prediction network that performs image recognition on the input image, and a head error prediction network that was initially trained using point-wise mapping and error calculation.The association and error calculation per point comprises, firstly, mapping a selected basic truth track to a selected set of data points relating to a predicted track, and secondly, mapping at least one point of the selected basic truth track to a corresponding data point from the selected set of data points relating to the predicted track. The computer processor is further capable of generating a commanded navigation representation based on the lane prediction and the lane uncertainty prediction. The carrier vehicle also includes a vehicle control system that steers the carrier vehicle based on the commanded navigation representation.

[0018] In some embodiments, the use of the convolutional neural network involves the use of a trained lane and uncertainty network, wherein the trained lane and uncertainty network includes a combination of the trained deep tile prediction network and the head error prediction network.

[0019] In some embodiments, the per-point mapping and error calculation further includes determining an error with respect to the at least one point of the selected basic truth track and the corresponding data point from the selected set of data points with respect to the predicted track.

[0020] In some embodiments, the convolutional neural network uses a tile matrix to analyze the input image. The per-point mapping and error calculation further include identifying a corresponding tile of the tile matrix for each point of the selected set of data points relating to the predicted lane, and identifying a tile center for each of the identified tiles. Determining the error with respect to the at least one point of the selected baseline lane and the corresponding data point from the selected set of data points relating to the predicted lane involves determining a difference between the at least one point of the selected baseline lane and a selected tile center.

[0021] According to an alternative embodiment, a method for end-to-end prediction of lane detection uncertainty is provided. The process comprises, in a computerized processor, first training a convolutional neural network using per-point mapping and error calculation. The per-point mapping and error calculation includes, firstly, mapping a selected basic truth lane to a selected set of data points with respect to a predicted lane, and secondly, mapping at least one point of the selected basic truth lane to a corresponding data point from the selected set of data points with respect to the predicted lane. The method further comprises the subsequent use of the convolutional neural network within the computerized processor for operational use in deriving lane geometries.This use includes monitoring data, including an input image from a sensor device on a carrier vehicle, which generates data relating to a road surface in front of the carrier vehicle, including lane markings; using the convolutional neural network to analyze the input image and output a lane prediction and a lane uncertainty prediction; and generating a commanded navigation display based on the lane prediction and the lane uncertainty prediction. The method further includes controlling the carrier vehicle based on the commanded navigation display.

[0022] In some embodiments, the use of the neural convolutional network for operational use in deriving lane geometries further includes the selective localization of the carrier vehicle with respect to a digital map based on the prediction of lane uncertainty.

[0023] In some embodiments, the use of the per-point mapping and error calculation further includes determining an error with respect to the at least one point of the selected basic truth track and the corresponding data point from the selected set of data points with respect to the predicted track.

[0024] In some embodiments, the use of the convolutional neural network to analyze the input image involves the use of a tile matrix for analyzing the input image. In some embodiments, the use of per-point mapping and error calculation further involves identifying a corresponding tile of the tile matrix for each point of the selected set of data points relating to the predicted track, and identifying a tile center for each of the identified tiles. In some embodiments, determining the error with respect to the at least one point of the selected basic truth track and the corresponding data point from the selected set of data points relating to the predicted track involves determining a difference between the at least one point of the selected basic truth track and a selected tile center.

[0025] In some embodiments, determining the difference between the at least one point of the selected basic truth track and the selected tile center point involves identifying the selected tile center point from the tile centers based on the distances of each tile center point to the at least one point of the selected basic truth track.

[0026] In some embodiments, the identification of the selected tile center from the tile centers continues to be based on incremental progression along the selected basic truth line and the predicted lane.

[0027] The above features and advantages and other features and advantages of the present disclosure are readily apparent from the following detailed description of the preferred embodiments for carrying out the disclosure when taken in conjunction with the accompanying figures. BRIEF DESCRIPTION OF THE FIGURES Fig. Figure 1 schematically shows an exemplary CNN training information flow according to the present disclosure; Fig. Figure 2 schematically shows an exemplary CNN inference information flow according to the present disclosure; Fig. Figure 3A graphically illustrates an analytical tile matrix that can be used to analyze an input image corresponding to a road surface in accordance with the present disclosure; Fig. 3B graphically shows one of the occupied tiles from Fig. 3A in greater detail, according to the present disclosure; Fig. Figure 4 graphically illustrates the use of the data points corresponding to the lane point predictions to predict an entire predicted lane curve by point clustering in accordance with the present disclosure; Fig. Figure 5 graphically illustrates the use of a basic truth trace to determine errors in the data points according to the present disclosure; Fig. Figure 6 graphically shows a first basic truth trace marker and a second basic truth trace marker in comparison to a first total predicted trace curve and a second total predicted trace curve, in accordance with the present disclosure; Fig. Figure 7 shows an exemplary vehicle controlled by the disclosed method and system, including devices and modules useful for collecting input images and processing the input images in accordance with the present disclosure; Fig. Figure 8 schematically shows an exemplary computerized navigation control system according to the present disclosure; and Fig. Figure 9 schematically shows an exemplary data communication system within a vehicle to be controlled according to the present disclosure. DETAILED DESCRIPTION

[0028] A method and system for end-to-end prediction of lane detection uncertainty are provided. The method can utilize a convolutional neural network (CNN) to process an input image and generate an output that analyzes the image. Specifically, images relating to lanes on a road surface in front of, behind, or near a vehicle can be analyzed to assist or guide the vehicle's navigation within a lane or between lanes. The method employs an efficient calculation of the point lateral error of the lane when a basic truth lane is present.

[0029] CNNs are trained with weights and biases that can be used to improve output accuracy. One aspect of training a CNN is uncertainty prediction. The method described here is a learning-based approach that predicts the positional uncertainty of lane points "on the fly," without relying on hand-built offline error models. Quantifying uncertainty prediction in a lane detection process is useful because it provides a measure of the quality of the lane detection output. If the uncertainty prediction indicates that the uncertainty is low, or that the lane detection output is of high quality, the lane detection output can be used for carrier vehicle navigation.If the uncertainty prediction determines that the uncertainty is high or that the lane detection output is of low quality, a corrective action can be initiated. Such a corrective action might involve, for example, displaying a warning to the driver or implementing an alternative or additional process to improve the lane detection uncertainty, such as processes for estimating lanes based on the movement of other vehicles in traffic. In another example, uncertainty can be used for the selective localization of the vehicle relative to a digital map. At low uncertainty, features of the detected lane can be used to accurately locate or identify the vehicle's current position on the digital map. At high uncertainty, other localizers, such as...Traffic signs are used to achieve or increase the accuracy of localization.

[0030] During the uncertainty prediction training, the process can perform point clustering for given track point predictions to form an overall track. Additionally, the process can then use an association algorithm to match each track to a baseline truth track. Furthermore, the process can subsequently compute point mappings to obtain accurate measurements of the lateral distance between each point and the baseline truth track. The process can use these lateral errors to monitor the uncertainty prediction.

[0031] The process maps predicted lane point clusters to ground truth lanes, taking into account aggregated information from a multitude of points along the curve. The process assigns each predicted point to the curve to which it can be assigned. Some methods use the Euclidean distance between the predicted point and a ground truth lane point, which can lead to errors in the measured error due to lane mismatch. The disclosed method can compute the lateral distance between predicted lane points and ground truth lane points, which are not necessarily the nearest ground truth lane points. In this way, the method enables the prediction of lane point uncertainty considering the entire lane geometry and can account for large errors of the predicted points relative to the ground truth lane.The method can use vectorized calculations for faster uncertainty training.

[0032] The process achieves an interpretable uncertainty prediction per point. In one embodiment, given predicted lanes, the process involves performing a whole-lane mapping to match each lane with a corresponding ground truth lane. Subsequently, a per-point mapping is performed based on accurate and fast calculations. This allows for the estimation of the lateral and directional errors between each predicted point and its associated ground truth lane. This, in turn, is used to monitor the uncertainty prediction for each lane point.

[0033] Fig. Figure 1 schematically shows an exemplary CNN training information flow 10. The CNN training information flow 10 includes a sensor 12 that can capture images of a road surface. The CNN training information flow 10 illustrates the operation of a procedure in which a CNN can be trained with predetermined basic truth lanes and input images corresponding to the basic truth lanes. Information from the sensor 12 is passed to a trained tile prediction deep network 14. The trained tile prediction deep network 14 includes a neural network capable of receiving input images and applying analytical algorithms as well as trained weights and biases to perform image recognition on the input images. The trained tile prediction deep network 14 provides an output of lane tile predictions 16, or predicted lane geometries, based on the input images.Data relating to the predetermined basic truth lanes 18 are provided, and a per-point association and error calculation 20 compares the lane tile predictions 16 and the predetermined basic truth lanes 18, providing error values ​​as described here that associate points of the lane tile predictions 16 with the predetermined basic truth lanes 18. The error values ​​from the per-point association and error calculation 20 are provided to a header network for error prediction 22. The input is provided to the error prediction header network 22 as predictions for the lane tiles 16, and the monitoring is provided to the error prediction header network 22 as the error values ​​determined by the per-point association and error calculation 20. Through the input and monitoring, the header network for error prediction 22 is trained, and any loss relative to the actual lane prediction error 24 is optimized.

[0034] Throughout the disclosure, the input images may be images captured by a camera device, data generated by a LiDAR device, or data captured by another similar sensor device. Fig. Figure 2 schematically shows an exemplary CNN inference information flow 30. Once the head network for error prediction 22 is trained, it can be combined with the trained deep network with tile prediction 14 to form a trained lane and ambiguity network 34 for operational use in inferring lane geometries. In one embodiment, the head network for error prediction 22 is a small network that operates on top of the trained deep network with tile prediction 14. The head network for error prediction 22 is trained after the trained deep tile prediction network 14 because the uncertainty is based on the detected lanes. The CNN inference information flow 30 includes a sensor 32 that can acquire images of a road surface. Information from the sensor 32 is fed to the trained lane and ambiguity network 34. Based on the training, which is carried out in the Fig. When the process shown in 1 is carried out, the trained lane and blur network 34 can output lane predictions 36 and lane blur predictions 38 based on the input images supplied by the sensor 32.

[0035] Fig. Figure 3A graphically illustrates an analytical tile matrix 50 that can be used to analyze an input image representing a road surface. The analytical tile matrix 50 can be applied to, or used in conjunction with, an input image of a road surface containing lane markings or lane information. The resolution of the tiles within the analytical tile matrix 50 can vary, and the resolution shown in Fig. The six by seven matrix shown in Figure 3A is a non-restrictive example. Several data points 52 correspond to lane point predictions made by the trained lane and uncertainty network 34. Fig. 2 can be derived from the input image by a programmed lane tile prediction model. A large number of occupied tiles 56 can be identified, within which at least one of the data points 52 is located. A tile center 55 of each occupied tile 56 can be identified. Fig. 3B graphically shows one of the occupied tiles 56 from Fig. 3A in greater detail. The occupied tile 56 contains data point 52 and a tile center 55. Line 54 connects a data point 52 with a corresponding tile center 55.

[0036] Fig. Figure 4 graphically illustrates the use of data points 52, corresponding to lane point predictions, to predict an entire predicted lane curve 58 by point clustering. The entire predicted lane curve 58 can be fitted or approximated to the data points 52.

[0037] A ground truth track used for training purposes contains data representing actual lane geometry on a road surface, which is the source of the data shown in data points 52. Fig. Figure 5 graphically illustrates the use of a basic truth track 60 for determining errors in the data points 52. According to an exemplary embodiment, a method is used to employ one or more basic truth tracks 60 to determine the error in assigning a selected basic truth track 60 to a selected set of data points 52 that represent a predicted track, and secondly, to assign points on the selected basic truth track 60 to the selected set of data points 52. The assignment of points in this way can be described as an assignment of the predicted track to the basic truth track 60 for each point.

[0038] In one embodiment, points on the selected basic truth track 60 can be assigned to one of the tile centers 55 corresponding to the selected set of data points 52. An algorithm for assigning points on the selected basic truth track 60 can include a preference for assigning points on the selected basic truth track 60 to one of the nearest tile centers 55, while conserving the length of both the selected basic truth track 60 and the predicted track represented by the data points 52. Conserving length can be described as assuming that progression along the basic truth track 60 and the predicted track represented by the data points 52 can be assumed to occur in approximately equal steps per data point.Once the points on the selected basic truth track 60 and the tile centers 55 have been assigned, a distance 62 between the points on the selected basic truth track 60 and the tile centers 55 can be determined and used to determine the error in the predicted track. This error can be used to train the head network 22 for error prediction, as described in relation to . Fig. 1 described.

[0039] The mapping of predicted lane data points to a baseline lane can be complicated by the presence of multiple sets of predicted lane data points and / or multiple baseline lanes in a single area or input image. By first mapping a selected baseline lane 60 to a selected set of data points 52 representing a predicted lane, and then mapping points on the selected baseline lane 60 to the selected set of data points 52, confusion and inaccurate error detection can be avoided by ensuring that the correct set of data points 52 is mapped to and compared with the correct baseline lane 60.

[0040] Fig. Figure 6 graphically illustrates a first basic truth lane marker 60A and a second basic truth lane marker 60B compared to a first total predicted lane curve 58A and a second total predicted lane curve 58B. The first total predicted lane curve 58A and the second total predicted lane curve 58B can each contain and be fitted to a variety of data points generated by analyzing an input image. The first basic truth lane marker 60A and the second basic truth lane marker 60B are each data sets associated with a predetermined lane marker geometry corresponding to the lanes depicted in the input image used to create the first total predicted lane curve 58A and the second total predicted lane curve 58B.The first total predicted lane curve 58A contains predicted data points generated based on a portion of the input image that represents or corresponds to the first basic truth lane marker 60A. The second total predicted lane curve 58B contains predicted data points generated based on a portion of the input image that represents or corresponds to the second basic truth lane marker 60B. The first total predicted lane curve 58A, the second total predicted lane curve 58B, the first basic truth lane marker 60A, and the second basic truth lane marker 60B are located in a common space.Near an upper vertical section of the graph, the error between the graphs creates a condition where the second total predicted lane curve 58B is closer to the first ground truth lane marker 60A than the first total predicted lane curve 58A is to the first ground truth lane marker 60A. If the mapping of a point in this upper section to the first ground truth lane marker 60A were performed solely based on which predicted data point is closest to the point on the first ground truth lane marker 60A, an inaccurate mapping between the second total predicted lane curve 58B and the first ground truth lane marker 60A would occur. By first mapping a selected ground truth lane to a total predicted lane curve, an inaccurate mapping of the ground truth lane to an incorrect total predicted lane curve can be avoided.

[0041] Fig. Figure 7 shows an exemplary vehicle controlled by the disclosed method and system, including devices and modules useful for acquiring input images and processing the input images as described herein. A carrier vehicle 300 is shown on a road surface 310 with lane markings 320. The carrier vehicle 300 is shown with a navigation controller 220, a vehicle controller 230, a camera device 110, and a sensor device 210. The camera device 110 comprises a field of view 112 and is positioned to capture images of the road surface 310 and other objects and obstacles near the carrier vehicle 300, including input images representing the road surface 310. The sensor device 210 can additionally provide data about objects near the carrier vehicle 300.The navigation control unit 220 receives data from the camera device 110 and other sources and can generate a commanded navigation display based on the analysis of the available data. The vehicle control unit 230 uses the commanded navigation display to control the navigation of the carrier vehicle 300 on the road surface 310. The control of the carrier vehicle 300 can include controlling the steering, the output torque of the powertrain, and / or the brakes; it can include autonomous or semi-autonomous navigation of the vehicle according to a planned navigation route via a digital map or through a detected environment; and / or it can include providing warnings and instructions to the driver regarding navigation, lane keeping, object avoidance, and other relevant driving issues.

[0042] The navigation controller 220 can perform image recognition on input images generated by the camera device 110. The navigation controller 220 can include a CNN trained according to the disclosed method, such that the CNN provides the navigation controller with lane predictions and lane uncertainty predictions based on the input images supplied by the camera device 110. The lane uncertainty predictions can be used by algorithms programmed in the navigation controller 220 to assign a weight or quality rating to the lane predictions, so that the predicted uncertainty and the probable error in the lane predictions can be taken into account in the commanded navigation display generated by the navigation controller 220.

[0043] The carrier vehicle 300 is an example of a vehicle that uses the described method and system. Other embodiments are conceivable, and the disclosure is not limited to the examples listed here.

[0044] Within the disclosed system, various controllers can be used to operate the disclosed process. Controllers can comprise a computerized device containing a computerized processor with memory capable of storing programmed executable code. A controller can operate on a single computerized device or extend across multiple computerized devices. Fig. Figure 8 schematically shows an exemplary computerized navigation controller. The navigation controller 220 comprises a computer-based processor device 410, a communication module 430, a data input / output module 420, and a storage device 440. It should be noted that the navigation controller 220 may contain additional components, and some of the components are not present in some embodiments.

[0045] The processor unit 410 can include memory, such as read-only memory (ROM) and random-access memory (RAM), in which processor-executable instructions are stored, as well as one or more processors that execute the processor-executable instructions. In embodiments in which the processor device 410 includes two or more processors, the processors can operate in parallel or in a distributed manner. The processor device 410 can run the operating system of the navigation controller 220. The processor device 410 can include one or more modules that execute programmed code or computer-aided processes or methods with executable steps. The modules shown can comprise a single physical device or functionality that extends over several physical devices.In the illustrated embodiment, the processor device 410 also includes a trained lane and uncertainty network 412, a commanded navigation plot generator 414 and a high uncertainty response module 416, which are described in more detail below.

[0046] The 420 data input / output module is a device that can receive data from sensors and devices throughout the vehicle and process it into formats that can be used by the 410 processor unit. The 420 data input / output module can also process the output from the 410 processor unit and enable this output to be used by other devices or controllers in the vehicle.

[0047] The Communication Module 430 can include a communication / data link with a bus device configured to transmit data to various components of the system, and can include one or more wireless transceivers for wireless communication.

[0048] The storage device 440 is a device that stores data generated or received by the navigation controller 220. The storage device 440 may include, but is not limited to, a hard disk drive, an optical drive, and / or a flash memory drive.

[0049] The trained tracking and uncertainty network 412 can be a CNN programmed to perform the process described herein, including training, weights, and distortions, to enable the trained tracking and uncertainty network 412 to analyze input images and generate a tracking prediction and a tracking uncertainty prediction.

[0050] The commanded navigation display generator 414 can be programmed to use available data, including lane prediction and lane uncertainty prediction, generated by the trained lane and uncertainty network 412, to produce a commanded navigation display to assist or control the navigation of the carrier vehicle. In some embodiments, such as a semi-autonomous vehicle or a carrier vehicle equipped with warning systems, the commanded navigation display generator 414 can alternatively be described as a recommended navigation display generator.

[0051] The High Uncertainty Response Module 416 includes programming to respond to a lane uncertainty prediction by the trained lane and uncertainty network 412, indicating that the lane prediction generated by the trained lane and uncertainty network 412 may be temporarily of low quality or unreliable. The High Uncertainty Response Module 416 may include programming a visual or audible warning, programming autonomous braking, programming an increase in the minimum distance between the carrier vehicle and other nearby vehicles, and / or other similar reactive measures.

[0052] The navigation controller 220 is an exemplary computer-controlled device capable of executing programmed code to evaluate input images and selectively use them to generate and utilize lane prediction and lane uncertainty prediction. Several different embodiments of the navigation controller 220, the devices connected to it, and the modules operable therein are conceivable, and the disclosure is not intended to be limited to the examples given here.

[0053] Fig.Figure 9 schematically shows an exemplary data communication system 200 within a carrier vehicle. The data communication system 200 is depicted with a camera device 110, a digital map database 120, a sensor device 210, a navigation controller 220, and a vehicle control unit 230, each of which is communicatively connected to the vehicle data bus 240. The sensor device 210 can include one or more radar devices, LiDAR devices, ultrasonic devices, or other similar devices useful for collecting data about a vehicle's surroundings and the behavior of other vehicles in a lane. The vehicle data bus 240 comprises a communication network capable of rapidly transferring data back and forth between various connected devices and modules.Data can be collected from any of the camera devices 110, the digital map database 120, and the sensor device 210 and transmitted to the navigation control unit 220. The navigation control unit 220 comprises a computerized processor and programmed code used to create a commanded navigation display, which serves to navigate the controlled vehicle over a road surface in the vicinity of the vehicle.

[0054] The navigation diagrams described here can be useful for controlling the navigation of a fully autonomous vehicle. Likewise, the navigation plots described here can be useful for controlling the navigation of a semi-autonomous vehicle, for example, to enable automatic braking, lane keeping, or obstacle avoidance. Similarly, the navigation plots described here can be useful for providing navigation aids such as projected graphics or generated sounds to assist a driver in efficiently controlling a vehicle. Examples of how generated navigation plots can be used are given here. Other embodiments are conceivable, and the disclosure is not limited to the examples listed here.

[0055] The disclosed method and system describe an improvement in feature availability for autonomous and semi-autonomous vehicles. Under conditions where some navigation processes lack sufficient data and guidance to effectively navigate the vehicle, such as in a construction zone with missing, conflicting, or displaced lane markings, the disclosed method and system can be used to validate images of an area in front of the vehicle and successfully utilize them to navigate the vehicle through the exemplary construction zone.

[0056] While the preferred embodiments for carrying out the disclosure have been described in detail, those who are familiar with the prior art to which this disclosure relates will recognize various alternative designs and embodiments for carrying out the disclosure within the scope of the attached claims.

Claims

[1] An end-to-end lane detection uncertainty prediction system comprising: a sensor device for a carrier vehicle that can be operated on a road surface within a driving environment including lane markings, wherein the sensor device is operable to generate data relating to the road surface within the driving environment; a navigation control system that includes a computer-based processor capable of being operated for: Monitoring the data, including an input image from the sensor device; Using a convolutional neural network to analyze the input image and output a lane prediction and a lane uncertainty prediction; and Generating a commanded navigation display based on lane prediction and lane uncertainty prediction; and a vehicle control system that controls the carrier vehicle based on the commanded navigation display; the neural convolutional network is initially configured using an association and The training focuses on error calculation per point, with the association and error calculation per point including the following: First, mapping a selected basic truth track to a selected set of data points that are related to a predicted track, and Secondly, mapping at least one point of the selected basic truth track to a corresponding data point from the selected set of data points relating to the predicted track. [2] The system according to claim 1, wherein the neural convolution network comprises: a trained deep network with tile prediction that is operational to perform image recognition on the input image; and A head network for error prediction that can be trained using association and error calculation per point. [3] The system according to claim 2, wherein the use of the neural convolution network includes the use of a trained lane and uncertainty network, wherein the trained lane and uncertainty network includes a combination of the trained deep tile prediction network and the head error prediction network. [4] The system according to claim 1, wherein the association and error calculation per point further comprises determining an error with respect to the at least one point of the selected basic truth track and the corresponding data point from the selected set of data points with respect to the predicted track. [5] The system according to claim 4, wherein the neural convolutional network uses a tile matrix to analyze the input image; wherein the association and error calculation per point further includes: Identifying a corresponding tile in the tile matrix for each point of the selected set of data points with respect to the predicted track; and identifying a tile center for each of the identified tiles; and wherein determining the error with respect to the at least one point of the selected basic truth track and the corresponding data point from the selected set of data points with respect to the predicted track includes determining a difference between the at least one point of the selected basic truth track and a selected tile center. [6] The system according to claim 5, wherein determining the difference between the at least one point of the selected ground truth trace and the selected tile centers comprises identifying the selected tile center from the tile centers based on the distances of each of the tile centers to the at least one point of the selected ground truth trace. [7] The system according to claim 6, wherein the identification of the selected tile centers from the tile centers is further based on incremental progression along the selected basic truth track and the predicted track. [8] The system according to claim 1, wherein the computer-based processor is further operable in such a way that it selectively locates the carrier vehicle with respect to a digital map based on the prediction of lane uncertainty. [9] The system according to claim 1, wherein the sensor device comprises a camera device. [10] A method for end-to-end prediction of lane detection uncertainty, comprising: within a computer-based processor: Initial training of a neural convolutional network using association and error calculation per point, where the association and error calculation per point includes: First, mapping a selected basic truth track to a selected set of data points that are related to a predicted track; and Secondly, mapping at least one point of the selected basic truth track to a corresponding data point from the selected set of data points relating to the predicted track; Step-by-step use of the neural convolutional network for operational application in the inference of lane geometries, including: Monitoring data that includes an input image from a sensor device of a carrier vehicle, which generates the data relating to a road surface in front of the carrier vehicle including lane markings; Using the neural convolutional network to analyze the input image and output a lane prediction and a lane uncertainty prediction; and Generating a commanded navigation display based on lane prediction and lane uncertainty prediction; and Controlling the carrier vehicle based on the commanded navigation display.

Images