Extreme case detection and collection for path planning systems

Abstract

The present invention relates to a system and method for detecting and storing information describing errors in a predictive path planning function of a neural network.

Description

TECHNICAL FIELD

[0001] The present invention relates to a method operation and apparatus for detecting errors in a prediction algorithm. SUMMARY

[0002] Disclosed embodiments provide systems and methods for detecting and storing information describing errors in the path planning functionality of a prediction of a neural network.

[0003] According to at least some disclosed embodiments, systems, components, and methods, the functionality of a prediction of a prediction algorithm, such as a neural network, is tested, and errors are identified.

[0004] According to at least some disclosed embodiments, information detailing the circumstances surrounding an error can be collected, buffered, and logged in semi-permanent memory.

[0005] Additional features of the invention will become apparent to those skilled in the art upon consideration of the following detailed description of illustrative embodiments thereof. BRIEF DESCRIPTION OF DRAWINGS

[0006] The detailed description particularly refers to the accompanying figures in which:

[0007] Figure 1 A sensor, a path prediction neural network, and a predicted path for an autonomous driving transportation vehicle are shown.

[0008] Figure 2 A comparison of a predicted path and a reference path driven by a human over a series of time periods is shown.

[0009] Figure 3 An example of a system and constituent components for performing image data acquisition and error awareness and storage in memory according to the invention is shown.

[0010] Figure 4 Computing resources for performing error detection of a path prediction neural network and storage of error circumstance information according to the invention are shown. DETAILED DESCRIPTION

[0011] The drawings and description provided herein can have been simplified to illustrate aspects that are pertinent to the present devices, systems, and methods, while eliminating, for the purpose of clarity, other aspects that can be found in typical devices, systems, and methods. Those skilled in the art will recognize that other elements and / or operations can be desirable and / or necessary in implementing the present devices, systems, and methods. Because such elements and operations are well known in the art, and because they do not facilitate a better understanding of the present application, a discussion of such elements and operations is not provided herein. However, the present application is deemed to inherently include all such elements, variations, and modifications to the described aspects that would be known to those of ordinary skill in the art.

[0012] Identifying errors that arise during the operation of neural networks and / or other predictive algorithms can be difficult. Moreover, learning about such errors and the conditions that lead to and result from error instances can play a valuable role in the iterative development of neural networks.

[0013] Predictive algorithms, often used in connection with autonomous transportation vehicles, can be trained by providing a predicted path for a transportation vehicle while the transportation vehicle is operated by a human driver. According to at least some disclosed embodiments, systems, components, and methods, the predictive function of a predictive algorithm, such as a neural network, is tested, and errors are identified. Moreover, information detailing the circumstances surrounding the errors is collected, buffered, and recorded in semi-permanent memory.

[0014] Disclosed embodiments provide technical solutions for detecting errors and improving predictive algorithms, often used in connection with autonomous transportation vehicles. Such predictive algorithms, e.g., neural networks, can be trained by providing a predicted path for a transportation vehicle while the transportation vehicle is operated by a human driver, followed by a comparison between the predicted path and the path taken by the driver shortly thereafter.

[0015] For the purposes of the present application, the term "roadway" includes any pathway, thoroughfare, route, or street between two places that has been paved or otherwise improved to allow the travel of a transportation vehicle, including but not limited to a motor vehicle or other transportation vehicle that includes one or more wheels. Thus, it should be understood that such a roadway can include one or more lanes and intersections with other roadways, including on-ramps / off-ramps, merging areas, etc., which can be included in a parkway, boulevard, avenue, free road, toll road, interstate, highway, or a primary, secondary, or tertiary local road.

[0016] For the purposes of the present application, the term "vehicle positioning" is used to refer to the ability to determine the position of a transportation vehicle relative to a roadway or a portion of a roadway, such as a lane on which the transportation vehicle is traveling.

[0017] It is contemplated that further integration of autonomous and driver-assistance related transportation vehicle technologies will, in implementations, result in vehicle positioning being performed in an automated or semi-automated manner at least in part and possibly entirely relying on Global Positioning Service (GPS) data, data produced by a plurality of vehicle-mounted sensors, and machine learning algorithms and / or neural networks operably coupled to the plurality of sensors and / or GPS to process and interpret such data to facilitate vehicle positioning. To develop and train neural networks for performing vehicle positioning using various different types of sensors and various different types of feedback, different conventional approaches are known.

[0018] For purposes of the present invention, the phrase "autonomous and / or assistance functionality" refers to functionality capable of achieving partial, complete, or full automation of vehicle control, and is ordered and includes the five levels of driving automation currently known. Thus, it should be understood that autonomous and / or assistance functionality refers to operations performed by a vehicle in an automated manner through on-board equipment or outputting alerts, prompts, recommendations, and / or instructions to a user, where such outputs are produced by the on-board equipment in an automated manner. Further, autonomous and / or assistance functionality can include driver assistance functionality (Level One), where the on-board equipment assists but does not control steering, braking, and / or acceleration, but the driver ultimately controls acceleration, braking, and monitors the vehicle's surroundings.

[0019] Thus, it should be understood that such autonomous and / or assistance functionality can also include a lane departure warning system that provides a mechanism to warn the driver when the transportation vehicle begins to leave its lane on a free way and arterial road (unless a turn signal is in that direction). Such a system can include a system that warns the driver (lane departure warning) if the vehicle leaves the lane (visual, audible, and / or vibratory warning), and automatically takes action to ensure the vehicle remains within its lane (lane keeping system) if no action is taken.

[0020] Likewise, autonomous and / or assistance functionality can include partial automation (Level Two), where the transportation vehicle assists in steering or acceleration functions and correspondingly monitors the vehicle's surroundings to enable the driver to disengage from some of the tasks for driving the transportation vehicle. As is understood in the automotive industry, partial automation still requires the driver to be prepared to assume all tasks of operation of the transportation vehicle and continuously monitor the vehicle's surroundings at any time.

[0021] Autonomous and / or assistive functionality can include conditional automation (level three), in which the transport vehicle equipment is responsible for monitoring the vehicle's surroundings and controlling the vehicle's steering, braking, and acceleration without driver intervention. It will be appreciated that at this level and above, the on-board equipment used to perform autonomous and / or assistive functionality will cooperate with or include navigation functionality, such that the component has data for determining where the vehicle will travel. At level three and above, the driver is theoretically allowed to disengage from monitoring the vehicle's surroundings, but can be prompted to control transport vehicle operation under certain conditions, which can impede safe operation in a conditional automation mode.

[0022] Accordingly, it will be appreciated that autonomous and / or assistive functionality can include systems that take over steering and / or keep the transport vehicle in the relative middle of a traffic lane. Also, autonomous and / or assistive functionality can include high automation (level four) and full automation (level five), in which the on-board equipment is capable of implementing automatic steering, braking, and acceleration in an automated manner without driver intervention in response to monitoring the vehicle's surroundings.

[0023] Accordingly, it will be appreciated that autonomous and / or assistive functionality can entail monitoring the vehicle's surroundings, including the vehicle's roadway, and identifying objects in the surroundings, in order to be able to implement safe operation of the vehicle in response to traffic events and navigation directions, in which safe operation entails determining when to change lanes, when to change directions, when to change roads (exit / enter a road), when and in what order to merge or cross road intersections, and when to use turn signals and other navigation indicators to ensure that other vehicles / vehicle drivers are aware of the vehicle's impending maneuvers.

[0024] It will further be appreciated that high and full automation can include analyzing and considering data provided from sources outside the vehicle in order to determine whether such automation levels are safe. For example, autonomous and / or assistive functionality in such levels can involve determining the likelihood of pedestrians in the transport vehicle's surroundings, which can involve referencing data indicating whether the current road is a highway or a parkway. Further, autonomous and / or assistive functionality in such levels can involve accessing data indicating whether there is a traffic jam on the current road.

[0025] Conventional transportation vehicle navigation systems and conventional autonomous vehicles use GPS technology for their vehicle positioning. However, conventional use of GPS for global positioning has the drawback that positioning is limited to a certain level of accuracy, more specifically, 5-10 meters in the best case (which often requires a geographic area that provides an unobstructed open view to the sky). Moreover, in geographic areas that include relatively large buildings, trees, or geographic contours (such as canyons), the likelihood of lower accuracy is greater. This is because GPS-based positioning services require signals from GPS satellites. Dense materials (e.g., rock, steel, etc.), tall buildings, and large geographic features can block or degrade GPS signals.

[0026] Accordingly, GPS is often combined with local landmarks (e.g., lane markings) for vehicle positioning to improve the ability of vehicles with autonomous and / or assisted vehicle functionality to accurately perform vehicle positioning. Conventionally, these local landmarks are detected and identified from camera images or sensor data obtained from one or more cameras / sensors located on the vehicle. For example, it has been conventionally discussed to combine GPS data with data collected from front-facing cameras and LiDAR, and even data generated through ground-penetrating radar. It has also been discussed that such cameras can be used to extract a simplified feature representation of road characteristics from the on-board camera to generate data indicative of a roadside pattern that can be analyzed to perform vehicle positioning. Machine learning algorithms, such as neural networks, and models developed therefrom facilitate the combination and manipulation of different data inputs, including camera images, to provide autonomous and / or assisted functionality, and more specifically, to develop a predicted path along a road for a transportation vehicle.

[0027] The disclosed embodiments are based on the recognition that recent autonomous vehicle traffic accidents provide evidence that there is a technical and real-world need to increase the robustness and / or reliability of algorithms that govern autonomous and / or assisted functionality; in particular, to enable autonomous and / or assisted functionality and control the travel of a transportation vehicle in situations where the frequency of occurrence is low and / or requires more significant reactive measures by the system.

[0028] Autonomous and / or assisted functionality includes two broad functional systems or modules: a path planning system and a control and / or assistance system. The path planning system receives and processes sensor inputs and other information to determine steering, braking, acceleration, etc. of the transportation vehicle. The control and / or assistance system implements the path planned by the path planning system through fully or partially automated steering, braking, acceleration, and / or monitoring of the vehicle's surroundings.

[0029] During training and testing of autonomous and / or assistive functionality, it is helpful to identify extreme cases (i.e., cases where the predictions of the path planning system deviate from the actions taken by a human driver) and / or other error states or indicia. A common testing strategy for path planning systems is referred to as shadow mode and / or shadowing. When shadow mode is enabled, the autonomous and / or assistive functionality system runs in the background and performs its prediction functions, but does not command operation of the transportation vehicle in accordance with the prediction functions. In addition, during use of shadow mode, a human driver is fully responsible for operating the transportation vehicle. Thus, because the path planning system provides predictions for the same driving conditions experienced by the driver currently operating the transportation vehicle, a comparison can be made between the predictions and the actions of the human driver.

[0030] Such a comparison can then identify when the predictions of the path planning system deviate from the actions of the human driver beyond a predetermined threshold. When the predetermined threshold is exceeded, an error (i.e., an extreme case) is triggered and data associated with the error is stored in semi-permanent memory / storage 450 (see Figure 4 ). The semi-permanent memory can be one or more memory modules disposed between devices of the transportation vehicle (e.g., a hard drive), and / or the semi-permanent memory can include transmission of the error data to cloud storage (see Figure 4 ) via communication module 440 over a mobile or wireless connection. The error data is stored such that it can be used for troubleshooting of the prediction algorithms and path planning system thereafter. The data collection strategy of storing only extreme case / error data can be an improvement because it reduces the amount of data storage required as compared to a data collection strategy that stores all prediction data and all data describing human driver operations. This improvement is further emphasized if the autonomous and / or assistive functionality operates with a relatively low error rate such that errors occur infrequently or rarely exceed the error threshold, thus resulting in limited (and less) data storage and / or transmission.

[0031] In particular, the task of the path planning system in the form of neural network 110 (see Figure 1 ) is to predict the transportation vehicle 130 (see Figure 3) of 30 meters. The predicted path 120 is compared to a reference path 140, which corresponds to a human driver driving the transport vehicle 130 over the 30 meter path in the same scenario based on the images used in the prediction. The path planning system can be evaluated based on this comparison (i.e., the more closely the predicted path 120 and the reference path 140 align, the more accurate the path planning system can be considered for training purposes). It is contemplated in the present disclosure that three types of errors are learned for extreme cases of the path planning system / error detection and collection: deviation from the reference, temporal instability, and overall uncertainty.

[0032] For each predicted path 120 produced by the path planning system, a comparison is made with a corresponding reference path 140 produced by a human driver. The comparison operation can be performed in increments as depicted by images, time, distance, and / or other suitable units. For example, the predicted path 120 can be updated in increments of every meter of travel, or as each new road image is captured. However, the reference path 140 is not available at the same time as the predicted path 120. Rather, the reference path 140 is available only after the transport vehicle 130 travels along the reference path 140 as determined by the human driver and is subject to environmental and road conditions. First, the human driver performs the driving actions corresponding to the reference path 140, and then the transport vehicle 130 responds to the driving actions of the human driver and other conditions that affect the reference path 140 of the transport vehicle 130.

[0033] Neural network predictions such as employed by the path planning system can take tens of milliseconds on specialized hardware. Figure 1 The illustrated reference path 140 spans 30 meters in a gentle curve. In this example, the transport vehicle 130 crosses the reference path 140 in 5 seconds when the transport vehicle 130 is traveling at 15 miles per hour (i.e., relatively slow). Further, the transport vehicle 130 crosses the reference path 140 in 1 second when traveling at 70 miles per hour (i.e., relatively fast). As a result of the varying operation of the transport vehicle 130, the reference path 140 can become available between less than 1 second and more than 5 seconds after the predicted path 120 is produced. The delay can be even longer when the transport vehicle 130 is moving slower than the example discussed earlier. Memory management is important for storing information about a large number of predicted paths 120 and associated reference paths 140 observed after a significant delay.

[0034] In disclosed embodiments, the path planning system can include Figure 1 The illustrated deep neural network 110. The reference path 140 is stored in memory 420 and processor 430 (see Figure 4) running physical model generates, a memory and a processor are disposed on the transport vehicle 130 and are communicatively connected with one or more sensor devices 460 (e.g., camera 300) through a controller area network (CAN) 410. The physical model can receive one or more signals from the CAN bus 410 as input parameters (including but not limited to vehicle speed, acceleration, angular velocity, wheel tick counts, etc.). As described above. The reference path 140 can be delayed by a period of time (on the order of several seconds) compared to the availability of the predicted path 120. In the disclosed exemplary embodiment, after the predicted path 120 is generated, information describing the predicted path 120 is stored in the memory 420, and the processor 430 waits for the corresponding reference path 140 before generating the comparison. If a deviation greater than a predetermined threshold is detected, the relevant data is transferred to the semi-permanent memory 470 (see Figure 4 ).

[0035] To store image data, CAN bus data (e.g., sensor readings), and other continuously collected or streaming data, one or more ring buffers 450 can be implemented in the memory 420. In the exemplary embodiment, both the first and second ring buffers include three memory modules, where each memory module stores up to 5 seconds of collected data. The 5 seconds of collected data can include 5 seconds of images or 5 seconds of CAN bus signals. The first, second, and third memory modules are continuously filled. Then, when the third memory module is filled, the first memory module is emptied and new data is stored therein. This storage protocol continues to implement the ring buffer 450 on a continuous basis with one memory module after another. Thus, at a given moment, each of the one or more ring buffers 450 stores data describing the driving activity of the previous 10-15 seconds thereon. The physical model used to calculate the reference path 140 is likewise continuously running in shadow mode so that extreme cases and / or errors in the predicted path 120 can be detected.

[0036] When the predicted path 120 deviates from the reference path 140 beyond a threshold, an error is detected, as described previously, and a data collection function is triggered. After the collection function is triggered, the memory 420 waits for the current memory module and the subsequent memory module in the ring buffer 450 to be filled. Then, all of the data stored in the ring buffer 450 is transferred to the semi-permanent memory. Thus, data describing the error as well as data describing the previous 5 seconds and the subsequent 5 seconds are placed in the semi-permanent memory to inform the troubleshooting effort with the pre- and post-context of the subject error.

[0037] Extreme cases and error detection can involve not only deviations between a predicted path 120 and a corresponding reference path 140. Even if a series of predicted paths 120 generated by a path planning system for successive frames of a road image deviate from associated reference paths 140 by less than an error threshold, such predicted paths 120 can collectively exhibit temporal instability. For purposes of controlling and developing a path planning system, temporal instability can be recorded as an error. For example, referring now to Figure 2 At a first time increment 200, a predicted path 120a is slightly skewed to the right from a corresponding reference path 140a (when the transport vehicle 130 is moving from left to right through this illustrated example). Then, at a second time increment 210, a predicted path 120b is slightly skewed to the left from a corresponding reference path 140b. And again, at a next time increment 220, a predicted path 120c has a more pronounced left-right path movement than an associated reference path 140c. The reference paths 140a, b, c at successive time increments 200, 210, 220 are expected to extend smoothly from the preceding reference paths 140a, b, respectively, to the next reference paths 140b, c, as the transport vehicle 130 moves continuously along each of the reference paths 140a, 140b, 140c, and does not jump from one to the next. What results is that, even when taken in isolation, Figure 2 The predicted paths 120a, 120b, 120c of the example of FIG. 3 do not exhibit deviations from the corresponding reference paths 140a, 140b, 140c that exceed an error threshold, but the cumulative deviations produced over time present an error by the path planning system.

[0038] Still referring to Figure 2Time instability extreme cases can be detected from the predicted paths 120a, 120b, 120c that collectively take the associated reference paths 140a, 140b, 140c but omit. As described above, the reference paths 140a, b, c at successive time increments 200, 210, 220 are expected to smoothly extend from the preceding reference paths 140a, b to the next reference paths 140b, c as the transportation vehicle 130 continuously moves along each of the reference paths 140a, 140b, 140c and does not step from one path to the next. The quality of the motion of the transportation vehicle 130 can be described by a smoothness metric. A smoothness metric with the same parameters can be applied to assess the accuracy of the predicted paths 120a, 120b, 120c over a time frame. A series of predicted paths 120 should similarly follow a predetermined smoothness value, and the smoothness metric of a series of predicted paths 120 can indicate a deviation from the predetermined smoothness value. The deviation of the smoothness metric from the predetermined smoothness value can indicate an extreme case and record an error when testing the path planning system. According to an exemplary embodiment, specific points along a series of predicted paths 120a, b, c,... n are selected (e.g., 2 seconds) and fitted with a low order polynomial curve (e.g., a second order polynomial). Preferably, the selected points are within a shared distance that includes a 30 foot overlap of each predicted path 120. The root mean square error of the curve fit is used as the smoothness for the selected points (i.e., selected time instances) along the series of predicted paths 120n. Further, a weighted sum of all the smoothness for each available time point can yield an overall smoothness metric for assessing time instability.

[0039] Still further, the extreme case error can be associated with an overall uncertainty. The overall uncertainty value describes a measure of certainty associated with the predicted path 120 produced by the path planning system. In particular, measuring and communicating the certainty of the path planning system in the predicted path 120 produced thereby can be an important piece of information for the overall safety of the transportation vehicle 130. One way to obtain such an overall uncertainty measure is to use multiple path planning algorithms (i.e., a set of algorithms) simultaneously as a component of the autonomous and / or assisted functionality. The multiple path planning algorithms can independently produce multiple predictions in parallel. The multiple path planning algorithms can be completely independent or partially interdependent. The variance of the predicted paths 120 produced by the multiple path planning algorithms can be used as the overall uncertainty measure. As with the extreme case discussed above, if the overall uncertainty measure exceeds a predetermined threshold (i.e., the variance between the predicted paths produced by the multiple path planning algorithms exceeds a predetermined threshold), then an error is logged with the path planning system for the purpose of evaluation and improvement thereof. Further, in an exemplary embodiment, multiple parallel path predictions can be produced from the same neural network through dropout sampling, whereby the neural network makes a large number of predictions and each prediction omits (or drops) one or more inputs from the neural network, whereby flawed or inaccurate inputs can be identified.

[0040] Neural networks, such as those improved by the embodiments disclosed herein, can be constructed and tuned with the goal of facilitating operation of the same, i.e., the autonomous and / or assisted functionality in this case, across all types of inputs (e.g., all road conditions and situations) of the input domain of the model. The systems and methods provided in accordance with the disclosed embodiments can be used to develop autonomous and / or assisted functionality, such as when the neural network predicts a path of the vehicle across approximately the next 30 meters for each image. Optionally, further, supervised learning, which is the most common form of machine learning, involves enabling learning during a training phase based on a set of training data in order to be able to learn to recognize how to label input data for classification. Deep learning improves the supervised learning approach by considering multiple levels of presentation, in which each level uses information from the previous level to learn more deeply. The deeper architecture of multiple stacked layers is one aspect, i.e., convolutional neural networks (CNNs) consider 2D / 3D local neighborhood relations in the pixel / voxel space by convolving over spatial filters.

[0041] Supervised deep learning involves the application of multiple levels or stages of functional operations in order to improve the understanding of the data produced, which is then input into further functional operations. For example, supervised deep learning for classifying data into one or more categories can be performed, for example, by performing feature learning (involving one or more stages of convolution, rectifier linear unit (ReLU), and pooling) to enable subsequent classification of sample data, thereby identifying learned features by applying a softmax function to be able to distinguish between objects and background in input image data. These operations can be performed to produce image class labels for classification purposes.

[0042] Likewise, by operating in parallel on red-green-blue (RGB) image and distance / parallax to ground image data, by performing multiple convolutions, and by connecting the results via concatenation for subsequent processing, supervised deep learning operations can be performed for regression. These operations can be performed to produce image regression labels for subsequent use in analysis.

[0043] Further, supervised deep learning operations can be performed by inputting RGB image data into a convolutional encoder / decoder, which can include multiple convolutional stages, batch normalization (which is applicable not only to segmentation, but also to other networks), ReLU, and pooling, followed by multiple convolutional stages, batch normalization, and ReLU with upsampling, to perform semantic segmentation. The resulting data can then be processed by applying a softmax function to provide output data with segmentation labels for each pixel.

[0044] It is further understood that, while the disclosed embodiments can be used for the general purpose of facilitating robust autonomous and / or assisted transportation vehicle functionality, the disclosed embodiments can have particular utility in improving this functionality when lane markings are obscured due to weather conditions such as snow. In such cases, the driver often takes some unusual action. Comparison of the driver-directed path to the planned path can help to further improve the neural network that provides the path planning functionality.

[0045] It should be appreciated from this, that at least one embodiment can include a feedback mechanism that determines the quantity and / or quality of data produced and / or analyzed in the disclosed operations. This can be implemented, for example, by dynamically weighting data. It should also be appreciated that such a feedback mechanism can include a comparison to a threshold to at least maintain a minimum parameter, thereby ensuring safety of autonomous and / or assisted functionality operation. It should further be appreciated that the mechanism for dynamically weighting such data can be performed in one or more of a variety of conventionally known techniques that enable merging of sensor data, such as using a Kalman filter, processing performed based on a central limit theorem, Bayesian networks, Dempster-Shafer theorem, CNNs, or any other mathematical operation disclosed herein.

[0046] As noted above, the disclosed embodiments can be implemented in connection with components of autonomous and / or assisted driving systems included in a transportation vehicle. Accordingly, the utility of the disclosed embodiments in those technical contexts has been described in detail. However, the scope of the innovative concepts disclosed herein is not limited to those technical contexts. It should further be appreciated that the presently disclosed apparatus for evaluating one or more predicted paths to detect one or more errors in the predicted operation can include any combination of the sensors and functionality disclosed herein implemented in hardware and / or software, thereby providing the disclosed functionality.

[0047] It should further be appreciated that such assisted technologies can include, but are not limited to, technologies conventionally known as Driver Assistance Systems (DAS) or Advanced Driver Assistance Systems (ADAS) that are implemented using hardware and software included in a transportation vehicle. These conventionally known systems assist the driver in making decisions and controls, but the decisions and controls are inevitably the responsibility of the driver. Further, these systems can be “active” or “passive” in how they are implemented. An active DAS means that the vehicle itself controls various longitudinal and / or lateral aspects of vehicle driving behavior, or more specifically very specific driving tasks, through its sensors, algorithms, processing systems, and actuators. A passive DAS means that the vehicle will simply assist the driver in controlling various longitudinal and / or lateral aspects of vehicle control through its sensors, algorithms, processing systems, and human-machine interface (HMI). For example, in the case of collision avoidance, an active system would bring the vehicle to a stop or cause the vehicle to go around an obstacle in the current path. A passive system would provide some type of visual, audible, and tactile cues to the driver to bring the vehicle to a stop or go around the obstacle.

[0048] Accordingly, DAS systems can assist drivers with many tasks that are deeply embedded in the driving process and are implemented for the purpose of improving car and road safety and the convenience of drivers. Such DAS systems include, but are not limited to, cruise control, adaptive cruise control (ACC), lane-keeping active steering, lane-change assist, highway-merge assist, collision mitigation and avoidance systems, pedestrian protection systems, autonomous and / or assisted parking, sign recognition, blind spot detection for collision mitigation, and parking and traffic flow assist. Accordingly, the disclosed embodiments can assist in identifying inaccurate data collected by DAS systems to provide such assistive functions.

[0049] While the functions of the disclosed embodiments and the system components for providing the functions have been discussed with reference to specific terminology that represents the functions to be provided, it should be understood that the component functions can be provided, at least in part, by components that are now and known to be included in conventional transportation vehicles.

[0050] For example, as described above, the disclosed embodiments use software to perform functions so as to implement the measurement and analysis of data at least in part using software code stored on one or more non-transitory computer-readable media running on one or more processors in the transportation vehicle. Such software and processors can be combined to constitute at least one controller that is coupled to other components of the transportation vehicle to support and provide autonomous and / or assisted transportation vehicle functions in conjunction with the vehicle navigation system and the plurality of sensors. These components can be coupled to the at least one controller for communication and control over the CAN bus of the transportation vehicle. It should be understood that such a controller can be configured to perform the functions disclosed herein.

[0051] It should further be understood that the presently disclosed embodiments can be implemented using dedicated or shared hardware included in the transportation vehicle. Accordingly, components of the modules can be used by other components of the transportation vehicle to provide vehicle functions without departing from the scope of the present invention.

[0052] Exemplary embodiments are provided so as to enable a thorough and complete disclosure of the present invention, and to convey fully the scope thereof to those skilled in the art. Numerous specific details are set forth such as examples of specific components, devices, and methods, to provide a thorough understanding of embodiments of the present invention. In some instances, well-known processes, well-known device structures, and well-known technologies are not described in detail, because such

[0053] The terminology used herein is for the purpose of describing particular illustrative embodiments only and is not intended to be limiting. The singular forms "a," "an," and "the" include plural referents unless the context clearly dictates otherwise. The terms "comprises," "comprising," "includes," "including," and the like can be used herein and are intended to permit a statement that comprises, includes, and the like that does not or do not also (excluding other

[0054] The disclosed embodiments include the methods described herein and their equivalents, non-transitory computer readable media programmed to perform the methods, and computer systems configured to perform the methods. In addition, a vehicle is included that comprises components of any method, non-transitory computer readable media programmed to implement instructions or perform a method, and systems for performing a method. Computer systems and any sub-computer systems typically include a machine-readable storage medium containing executable code; one or more processors; a memory coupled to the one or more processors; an input device and an output device connected to the one or more processors for executing code. The machine-readable medium can include any mechanism for storing or transmitting information in a form readable by a machine, for example, a computer processor. For example, the information can be stored in volatile or non-volatile memory. In addition, the embodiment functionality can be implemented using embedded devices and online connectivity to cloud computing infrastructure that is available through radio connections (e.g., wireless communication) with such infrastructure. Training data sets, image data, and / or transformed image data can be stored in one or more memory modules coupled to the memory of the one or more processors.

[0055] While some embodiments have been described and illustrated in exemplary forms with a certain degree of particularity, it is noted that the description and illustrations have been made by way of example only. Numerous changes in the details of construction, combination, and arrangement of parts and operations can be made by those skilled in the art. Accordingly, such changes are intended to fall within the scope of the application, which is to be limited only by the appended claims.

[0056] The embodiments detailed above can be combined in whole or in part with any of the alternative embodiments described.

Claims

1. A transport vehicle device for detecting errors in a path prediction neural network, the device arranged on a transport vehicle and comprising: one or more sensors for collecting data describing a road traveled by the transport vehicle and a reference path; a processor and a memory; at least one neural network for predicting one or more predicted paths of the transport vehicle; wherein the processor is means for evaluating the one or more predicted paths to detect one or more errors in the prediction run, wherein the one or more errors comprise an indication that the one or more predicted paths of the path planning system deviate from the reference path; and wherein the memory comprises a ring buffer for continuously storing data from the one or more sensors and a semi-permanent memory, wherein data is transferred from the ring buffer to the semi-permanent memory in response to the one or more errors detected in the prediction run, and wherein the data transferred from the ring buffer to the semi-permanent memory comprises a first portion of data corresponding to a time period before the detected error and a portion of data corresponding to a time period after the detected error.

2. The transportation vehicle apparatus of claim 1, wherein, The means for evaluating compares the one or more predicted paths to the reference path.

3. The transportation vehicle apparatus of claim 2, wherein, The means for evaluating detects a deviation of the one or more predicted paths from the reference path.

4. The transport vehicle device of claim 3, further comprising a memory for storing data describing the one or more predicted paths and the reference path.

5. The transportation vehicle apparatus of claim 1, wherein, The means for evaluating determines a smoothness measure from the one or more predicted paths.

6. The transportation vehicle apparatus of claim 5, wherein, The one or more predicted paths comprise a plurality of predicted paths, and wherein the smoothness measure is based in part on a comparison of the plurality of predicted paths to each other.

7. The transportation vehicle apparatus of claim 1, wherein, The one or more predicted paths comprise a plurality of predicted paths, and wherein the plurality of predicted paths are generated by a plurality of neural networks.

8. The transportation vehicle apparatus of claim 1, wherein, The one or more predicted paths comprise a plurality of predicted paths, and wherein the plurality of predicted paths are generated by a single neural network according to dropout sampling.

9. The transportation vehicle apparatus of claim 7, wherein, The means for evaluating compares the plurality of predicted paths and detects a variance between the plurality of predicted paths to measure an uncertainty value.

10. The transportation vehicle apparatus of claim 8, wherein, The means for evaluating compares the plurality of predicted paths and detects a variance between the plurality of predicted paths to measure an uncertainty value.

11. A method for error detection of a path prediction neural network, the method comprising: analyzing data describing a road collected by one or more sensors arranged on a transport vehicle; predicting one or more predicted paths of the transport vehicle with at least one neural network; collecting data describing a reference path traveled by the transport vehicle; evaluating the one or more predicted paths to detect one or more errors in the prediction run, wherein the one or more errors comprise an indication that the one or more predicted paths of the path planning system deviate from the reference path, and In response to one or more errors detected in the predicted run, data is transferred from a ring buffer used to continuously store data from one or more sensors to semi-permanent storage, wherein the data transferred from the ring buffer to the semi-permanent storage includes a first portion of data corresponding to a time period prior to the detected error and a portion of data corresponding to a time period after the detected error.

12. The method for error detection of a path prediction neural network of claim 11, further comprising comparing the one or more predicted paths to a reference path.

13. The method for error detection of a path prediction neural network of claim 12, further comprising detecting a deviation of the one or more predicted paths from the reference path.

14. The method for error detection of a path prediction neural network of claim 13, further comprising storing data describing the one or more predicted paths and the reference path.

15. The method for error detection of a path prediction neural network of claim 11, further comprising determining a smoothness metric from the one or more predicted paths.

16. The method for error detection of a path-predicting neural network of claim 15, wherein, The one or more predicted paths comprise a plurality of predicted paths, and wherein the smoothness metric is based at least in part on a comparison of the plurality of predicted paths to each other.

17. The method for error detection of a path-predicting neural network of claim 11, wherein, The one or more predicted paths comprise a plurality of predicted paths, and wherein the plurality of predicted paths are generated by a plurality of neural networks.

18. The method for error detection of a path-predicting neural network of claim 11, wherein, The one or more predicted paths comprise a plurality of predicted paths, and wherein the plurality of predicted paths are generated by the neural network.

19. The method for error detection of a path-predicting neural network of claim 18, wherein, The plurality of predicted paths are generated by the neural network from dropout sampling.

20. The method for error detection of a path prediction neural network of claim 17, further comprising comparing the plurality of predicted paths, and detecting a variance between the plurality of predicted paths to measure an uncertainty value.

21. The method for error detection of a path prediction neural network of claim 18, further comprising comparing the plurality of predicted paths, and detecting a variance between the plurality of predicted paths to measure an uncertainty value.

22. A computing system disposed on a transportation vehicle for planning a transportation vehicle path, comprising: at least one memory configured to receive sensor data indicative of a reference path of the transportation vehicle; at least one processor communicatively coupled to the at least one memory, wherein the at least one processor is configured to execute program instructions to cause the system to perform the steps of: predicting one or more predicted paths of the vehicle by at least one neural network; detecting a deviation from the one or more predicted paths; determining an extreme case based on the reference path and the detected deviation; storing the extreme case; wherein storing comprises transferring data from a ring buffer used to continuously store data from one or more sensors to semi-persistent storage in response to one or more errors detected in the predicted run, wherein the data transferred from the ring buffer to the semi-persistent storage includes a first portion of data corresponding to a time period before the detected error and a portion of data corresponding to a time period after the detected error, and training at least one neural network with the stored extreme case.

Images