Vehicle autonomy level features

Abstract

A computer-implemented method for performing an autonomous level function associated with a moving vehicle is disclosed. The method includes extracting multiple features from multiple image frames obtained by a camera within the moving vehicle using a convolutional layer of a convolutional neural network. A correlation sequence between the multiple features is generated using a first fully connected (FC) layer of the convolutional neural network. The correlation sequence is reduced in dimension using a spatial long-short-term memory (LSTM) to generate a modified correlation sequence. Road parameter estimation data associated with the moving vehicle is generated using a second FC layer of the convolutional neural network and based on the modified correlation sequence. A lane keep assist system (LKAS) warning is provided based on the road parameter estimation data.

Description

[0001] Cross-reference to related applications Technical Field

[0002] The present invention relates to performing automated functions in a moving vehicle. Background Art

[0003] Over time, vehicles have become increasingly “intelligent” by achieving some degree of autonomy (e.g., through the fusion of perception sensor technology and artificial intelligence (AI)) and by cooperating with neighboring vehicles and infrastructure through vehicle-to-vehicle (V2V) or vehicle-to-infrastructure (V2I) communications. For example, vehicles on a highway communicate with each other through V2V so that every vehicle on the road can understand nearby vehicles.

[0004] The Society of Automotive Engineers International (SAE International) is a globally active professional association and standards-setting organization for engineering professionals in various industries, including the automotive and aerospace industries. SAE uses six levels (0 to 5) to classify vehicle autonomy. In a Level 0 vehicle, the human driver independently controls everything, including steering, throttle, brakes, etc. In a Level 1 vehicle, steering, acceleration, and braking are each independently and automatically controlled. Currently, there are a large number of vehicles operating on the road that include sensor technology such as lane keep assist system (LKAS) cameras. LKAS cameras extract lane geometry from road lane markings and enable Level 1 vehicles to perform lane keeping assist functions such as lane departure warning (LDW), road departure warning (RDW), and lane centering (LC).

[0005] In a Level 2 vehicle, multiple operations are done automatically and simultaneously. For example, simultaneous steering and acceleration to achieve a lane change can be performed automatically. In another example, General Motors' "Super Cruise" hands-free highway driving system operates by utilizing high-precision maps (e.g., high definition (HD) maps) and high-precision global positioning system (GPS) technology.

[0006] In a Level 3 vehicle, the vehicle driver will have more freedom to completely turn their attention away from the road under certain conditions. In other words, the vehicle driver will be able to completely hand over driving control to the vehicle, with the driver still able to monitor various vehicle systems and intervene when needed. Safety-critical functions can be transferred to the vehicle in specific situations.

[0007] In a Level 4 vehicle, the driver is not required to perform vehicle system monitoring. Level 4 vehicles are designed to operate safety-critical functions and monitor road conditions throughout the journey.

[0008] In a Level 5 vehicle, the driver does not need to be fit to drive or even have a driver's license. Level 5 vehicles can perform any and all driving tasks without human intervention. Level 5 vehicles have no driver cockpit and everyone in the car is a passenger. Level 3-5 vehicles fully delegate driving functions to the vehicle AI using perception sensor technology such as light detection and ranging (LIDAR) and high-precision positioning technology.

[0009] Currently, the vast majority of vehicles operating on the road are non-intelligent (e.g., Level 0) vehicles, as these vehicles do not have any high-precision sensor technology required to perform the autonomous functions of higher-level vehicles (e.g., LKAS for providing LDW, RDW, and LC). Even with the emergence of smarter vehicles, Level 0 vehicles are expected to maintain their popularity for a long time to come, as higher-level vehicles are expensive due to the use of high-precision sensors (e.g., high-precision LIDAR radar and GPS). In addition, traditional positioning technologies mostly rely on GPS technology, which cannot meet the accuracy requirements of modern intelligent transportation systems. Summary of the invention

[0010] Various examples are now described to briefly introduce a range of concepts that are further described in the detailed description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter.

[0011] According to a first aspect of the present invention, a computer-implemented method for performing an autonomous level function associated with a mobile vehicle is provided. The method includes extracting multiple features from multiple image frames obtained by a camera in the mobile vehicle using a convolutional layer of a convolutional neural network. A correlation sequence between the multiple features is generated using a first fully connected (FC) layer of the convolutional neural network. The correlation sequence is reduced in dimension using a spatial long short-term memory (LSTM) to generate a modified correlation sequence. Using a second FC layer of the convolutional neural network, road parameter estimation data associated with the mobile vehicle is generated based on the modified correlation sequence. Based on the road parameter estimation data, a lane keep assist system (LKAS) warning is provided.

[0012] In a first implementation of the method described in the first aspect, a temporal LSTM is used to detect one or more time series data patterns in the road parameter estimation data.

[0013] According to the first aspect or any of the above-mentioned implementations of the first aspect, in a second implementation of the method, the time LSTM is used to modify the road parameter estimation data according to the detected one or more time series data patterns to generate modified road parameter estimation data.

[0014] According to the first aspect or any of the above implementations of the first aspect, in a third implementation of the method, the modified road parameter estimation data includes lane parameter information and lane context information.

[0015] According to the first aspect or any one of the above implementations of the first aspect, in a fourth implementation of the method, the lane parameter information includes one or more of the following items: lane marking heading angle, lane marking offset, lane marking curvature, lane marking curvature derivative and lane marking type.

[0016] According to the first aspect or any of the above implementations of the first aspect, in a fifth implementation of the method, the lane context information includes one or more of the following items: a relative road distance, indicating the distance traveled by the mobile vehicle between two consecutive frames of the multiple image frames; a lane number indication of the lane of the road on which the mobile vehicle is traveling; and the number of lanes associated with the road on which the mobile vehicle is traveling.

[0017] According to the first aspect or any of the above implementations of the first aspect, in a sixth implementation of the method, the convolutional layer of the second convolutional neural network is used to extract the geographic location information of the mobile vehicle using the multiple image frames obtained by the camera.

[0018] According to the first aspect or any of the above implementations of the first aspect, in a seventh implementation of the method, one or more spatial constraints are applied to the geographic location information using Bayesian filtering to generate updated geographic location information of the mobile vehicle. The one or more spatial constraints are based on the road parameter estimation data. The updated geographic location information of the mobile vehicle is output.

[0019] According to a second aspect of the present invention, a system for performing autonomous level functions associated with a mobile vehicle is provided. The system includes a memory storing instructions and one or more processors communicating with the memory. The one or more processors execute the instructions to extract multiple features from multiple image frames obtained by a camera in the mobile vehicle using a convolutional layer of a convolutional neural network. A correlation sequence between the multiple features is generated using a first fully connected (FC) layer of the convolutional neural network. The correlation sequence is reduced in dimension using a spatial long short-term memory (LSTM) to generate a modified correlation sequence. Using a second FC layer of the convolutional neural network, road parameter estimation data associated with the mobile vehicle is generated based on the modified correlation sequence. Based on the road parameter estimation data, a lane keep assist system (LKAS) warning is provided.

[0020] In a first implementation of the system described in the second aspect, the one or more processors are further used to execute the instructions to use the temporal LSTM to detect one or more time series data patterns in the road parameter estimation data.

[0021] According to the second aspect or any of the above-mentioned implementations of the second aspect, in the second implementation of the system, the one or more processors are also used to execute the instructions to use the time LSTM to modify the road parameter estimation data according to the detected one or more time series data patterns to generate modified road parameter estimation data.

[0022] According to the second aspect or any of the above implementations of the second aspect, in a third implementation of the system, the modified road parameter estimation data includes lane parameter information and lane context information.

[0023] According to the second aspect or any of the above implementations of the second aspect, in a fourth implementation of the system, the lane parameter information includes one or more of the following items: lane marking heading angle, lane marking offset, lane marking curvature, lane marking curvature derivative and lane marking type.

[0024] According to the second aspect or any of the above implementations of the second aspect, in a fifth implementation of the system, the lane context information includes one or more of the following items: a relative road distance, indicating the distance traveled by the mobile vehicle between two consecutive frames of the multiple image frames; a lane number indication of the lane of the road on which the mobile vehicle is traveling; and the number of lanes associated with the road on which the mobile vehicle is traveling.

[0025] According to the second aspect or any of the above implementations of the second aspect, in a sixth implementation of the system, the one or more processors are also used to use the convolutional layer of the second convolutional neural network to extract the geographic location information of the mobile vehicle using the multiple image frames obtained by the camera.

[0026] According to the second aspect or any of the above implementations of the second aspect, in a seventh implementation of the system, the one or more processors are further configured to apply one or more spatial constraints to the geographic location information using Bayesian filtering to generate updated geographic location information of the mobile vehicle. The one or more spatial constraints are based on the road parameter estimation data. The updated geographic location information of the mobile vehicle is output.

[0027] According to a third aspect of the present invention, a non-transitory computer-readable medium is provided, storing instructions for executing autonomous level functions associated with a mobile vehicle. When the instructions are executed by one or more processors of a computing device, the one or more processors use a convolutional layer of a convolutional neural network to extract multiple features from multiple image frames obtained by a camera in the mobile vehicle. A correlation sequence between the multiple features is generated using the first fully connected (FC) layer of the convolutional neural network. The correlation sequence is reduced in dimension using spatial long short-term memory (LSTM) to generate a modified correlation sequence. Road parameter estimation data associated with the mobile vehicle is generated based on the modified correlation sequence using the second FC layer of the convolutional neural network. A lane keep assist system (LKAS) warning is provided based on the road parameter estimation data.

[0028] In a first implementation of the non-transitory computer-readable medium described in the third aspect, the instructions further cause the one or more processors to use a temporal LSTM to detect one or more time series data patterns in the road parameter estimation data.

[0029] According to the third aspect or any of the above-mentioned implementations of the third aspect, in a second implementation of the non-transitory computer-readable medium, the instructions also cause the one or more processors to use the temporal LSTM to modify the road parameter estimation data according to the detected one or more time series data patterns to generate modified road parameter estimation data.

[0030] According to the third aspect or any of the above implementations of the third aspect, in a third implementation of the non-transitory computer-readable medium, the modified road parameter estimation data includes lane parameter information and lane context information.

[0031] According to the third aspect or any of the above implementations of the third aspect, in a fourth implementation of the non-transitory computer-readable medium, the lane parameter information includes one or more of the following items: lane marking heading angle, lane marking offset, lane marking curvature, lane marking curvature derivative, and lane marking type.

[0032] According to the third aspect or any of the above implementations of the third aspect, in a fifth implementation of the non-transitory computer-readable medium, the lane context information includes one or more of the following items: a relative road distance, indicating the distance traveled by the mobile vehicle between two consecutive frames of the multiple image frames; a lane number indication of the lane of the road on which the mobile vehicle is traveling; and the number of lanes associated with the road on which the mobile vehicle is traveling.

[0033] According to the third aspect or any of the above implementations of the third aspect, in a sixth implementation of the non-transitory computer-readable medium, the instruction further causes the one or more processors to use the convolutional layer of the second convolutional neural network to extract the geographic location information of the mobile vehicle using the multiple image frames obtained by the camera.

[0034] According to the third aspect or any of the above implementations of the third aspect, in a seventh implementation of the non-transitory computer-readable medium, the instructions further cause the one or more processors to apply one or more spatial constraints to the geographic location information using Bayesian filtering to generate updated geographic location information of the mobile vehicle. The one or more spatial constraints are based on the road parameter estimation data. The updated geographic location information of the mobile vehicle is output.

[0035] According to a fourth aspect of the present invention, a system for performing an autonomous level function associated with a mobile vehicle is provided. The system includes an extraction module for extracting multiple features from multiple image frames obtained by a camera in the mobile vehicle. The system includes a correlation module for generating a correlation sequence between the multiple features. The system includes a dimensionality reduction module for performing dimensionality reduction of the correlation sequence to generate a modified correlation sequence. The system includes an estimation module for generating road parameter estimation data associated with the mobile vehicle based on the modified correlation sequence. The system includes a notification module for providing a lane keep assist system (LKAS) warning based on the road parameter estimation data.

[0036] Any of the above examples may be combined with any one or more of the other examples described above to create new embodiments within the scope of the invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] In the drawings, which are not necessarily drawn to scale, like numerals may describe similar components in different views.The drawings generally illustrate, by way of example and not limitation, various embodiments described herein.

[0038] Figure 1 It is a block diagram of using a DL architecture (DLA) to train a deep learning (DL) model provided by some example embodiments.

[0039] Figure 2 is a diagram of generating a trained DL model using a neural network model trained within DLA provided by some example embodiments.

[0040] Figure 3 Various SAE autonomy levels provided by some example embodiments are shown and using the techniques disclosed herein to enable a lower SAE autonomy level vehicle to use a computing device to perform higher SAE autonomy level functions.

[0041] Figure 4 Various lane parameter information is shown that may be used in conjunction with some example embodiments.

[0042] Figure 5 Lane context information is shown that may be used in conjunction with some example embodiments.

[0043] Figure 6 An LKAS module for estimating lane parameter information and lane context information provided by some example embodiments is shown.

[0044] Figure 7 Some example embodiments provide for Figure 6 The road parameters estimator sub-network (RPESN) of the LKAS module.

[0045] Figure 8 Some example embodiments provide for Figure 6 The road parameters estimator network (RPEN) of the LKAS module.

[0046] Figure 9A-9D Several image frames are shown for determining RPEN odometry as part of lane context information provided by some example embodiments.

[0047] Fig.10 Examples of continuous pose estimation through position tracking and position recognition provided by some example embodiments are shown.

[0048] Fig.11 An example feature sparse environment for use in conjunction with continuous pose estimation techniques provided by some example embodiments is shown.

[0049] Fig.12 A block diagram of a positioning module for high-precision positioning provided by some example embodiments is shown.

[0050] Fig.13 Some example embodiments provide for use of Fig.12 An example of a positioning module that performs high-precision positioning with lane accuracy.

[0051] Fig.14 Some example embodiments provide methods for using Fig.12 Examples of localization modules for suburban and urban environments.

[0052] Fig.15 is a flow chart of a method provided by some example embodiments for performing autonomous level functions associated with a mobile vehicle.

[0053] Fig.16 is a block diagram of a representative software architecture provided by some example embodiments that may be used in conjunction with the various device hardware described herein.

[0054] Fig.17 is a block diagram of circuits of devices implementing algorithms and executing methods provided by some example embodiments. DETAILED DESCRIPTION

[0055] First, it should be understood that although the following provides an illustrative implementation of one or more embodiments, Figure 1-Figure 17The disclosed systems and methods described may be implemented using any number of techniques, whether currently known or not yet in existence. The present invention should in no way be limited to the illustrative implementations, drawings, and techniques described below, including the exemplary designs and implementations illustrated and described herein, but may be modified within the scope of the appended claims along with their full scope of equivalents.

[0056] The present invention relates to enabling SAE International (originally Society of Automotive Engineers) vehicle autonomy level functionality in a vehicle.

[0057] The following is a detailed description in conjunction with the accompanying drawings, which are part of the description and show specific embodiments that can be practiced by way of diagrammatic illustration. These embodiments are described in sufficient detail to enable those skilled in the art to practice the disclosed subject matter, and it should be understood that other embodiments may be utilized and structural, logical, and electrical changes may be made without departing from the scope of the invention. Therefore, the description of the following example embodiments is not intended to be limiting, and the scope of the invention is defined by the appended claims.

[0058] As used herein, the terms "forward computation" and "backward computation" refer to computations performed in a work machine in connection with the training of a neural network model (or another model). The computations performed during forward computation and backward computation modify weights based on the results of previous iterations (e.g., based on the gradients generated at the end of previous backward computations). Gradients are a measure of the change in the output of a work machine when the weights of the model being computed by the work machine change. Gradients measure the change in all weights relative to the change in error. The larger the gradient value, the faster the model learns.

[0059] The term LKAS as used herein refers to a group of assistance systems that help the driver keep the vehicle in the proper lane. Examples of LKAS functions are LDW, RDW, LC, etc. LDW is a warning system that delivers a warning to the driver when the vehicle deviates from the lane without proper signal notification. RDW (also known as road departure mitigation (RDM)) is a warning system that delivers a warning to the driver when the vehicle leaves the road boundary. LC is a mechanism designed to keep the moving vehicle in the center of the lane, relieving the driver of the steering task.

[0060] As used herein, the term "High Definition Map" (or HD Map) refers to the category of maps built for the purpose of autonomous driving that are relevant to higher SAE autonomy level vehicles. HD Maps are characterized by extremely high precision, such as centimeter-level accuracy. HD Maps include information such as lane location, road boundary location, location and height of curbs, location of traffic signs and road markings, etc.

[0061] As used herein, the term "lane map" refers to a map category that includes georeferenced lane lines for roads. A lane map is a subset of an HD map that includes only lane line information.

[0062] As used herein, the term "class x vehicle" (where x is an integer between 0 and 5) refers to the SAE's level of vehicle autonomy.

[0063] As used in this article, the term “inertial measurement unit (or IMU)” refers to a device that combines an accelerometer, gyroscope, and magnetometer. An IMU is a device designed to track the position and orientation of an object and is embedded in most modern computing devices such as smartphones.

[0064] The technology disclosed herein can be used to enable a Level 0 vehicle to perform higher level functions (e.g., functions of a Level 1 or Level 2 vehicle) without incurring the additional cost of high precision sensors used by higher level vehicles. This can be accomplished by using the sensing capabilities of computing devices that are widely available to users (e.g., smartphone devices).

[0065] In the proposed technology, sensors of a computing device (e.g., camera, IMU, and GPS) use a neural network (NN) to extract lane information (e.g., road parameters) of a road, thereby enabling LKAS functionality on a vehicle without the cost of additional high-precision equipment. A training database is used, which includes image sequences of roads recorded from the driver's perspective while driving along the route, as well as corresponding consumer-level GPS data, industrial-level GPS data, and lane line maps of the route. The road parameter ground truth (which is the true value of the road parameters associated with the image) can be obtained from the high-precision geolocation provided by the lane line map and industrial-level GPS data. The dataset includes various road geometries, viewpoints from the driver's perspective, weather, and different times of the day. Then, a deep NN called a road parameter estimator network (RPEN) can be trained using input data (e.g., video frames) and corresponding output data (e.g., ground truth of road parameters associated with video frames). The trained NN is able to generalize, so it learns to detect lane lines and extract road parameters of roads that have never been seen. The trained NN can subsequently be used by a Level 0 vehicle using a computing device camera as input to generate road parameter estimation data including lane parameter information and lane context information for output. Lane parameter information can include lane marking heading angle, lane marking offset, lane marking curvature, lane marking curvature derivative, and lane marking type. Lane parameter information is used by the vehicle to perform LKAS functions, such as providing LDW and RDW warning notifications, which can be delivered to the vehicle driver via a display or other notification device.

[0066] The techniques disclosed herein can also be used to provide accurate vehicle positioning (e.g., lane-level positioning). The same training database and NN used to enable the LKAS function are used to provide road parameter estimation data including lane context information. Using a second NN (e.g., a vision-based relocalization algorithm), the second NN can predict the accurate geographic location (e.g., vehicle latitude and longitude) using an image sequence of the road from the driver's perspective and rough GPS data from a computing device (e.g., a smartphone used by the driver). Sensor fusion technology is used to combine the results of the two NNs to provide lane-level accuracy of the vehicle's current position. In this regard, a Level 0 vehicle driver can use the sensing capabilities of a computing device (e.g., a smartphone) to provide accurate positioning of the vehicle.

[0067] Existing technologies use expensive high-precision radar and GPS systems to perform higher levels of vehicle autonomy. In contrast, the technology disclosed herein can use neural networks running on consumer devices (e.g., smartphones) and device sensor capabilities (e.g., camera, IMU, GPS) to provide sensing functions for increasing the SAE autonomy level of the vehicle by enabling LKAS functions (e.g., LDW and RDW notifications) and providing lane-level vehicle localization.

[0068] The technology disclosed herein estimates new lane context information, such as the lane number in which the vehicle is located (also referred to as "egoLaneNo"), the number of lanes associated with the road on which the vehicle is traveling (also referred to as "noOfLanes"), and the relative road distance traveled by the vehicle between two consecutive image frames (also referred to as RPEN odometry or "relativeRoadDistance"), which is not estimated by existing navigation systems. In addition, the disclosed technology is used to improve the geolocation accuracy of existing positioning systems of vehicles by imposing spatial constraints on possible vehicle positions based on real-time extraction of lane context information (such as egoLaneNo) and other lane parameter information and lane context information.

[0069] Figure 1 A block diagram 100 of some example embodiments for training a deep learning (DL) model using a DL architecture (DLA) to generate a trained DL model 110. In some example embodiments, a machine-learning program (MLP) (including a deep learning program, also collectively referred to as a machine learning algorithm or tool) is used to perform operations associated with correlating data or other artificial intelligence (AI)-based functions.

[0070] like Figure 1 As shown, deep learning model training 108 is performed within DLA 106 based on training data 102 (which may include features). During deep learning model training 108, features from training data 102 may be evaluated to further train the DL model. DL model training 108 produces a trained DL model 110. The trained DL model 110 may include one or more classifiers 112, which may be used to provide DL evaluation results 116 based on new data 114.

[0071] In some aspects, the training data 102 may include input data 103, such as image data obtained from the driver's perspective, GPS data (e.g., geographic location information associated with the image data), and lane map data (e.g., lane map geographic location data associated with the image data). The training data 102 may also include road parameter ground truth data corresponding to the input data 103. The input data 103 and the output data 105 are used to train the DL model 110 during DL model training 108. In this regard, the trained DL model 110 receives new data 114 (e.g., images taken from the driver's perspective using a computing device such as a smartphone), detects lane lines, and uses the new data 114 to extract road parameter information (e.g., lane parameter information and lane context information) for roads that have never been seen before. The extracted road parameter information (e.g., output as a DL evaluation result 116) is used to provide the LKAS-related functions described herein (e.g., LDW / RDW notification and lane centering) as well as lane-level vehicle positioning information.

[0072] Deep learning is part of machine learning, a field of study that enables computers to learn without explicit programming. Machine learning explores the study and construction of algorithms (also referred to herein as tools) that can learn from existing data, can correlate data, and can make predictions about new data. Such machine learning tools work by building models based on example training data (e.g., training data 102) in order to make data-driven predictions or decisions represented by outputs or evaluation results 116. Although example embodiments are presented for some machine learning tools (e.g., deep learning architectures), the principles presented herein may also be applicable to other machine learning tools.

[0073] In some example embodiments, different machine learning tools may be used. For example, logistic regression, Naive-Bayes, random forest, neural network, matrix factorization, and support vector machine tools may be used during deep learning model training 108 (e.g., for associating training data 102).

[0074] Two common types of problems in machine learning are classification problems and regression problems. Classification problems (classification / categorization problems) aim to classify items into one of several category values ​​(e.g., is this object an apple or an orange?). Regression algorithms aim to quantify certain items (e.g., by providing real values). In some embodiments, DLA 106 can be used to use machine learning algorithms that use training data 102 to find correlations between identification features that affect the results.

[0075] The machine learning algorithm analyzes new data 114 using features in the training data 102 to generate an evaluation result 116. These features include individual measurable properties of the observed phenomena used to train the machine learning model. The concept of features is related to the concept of explanatory variables used in statistical techniques such as linear regression. Selecting informative, discriminative, and independent features is very important for the efficient operation of MLP in pattern recognition, classification, and regression. Features can be of different types, for example, numeric features, strings, and graphics. In some aspects, the training data can be of different types, and the features are numeric for use by a computing device.

[0076] In some aspects, features used during DL model training 108 may include input data 103, output data 105, one or more of the following: sensor data from multiple sensors (e.g., audio sensors, motion sensors, GPS sensors, image sensors); brake event data from multiple actuators (e.g., wireless switches or other actuators); external information from multiple external sources; timer data associated with sensor state data (e.g., acquisition time sensor data), actuator event data, or external information source data; user communication information; user data; user behavior data, etc.

[0077] The machine learning algorithm uses the training data 102 to find correlations between identification features that affect the evaluation results 116. In some example embodiments, the training data 102 includes labeled data, which is known data of one or more identification features and one or more results. The DL model training 108 within the DLA 106 trains the DL model using the training data 102 (which may include identification features). The training result is the trained DL model 110. When the DL model 110 is used for evaluation, the new data 114 is provided as input to the trained DL model 110, and the DL model 110 generates the evaluation results 116 as output. For example, the DLA 106 can be deployed on a mobile device, and the new data 114 can include LR images (e.g., frames from LR videos, such as real-time LR video feeds). The DLA 106 performs UR functions (e.g., improving image resolution while reducing noise, removing block artifacts, and improving image contrast) on the LR images to generate HR output images in real time.

[0078] Figure 2 FIG. 2 is a block diagram 200 for generating a trained DL model 206 using a neural network model 204 trained within a DLA 106 according to some example embodiments. Figure 2, source data 202 is analyzed by a neural network model 204 (or another type of machine learning algorithm or technique) to generate a trained DL model 206 (which may be the same as the trained DL model 110). Source data 202 includes a training data set, such as 102 (including input data 103 and output data 105), which includes data identified by one or more features. The terms "neural network" and "neural network model" are used interchangeably herein.

[0079] Machine learning techniques train models to accurately predict data that is fed into the model (e.g., what a user said in a given conversation sentence; whether a noun is a person, place, or thing; what the weather will be like tomorrow). During the learning phase, the model is developed based on the input training data set to optimize the model so that it correctly predicts the target output for a given input. Typically, the learning phase can be supervised, semi-supervised, or unsupervised; indicating the descending level of providing the "correct" output corresponding to the training input. In the supervised learning phase, all target outputs are provided to the model and the model is guided to develop general rules or algorithms that map inputs to outputs. In contrast, in the unsupervised learning phase, the desired outputs are not provided for the inputs so that the model can develop its own rules to discover relationships within the training data set. In the semi-supervised learning phase, an incompletely labeled training set is provided, where some outputs of the training data set are known and some are unknown.

[0080] The model may be run against a training data set for several cycles, where the training data set is repeatedly input into the model to refine its results (i.e., the entire data set is processed in one cycle). During iterations, the model (e.g., a neural network model or another machine learning model) is run against small batches (or portions) of the entire data set. In the supervised learning phase, the model is developed to predict the target output for a given set of inputs (e.g., source data 202) and evaluated over several cycles to more reliably provide an output that is specified to correspond to a given input for the maximum number of inputs of the training data set. In another example, for the unsupervised learning phase, a model is developed to cluster the data set into n groups, and the consistency of the model in placing a given input into a given group is evaluated over several cycles, as well as the reliability of the model in producing the n desired clusters in each cycle.

[0081] After running a cycle, the model is evaluated and the values ​​of the variables of these models (e.g., weights, biases, or other parameters) are adjusted to try to better refine the model through iteration. As used herein, the term "weights" is used to refer to the parameters used by the machine learning model. During the backward calculation, the model can output gradients, which can be used to update the weights associated with the forward calculation.

[0082] In various aspects, the assessment is biased towards false negatives, biased towards false positives, or uniformly biased towards the overall accuracy of the model. Depending on the machine learning technique used, the values ​​can be adjusted in a variety of ways. For example, in a genetic algorithm or evolutionary algorithm, the values ​​of the model that predicts the most successful desired output are used to develop values ​​used by the model in subsequent cycles, and these values ​​may include random changes / mutations to provide additional data points. Those of ordinary skill in the art are familiar with several other machine learning algorithms that can be applied to the present invention, including linear regression, random forests, decision tree learning, neural networks, deep neural networks, etc.

[0083] Each model develops rules or algorithms over several cycles by changing the values ​​of one or more variables that affect the input to more closely map to the desired outcome, but perfect accuracy and precision may not be achieved because the training data set may vary and is preferably very large. Therefore, the number of cycles that make up the learning phase can be set to a given number of trials or a fixed time / computational budget, or can be terminated before reaching that number / budget when the accuracy of a given model is high enough or low enough or reaches a plateau in accuracy. For example, if the training phase is designed to run for n cycles and generate a model with at least 95% accuracy, and such a model is generated before the nth cycle, the learning phase can end early and use the generated model that meets the final target accuracy threshold. Similarly, if the accuracy of a given model is not sufficient to meet the random probability threshold (e.g., the model is only 55% accurate in determining the true / false output for a given input), the learning phase for that model may be terminated early, but other models in the learning phase can continue to be trained. Similarly, when a given model continues to provide similar accuracy or fluctuations in results over multiple epochs (having reached a performance plateau), the learning phase for a given model may terminate before the epoch number / computational budget is reached.

[0084] After the learning phase is complete, the model is finalized. In some example embodiments, the finalized model is evaluated according to a test criterion. In a first example, a test data set including known outputs of the input is input into the finalized model to determine the accuracy of the model when processing data that has not been trained. In a second example, the false positive rate or the false negative rate can be used to evaluate the finalized model. In a third example, the demarcation between the data clusters in each model is used to select the model that produces the clearest boundaries for its data clusters.

[0085] In some example embodiments, the DL model 206 is trained by a neural network model 204 (e.g., a deep learning network, a deep convolutional network, or a recursive neural network), which includes a series of "neurons" (e.g., long short term memory (LSTM) nodes) arranged into a network. Neurons are architectural elements used for data processing and artificial intelligence, particularly machine learning, which include memory that can determine when to "remember" and when to "forget" the values ​​stored in the memory based on the weights of the inputs provided to a given neuron. Each neuron used in this article is used to receive a predefined number of inputs from other neurons in the network in order to provide relational and sub-relational outputs for the content of the frame being analyzed. Individual neurons can be linked together or organized into a tree structure in various configurations of neural networks to provide interactive and relational learning modeling to determine how each frame in a dialogue sentence is related to each other.

[0086] For example, an LSTM as a neuron includes several gates for processing an input vector (e.g., phonemes from a conversation sentence), a memory cell, and an output vector (e.g., a contextual representation). The input gate and output gate control the flow of information into and out of the memory cell, respectively, while the forget gate optionally removes information from the memory cell based on the input of earlier linked units in the neural network. The weights and bias vectors of the various gates are adjusted during the training phase, and once the training phase is complete, these weights and biases are finalized for normal operation. Those skilled in the art will appreciate that neurons and neural networks can be constructed programmatically (e.g., through software instructions) or by dedicated hardware that connects each neuron to form a neural network.

[0087] Neural networks use features to analyze data to generate evaluation results (e.g., recognize speech units). Features are individual measurable properties of the observed phenomenon. The concept of features is related to the concept of explanatory variables used in statistical techniques such as linear regression. In addition, deep features represent the output of nodes in the hidden layer of a deep neural network.

[0088] A neural network (e.g., neural network model 204), sometimes referred to as an artificial neural network (ANN) or neural network model, is a computing system based on the biological neural networks of animal brains. These systems gradually improve performance (called learning) to perform tasks, usually without task-specific programming. For example, in image recognition, a neural network can be taught to recognize images containing objects by analyzing example images that have been labeled with the names of the objects, and after learning the objects and names, the neural network can use the analysis results to recognize objects in unlabeled images. A neural network is based on a collection of connected units called neurons, where each connection between neurons (called a synapse) can transmit a unidirectional signal, and its activation strength varies with the strength of the connection. A receiving neuron can activate and propagate a signal to downstream neurons connected to it, which is usually based on whether the combined input signal from potentially many transmitting neurons has sufficient strength, where strength is a parameter.

[0089] A deep neural network (DNN), also known as a convolutional neural network (CNN), is a stacked neural network consisting of multiple convolutional layers. These layers are made up of nodes, which are locations where computations occur, loosely laid out to form patterns like neurons in the human brain, which fire when they encounter enough stimulation. A node combines the input of data with a set of coefficients or weights that can amplify or suppress that input, which assigns importance to the input for the task the algorithm is trying to learn. These input-weight products are summed and the sum is passed through the node's activation function to determine whether and to what extent the signal passes further through the network to affect the final result. DNNs use a cascade of multiple layers of nonlinear processing units for feature extraction and transformation. Each successive layer uses the output of the previous layer as input. High-level features are derived from low-level features to form a hierarchical representation. The layers after the input layer can be convolutional layers, which produce feature maps that are the filtered results of the input and are used by the next convolutional layer.

[0090] In the training of the DNN architecture, regression can include the minimization of a cost function, where regression is constructed as a set of statistical processes for estimating the relationship between variables. The cost function can be implemented as a function that returns a numerical value representing the performance of the neural network in mapping the training examples to correct the output. During training, if the cost function value is not within a predetermined range, back propagation is used based on known training images, where back propagation is a common method for training artificial neural networks, used together with optimization methods such as stochastic gradient descent (SGD) method.

[0091] The use of backpropagation can include both propagation and weight updates. When an input is presented to a neural network, that input is propagated forward through the neural network layer by layer until it reaches the output layer. The output of the neural network is then compared to the desired output using a cost function, and an error value is calculated for each node in the output layer. The error values ​​are propagated backward from the output until each node has an associated error value that roughly represents its contribution to the original output. Backpropagation can use these error values ​​to calculate the gradient of the cost function with respect to the weights in the neural network. The calculated gradients are input into the optimization method of choice to update the weights in an attempt to minimize the cost function.

[0092] Even though the training architecture 106 is referred to as a deep learning architecture using a neural network model (and the trained models are referred to as trained deep learning models, such as the trained DL models 110 and 206), the present invention is not limited in this regard, and other types of machine learning training architectures may also be used for model training using the techniques disclosed herein.

[0093] Figure 3 A block diagram 300 is shown of various SAE autonomy levels provided by some example embodiments and the use of the techniques disclosed herein to enable a lower SAE autonomy level vehicle to use a computing device to perform higher SAE autonomy level functions. Figure 3 , block diagram 300 shows vehicles 302, 304, 306, ... 308 associated with SAE autonomy levels 0, 1, 2, ... 5, respectively. Vehicle 304 performs level 1 functions, such as providing lane departure warning 310. Vehicle 306 performs level 2 functions, such as lane centering 316. Vehicle 308 performs level 5 functions, such as automated driving 322.

[0094] like Figure 3As shown, lane departure warning 310 can be based on road parameter information, such as lane line information 314 detected by a high-precision LKAS camera 312. Lane centering 316 and autonomous driving 322 functions use high-precision positioning 320, which is performed using high-precision GPS 318 and high-precision radar or LIDAR (such as LIDAR 324) present on more advanced vehicles.

[0095] In some aspects, a computing device 330 (e.g., a smartphone or other consumer device) can be used within a Level 0 vehicle 302 to perform techniques 326 (e.g., for determining road parameter estimation data) and techniques 328 (e.g., for determining lane-level vehicle positioning) described herein. In this regard, by using computing device 330 to perform techniques 326 and 328, Level 0 vehicle 302 can be upgraded to a higher-level vehicle 332 without the need for expensive high-precision equipment, such as LKAS camera 312, high-precision GPS 318, and LIDAR 324.

[0096] Figure 4 Various lane parameter information that may be used in conjunction with some example embodiments is shown. Figure 4 , schematic 400 shows a vehicle 406 located on a road including lanes 402 and 404. In some aspects, a front-facing camera (e.g., an LKAS camera or a camera of a computing device such as a smartphone) mounted within vehicle 406 can be used to capture multiple images of the road and extract lane parameter information, the lane parameter information including one or more of the following: lane marking heading angle (C1), lane marking offset (C0), lane marking curvature (C2), lane marking curvature derivative (C3), and lane marking type. Figure 4 In , C2 and C3 are zero because the road is straight and has no curvature.

[0097] In some aspects, a front-facing camera installed in the vehicle 406 can use the techniques disclosed herein to extract lane parameter information. More specifically, the camera can detect lane markings (or lane lines), each of which can be modeled by a third-order polynomial model describing the function X(Z), where Z is the physical longitudinal distance from the camera and X is the physical lateral offset from the camera. The following formula can be used to derive the physical lateral offset X from the camera: X = C3·Z 3 +C2·Z 2+C1·Z+C0, where C0 is the lane marking offset 408 at Z=0 (e.g., the distance from the center of the vehicle 406 to the left lane line or the right lane line at Z=0 when the vehicle is in lane 1 402 or lane 2 404, in meters), C1 is the lane marking heading angle at Z=0 (e.g., the angle between the direction of the road 412 and the direction of the vehicle 410), C2 is the lane marking curvature at Z=0, and C3 is the lane marking curvature derivative at Z=0. For a straight lane, the lane marking curvature C2≈0 and the lane marking curvature derivative C3≈0.

[0098] Figure 5 Lane context information that may be used in conjunction with some example embodiments is shown. Figure 5 , schematic diagram 500 shows a vehicle 506 moving on a road having two lanes (lane 1 502 and lane 2 504). In some aspects, lane context information determined according to the techniques disclosed herein may include one or more of the following: a relative road distance (Δs) 512 indicating the distance traveled by the moving vehicle 506 between two consecutive image frames captured by an onboard camera (e.g., the distance traveled between vehicle positions 508 and 510); an indication of the lane number of the lane of the road on which the moving vehicle is traveling (e.g., egoLaneNo information indicating that lane 1 502 is the lane in which the vehicle 506 is traveling); and a lane number associated with the road on which the moving vehicle is traveling (e.g., for Figure 5 In the example shown in , noOfLanes is 2).

[0099] Figure 6 The LKAS module 600 for estimating lane parameter information and lane context information provided by some example embodiments is shown. The module may include hardware and / or software designed to perform one or more functions. Figure 6 , the LKAS module 600 may include a DNN 604, which may be trained using a plurality of video frames 602 to generate road parameter estimation data 610, the plurality of video frames 602 being captured by an onboard camera (e.g., a camera on a computing device of a user of the vehicle). In some aspects, the DNN 604 may use a neural network model (e.g., Figure 2A neural network model 204 has been trained using input data 103 and output data 105 so that road parameters can be estimated from image data from a driver's perspective (e.g., video frames captured from a front camera installed in a moving vehicle). During training, a database is used that includes input data 103 (e.g., image sequences of roads recorded from a driver's perspective while driving and their corresponding lane maps of consumer-grade GPS, industrial-grade GPS, and routes) and output data 105 (e.g., ground truth values ​​of road parameters that can be obtained from lane maps and high-precision geolocation provided by industrial-grade GPS). The training dataset (e.g., data 103 and 105) also includes training images with various road geometries and different viewpoints from the driver's perspective, taken in different weather and times of the day.

[0100] In some aspects, the DNN 604 is also referred to as a road parameters estimator network (RPEN) and includes a road parameters estimator sub-network (RPESN) 702 and at least one LSTM node 806, such as in combination with Figure 7 and Figure 8 The trained DNN 604 is able to generalize, and thus learn to observe lane lines and extract lane information for roads that have never been seen.

[0101] In operation, the LKAS module 600 receives a video frame 602 (e.g., via a Level 0 vehicle driver's computing device camera), which is passed as an input to a DNN 604. The DNN 604 uses the RPESN 702 and the LSTM 806 to generate road parameter estimation data 610, which may include lane parameter information 608 and lane context information 606. The lane context information 606 may include relative road distance (or odometer information, such as Figure 5 Relative road distance 512), egoLaneNo information and noOfLanes information, such as combined Figure 5 The lane parameter information 608 may include lane marking heading angle information, lane marking offset information, lane marking curvature information, lane marking curvature derivative information, and lane marking type, such as in combination with Figure 4In this regard, the trained DNN 604 uses images received from the Level 0 vehicle user's computing device as input to produce lane context information and lane parameter information as output. The road parameter estimation data 610 can be used to generate LKAS warnings including LDW or RDW. In addition to providing sensor functionality for the LKAS function, the consumer device can also communicate warnings to the user through its display, effectively providing an end-to-end solution without the help of external components.

[0102] Figure 7 Some example embodiments provide for Figure 6 The road parameters estimator sub-network (RPESN) of the LKAS module. Figure 7 , RPESN 702 includes a pre-trained CNN 706, a pooling layer 708, a first fully connected (FC) layer 710, a spatial LSTM 712 and a second FC layer 714.

[0103] The input of RPESN 702 is a plurality of video frames 704 captured by a camera of a consumer device (e.g., a computing device, such as a smartphone of a Level 0 vehicle driver), which are first fed to a pre-trained CNN 706. The pre-trained CNN 706 is used as a classification network instead of training a deep neural network from scratch for two reasons: (a) training a convolutional network relies on a very large labeled image dataset, which increases the implementation cost and processing time; (b) unlike the classification problem, each output label is covered by at least one training sample, and the output space in regression is theoretically continuous and infinite. Therefore, training a convolutional neural network from scratch can be omitted, and instead a pre-trained classification network that has been trained on a large image dataset can be used. After the convolutional layer of the pre-trained CNN 706, an average pooling layer 708 collects information of each feature channel of the entire image. After the pooling layer 708, the first FC layer 710 learns to extract the correlation between the features. The output of the first FC layer 710 can be viewed as a sequence that is fed to the spatial LSTM 712 whose memory block performs dimensionality reduction. The spatial LSTM 712 is used to evaluate the spatial relationship of features based on the sequence output from the first FC layer 710. In this regard, the spatial LSTM 712 memory unit identifies the most useful feature correlations in the road parameter estimation task. The output of the spatial LSTM 712 is then passed to the second FC layer 714, which generates road parameter estimation data 716 including lane context information.

[0104] Figure 8Some example embodiments provide for Figure 6 The road parameters estimator network (RPEN) 800 of the LKAS module. Figure 8 , RPEN 800 receives as input a plurality of images and 802 processed by RPESN 804. In some aspects, the road parameter estimation data generated by RPESN 804 is further processed by temporal LSTM 806 to generate modified (or final) road parameter estimation data 808.

[0105] For time instant i, RPEN 800 processes a fixed-length visual input of size k frames using RPESN 804 (same as RPESN 702), the output of which is fed into a stack of recurrent sequence models (temporal LSTM 806). Temporal LSTM 806 generates final road parameter estimation data 808 including lane context information (e.g., relative road distance information) and land parameter information. In some aspects, the relative road information in the lane context information is also referred to as RPEN odometry information.

[0106] Unlike a traditional feedforward neural network, which has no concept of temporal order and the only input it considers is the current example it is exposed to, the temporal LSTM 806 can use its internal state (memory) to recognize patterns in time series data (e.g., sequence of image frames 802) and generate modified (or final) road parameter estimation data 808. Figure 8 As shown, the temporal LSTM 806 forms a network with a loop in which the output from the previous time is fed as input to the current time, allowing past information to persist. The temporal LSTM 806 is used to model the temporal dynamics and dependencies of the road parameter estimation data. For example, the temporal constraint binds the current state of the road parameter to its previous state, and a significant change in the lane marking offset information (C0) (e.g., above a predetermined threshold) can trigger a change in the egoLaneNo information. Changes in the noOfLanes information are usually accompanied by changes in the lane line type (e.g., a dashed line becomes a solid line or a road merge sign is ahead).

[0107] Figure 9A-9D A plurality of image frames for determining RPEN odometry as part of lane context information provided by some example embodiments are shown. Figure 9A-9D, showing a sequence of image frames 900A, 900B, 900C, and 900D received by the RPEN 800 for processing. More specifically, a pre-trained CNN 706 within the RPESN 702 is trained to model the shape and contours of lane lines (e.g., virtual lane lines 906), and how the view of the lane lines changes as the distance from these lane lines changes. These high-level features can be reused to estimate RPEN odometer information associated with the moving vehicle 904 (e.g., relativeRoadDistance information or Δs 902). The high-dimensional representation of the virtual lane lines 906 on the road can form a smoothly varying introjective (one-to-one) function of Δs 902. The virtual lane lines 906 are constantly present in the road and are therefore good candidates for modeling by the RPEN 800 in order to regress Δs 902. The dimensionality reduction capability of the spatial LSTM enables the RPEN 800 to identify the virtual lane line 906 as the most useful feature correlation for the task of regressing the relativeRoadDistance information 902 directly from the image frames 900A-900D. Since the estimation of the relativeRoadDistance information 902 depends on the features from the previous frame, the temporal LSTM 806 retains this information about the previous frame. Therefore, the RPEN 800 can estimate the relativeRoadDistance information 902 by observing only the road.

[0108] Vision-based relocalization (i.e., continuously estimating the pose (3D position and orientation) of a moving camera) using monocular images is a fundamental problem in computer vision and robotics. With the advent of autonomous vehicles, localizing vehicles on the road using inexpensive perception sensors (e.g., monocular cameras) has recently become a very popular research topic. There are generally two approaches to tackle the problem of continuous pose estimation of a moving camera: (1) via position tracking (visual odometry) and (2) via scene recognition (vision-based relocalization).

[0109] Fig.10 Examples 1000A and 1000B of continuous pose estimation through position tracking and position recognition provided by some example embodiments are shown. Fig.10 , example 1000A illustrates continuous pose estimation via position tracking (also known as visual odometry). In example 1000A, robot 1002 moves from position 1004A to position 1006A in a landscape whose map is unknown, but the initial robot pose at position 1004A is known. Fig.10 As shown, visual odometry localization techniques are locally accurate but typically result in drift 1008A over time.

[0110] refer to Fig.10, example 1000B illustrates continuous pose estimation by location recognition (also known as vision-based relocalization). In example 1000B, robot 1002 moves from location 1004B to location 1006B in a landscape where the map is known but the initial robot pose at location 1004B is unknown. Fig.10 As shown, vision-based relocalization techniques are associated with noise predictions having minimal or no drift 1008B.

[0111] As described above, in the case of pose estimation by position tracking (e.g., 1000A), the initial pose of the camera is known and a map of the operating area is not necessarily required. In example 1000A, the current pose is updated by integrating the changes in the pose estimate provided by the perception sensors over time. For example, visual odometry can be used to compute the relative pose between consecutive frames based on sparse feature detections. The pose estimates are locally accurate but typically drift over time. Another limitation of visual odometry is that it performs poorly in environments where features cannot be extracted, such as Fig.11 , which may be a feature-sparse rural road environment. RPEN 800, on the other hand, is trained to detect lane lines. RPEN 800 learns the shape and contours of lane lines, and also learns how the appearance of lane lines changes with distance. In this regard, RPEN-based odometers work where visual odometers fail.

[0112] In the case of pose estimation by scene recognition (often called vision-based relocalization), the initial pose of the camera is unknown, while a map of the operating area is known. Based on the observed surrounding scene, the scene can be classified in a finite number of discrete positions, resulting in a coarse estimate of the camera pose. Here, the pose prediction is noisy but drift-free. Modern vehicle localization systems can exploit the complementary, rough but drift-free predictions of vision-based relocalization algorithms and the locally smooth but drifting pose estimates of visual odometry.

[0113] There are generally two approaches to vision-based relocalization (Example 1000B): point feature-based relocalization and machine learning-based relocalization. In the point feature-based approach, given a 3D model and a query image, the first step is to establish a set of 2D-3D matches that associate pixel locations in the query image with 3D points in the 3D model. Next, the camera pose is estimated by applying an n-point pose solver that calculates the camera pose based on n 2D-3D matches within a random sample consensus (RANSAC) loop to handle outliers. The classic approach to building a 3D model is to use structures in motion. These points are essentially reconstructed by essentially triangulating local feature matches, creating a many-to-one mapping from feature descriptors to 3D points. One of the challenges of this approach is the inefficient scaling of scene size. The computational complexity of matching increases as the size of the scene increases. In addition, as the probability of finding multiple 3D points with similar local appearances increases, the uniqueness of 2D-3D matching decreases. Another challenge is that this approach requires a good initial pose estimate, which can be challenging to handle when the scene has inconsistent illumination, motion blur, textureless surfaces, and lack of overlap between images. This can be a concern in vehicle localization applications, especially when the vehicle is moving quickly on the highway, which is often a feature-free area.

[0114] Machine learning-based localization (especially deep learning) has recently shown great promise in vision-based localization using monocular images. Some localization techniques (e.g., PoseNet) use deep CNNs to regress the continuous pose of the camera directly based on RGB images. However, the accuracy of PoseNet is far from the requirements of practical applications, and there are difficulties in learning between position and orientation in the same model. Some improvements to PoseNet include a method called Posenet17. This deep learning method attempts to learn and infer weights based on scene geometry. In other methods, such as MapNet, sensory inputs such as visual odometry and GPS are fused in addition to images for camera localization. Other methods introduce CNN and LSTM architectures to directly regress geolocation (e.g., through geospatial deep NNs), using monocular camera images and coarse GPS in mobile phones to predict geolocation with close to lane-level accuracy. Compared to traditional methods, machine learning methods have several advantages that make them suitable for real-time applications such as vehicle relocalization: (a) deep learning methods do not require an initial pose estimate; (b) machine learning methods are more robust to environmental challenges such as lighting, weather, dynamic objects, and textureless scenes (this is because it can learn richer features than can be obtained from point features); (c) deep learning methods have faster inference capabilities; and (d) machine learning methods for localization can scale efficiently with scene size because it is able to learn a compact representation of the map using a deep learning model. However, the accuracy of machine learning methods is not as optimal as traditional feature-based methods, especially for indoor environments. Vision-based relocalization relies on landmark recognition. Therefore, it performs well in urban areas but performs poorly in suburban or rural areas. Unfortunately, even in urban areas, no known localization technology can provide lane-level localization.

[0115] Lane-level accuracy in vehicle localization is highly desirable because it opens up many applications that consumer devices can rely on to provide an end-to-end solution without the help of external components. For example, with lane-level pose perception and localization, the following goals can be achieved: (a) the estimated "time of arrival (TOA)" prediction made by the car navigation system in the consumer device can be more accurate if it knows the lane the car is currently traveling in; and (b) the car can be warned in advance if the lane it is currently traveling in will soon merge or be congested due to an accident or construction zone.

[0116] Fig.12 FIG. 1 is a block diagram of a positioning module 1200 for high-precision positioning provided by some example embodiments. Fig.12 , the positioning module 1200 includes a RPEN 1202 (which can be used with Figure 8The present invention also includes a sensor fusion module 1206 and a neural network 1204.

[0117] The sensor fusion module 1206 may receive as input: RPEN odometer information 1216 and road parameter estimation data from the RPEN 1202, longitude and latitude information 1220 from the neural network 1204, longitude and latitude information 1222 from the coarse GPS 1212, additional information such as the distance 1224 traveled by the vehicle between adjacent poses from the visual odometry or IMU input 1208 (which may be extracted from the computing device IMU sensor), and lane map information 1214.

[0118] The neural network 1204 may be an off-the-shelf relocalization algorithm, such as a publicly available geospatial NN, which is deployed with the RPEN 1202 to provide spatial constraints of possible vehicle positions. The geospatial NN 1204 predicts a geographic location 1220 (latitude and longitude) with lane-level accuracy using an image sequence of the road from the driver's perspective and latitude and longitude information 1222 from a coarse GPS 1212 of a computing device (e.g., the driver's consumer device, such as a smartphone).

[0119] RPEN 1202 may predict road parameters using a sequence of images of the road from the driver's perspective obtained from a computing device (e.g., as combined with Figure 3-9D RPEN 1202, the output of geospatial NN 1204, and lane map information 1214 and longitude and latitude information 1222 are passed as inputs to a Bayesian filtering algorithm within the sensor fusion module 1206, which predicts an accurate (e.g., lane-based) geographic location as output 1226. Sensor inputs to the Bayesian filtering-based sensor fusion algorithm used by the sensor fusion module 1206 include: spatial constraints imposed by RPEN odometer information 1216 and 1218 and lane map information 1214, absolute position information from coarse GPS 1212 and an off-the-shelf relocalization algorithm (e.g., geospatial NN 1204), and constraints between adjacent poses from the IMU, visual odometry, and odometer output from the RPEN.

[0120] In some aspects, the Bayesian filtering-based vehicle pose estimation algorithm used by the sensor fusion module 1206 includes the following processing functions: (a) predicting the vehicle's geographic location after time t+1 based on previous knowledge of the vehicle's location (time t) and the kinematic equations; (b) given sensor observations, the Bayesian filter compares these observations to the prediction; and (c) updating knowledge about the vehicle's geographic location based on the predictions and sensor readings transmitted as input to the sensor fusion module 1206.

[0121] Fig.13 Some example embodiments provide for use of Fig.12 Example 1300 of a positioning module performing high-precision positioning with lane accuracy.

[0122] Given the lane map information, the road parameter estimation data (e.g., 1216 and 1218) of RPEN 1202 has a higher lateral accuracy than the longitudinal accuracy of the road. Fig.13 As shown, the RPEN 1202 output provides a lateral constraint in a probability distribution of vehicle positions. The lane map can then be used to convert this lateral constraint into a geographic coordinate space (latitude, longitude).

[0123] like Fig.13 As shown, a coarse GPS accuracy position 1302 is achieved based on the output from the coarse GPS 1212. The geospatial NN 1204 is able to almost reach the lane level accuracy position 1304 using only images from the driver's perspective and coarse GPS data. The RPEN odometer information 1216 (e.g., relative road distance information) and inputs such as visual odometry and IMU generate constraints between adjacent poses, while the off-the-shelf relocalization algorithm, coarse GPS output, and RPEN road parameters 1218 impose spatial constraints on the position of the vehicle, thereby generating a RPEN positioning accuracy position 1306. The Bayesian filtering algorithm (e.g., particle filter (PF) or extended Kalman filter (EKF)) used by the sensor fusion module 1206 utilizes these constraints to accurately locate the vehicle and achieve a lane level accuracy position 1308 by fusing the coarse GPS accuracy position 1302, the geospatial NN accuracy position 1304, and the RPEN accuracy position 1306. In some aspects, it may be possible to achieve accurate lane-level positioning of the vehicle without visual odometry and IMU input 1208 .

[0124] Fig.14 Some example embodiments provide methods for using Fig.12 An example suburban environment 1402 and an example urban environment 1404 of the positioning module.

[0125] Existing vision-based relocalization algorithms (e.g., geospatial NN 1204) rely on landmark recognition. In this regard, geospatial NN 1204 performs well in urban areas (e.g., 1404), but performs poorly in suburban or rural areas (e.g., 1402). Urban areas can be problem areas for GPS, but GPS performs very well in rural or suburban areas. RPEN relies on roads and their lane markings, which are typically ignored by vision-based relocalization. Therefore, vision-based relocalization, GPS, and RPEN complement each other and can be used by the positioning module 1200 to provide accurate lane-based positioning.

[0126] Fig.15 1500 is a flow chart of a method 1500 for performing autonomous level functions associated with a mobile vehicle provided by some example embodiments. The method 1500 includes operations 1502, 1504, 1506, 1508, and 1510. By way of example and not limitation, the method 1500 may be performed by the LKAS module 1760, which is used to Fig.17 The device 1700 shown is executed in a mobile device.

[0127] refer to Fig.15 In operation 1502, a plurality of features are extracted from a plurality of image frames obtained by a camera in a moving vehicle using a convolutional layer of a convolutional neural network. For example, a pre-trained CNN (e.g., pre-trained CNN 706, which is part of CNN 604) extracts a plurality of features from an image (e.g., video frame 704) obtained by a vehicle-mounted camera. In operation 1504, a correlation sequence between the plurality of features is generated using a first FC layer of the deep neural network. For example, the first FC layer generates a correlation sequence between the plurality of features extracted by the pre-trained CNN.

[0128] In operation 1506, the correlation sequence is reduced in dimension using a spatial long-short-term memory (LSTM) model to generate a modified correlation sequence. For example, a spatial LSTM (e.g., LSTM712) within a road parameter estimator sub-network (RPESN) (e.g., RPESN 702) performs dimensionality reduction of the correlation sequence. In operation 1508, a second FC layer of the deep neural network is used to generate road parameter estimation data associated with a moving vehicle based on the modified correlation sequence. For example, the second FC layer 714 generates road parameter estimation data 716 associated with the moving vehicle. In operation 1510, a lane keep assist system (LKAS) warning is provided based on the road parameter estimation data. For example, an LDW or RDW notification can be provided by a computing device used to generate the road parameter estimation data 716.

[0129] Aspects disclosed herein use consumer devices (eg, 1700) to provide the sensing functionality required to increase the SAE autonomy level of a vehicle. The RPEN described herein estimates new parameters such as egoLaneNo, noOfLanes, and relativeRoadDistance.

[0130] In addition, the RPEN described in this paper provides constraints between adjacent poses by observing how the lane appearance changes with the distance from the lane. The pose estimation algorithm based on Bayesian filtering can take advantage of this constraint to improve the geolocation accuracy and provide lane-based vehicle positioning. The technology disclosed in this paper is used to improve the geolocation accuracy of existing vehicle positioning systems by imposing spatial constraints on the possible positions of the vehicle based on the lane context information such as egoLaneNo (which lane number the self-aware vehicle is on) and the real-time extraction of lane parameters provided by the visual sensor given the lane map of the road.

[0131] Fig.16 is a block diagram of a representative software architecture 1600 provided by some example embodiments, which may be used in conjunction with the various device hardware described herein. Fig.16 This is merely a non-limiting example of software architecture 1602, and it should be understood that many other architectures may be implemented to facilitate the functionality described herein. Software architecture 1602 may be implemented in the following manner: Fig.17 The method is executed in hardware such as device 1700, which includes a processor 1705, a memory 1710, storages 1715 and 1720, an input interface 1725, an output interface 1730, and the like.

[0132] A representative hardware layer 1604 is shown, which may represent Fig.17 The representative hardware layer 1604 includes one or more processing units 1606 with associated executable instructions 1608. The executable instructions 1608 represent executable instructions of the software architecture 1602, including implementing Figure 1-Figure 15 Methods, modules, etc. of the hardware layer 1604. The hardware layer 1604 also includes a memory or storage module 1610, which also has executable instructions 1608. The hardware layer 1604 may also include other hardware 1612, which represents any other hardware of the hardware layer 1604, such as other hardware illustrated as part of the device 1700.

[0133] exist Fig.16 In the example architecture of , software architecture 1602 can be conceptualized as a stack of layers, where each layer has a specific function. For example, software architecture 1602 may include layers such as operating system 1614, library 1616, framework / middleware 1618, application 1620, and presentation layer 1644. In operation, application 1620 or other components within each layer can call application programming interface (API) call 1624 through the software stack, and receive a response, return value, etc. shown as message 1626 in response to API call 1624. Fig.16 The layers shown are representative in nature, and not all software architectures 1602 have all layers. For example, some mobile or dedicated operating systems may not provide framework / middleware 1618, while other operating systems may provide such layers. Other software architectures may include other layers or different layers.

[0134] The operating system 1614 can manage hardware resources and provide public services. The operating system 1614 may include a kernel 1628, services 1630, and drivers 1632, among others. The kernel 1628 may serve as an abstraction layer between the hardware and other software layers. For example, the kernel 1628 may be responsible for memory management, processor management (e.g., scheduling), component management, networking, security settings, and the like. Services 1630 may provide other public services to other software layers. Drivers 1632 may be responsible for controlling or connecting to the underlying hardware. For example, depending on the hardware configuration, drivers 1632 may include display drivers, camera drivers, drivers, flash drives, serial communication drivers (e.g., universal serial bus (USB) drivers), Drivers, audio drivers, power management drivers, etc.

[0135] The library 1616 may provide a common infrastructure that can be used by the application 1620 or other components or layers. The function of the library 1616 is generally to enable other software modules to perform tasks in an easier manner than directly connecting to the underlying operating system 1614 function (e.g., kernel 1628, service 1630 or driver 1632). The library 1616 may include a system library 1634 (e.g., C standard library), which may provide functions such as memory allocation functions, string operation functions, mathematical functions, etc. In addition, the library 1616 may include an API library 1636, such as a media library (e.g., a library that supports the presentation and operation of various media formats (e.g., MPEG4, H.264, MP3, AAC, AMR, JPG, PNG)), a graphics library (e.g., an OpenGL framework that can be used to present 2D and 3D graphics content on a display), a database library (e.g., SQLite that can provide various relational database functions), a webpage library (e.g., WebKit that can provide web browsing functions), etc. The library 1616 may also include a variety of other libraries 1638 to provide many other APIs to the applications 1620 and other software components / modules.

[0136] Framework / middleware 1618 (sometimes also referred to as middleware) may provide a high-level common infrastructure that may be used by applications 1620 or other software components / modules. For example, framework / middleware 1618 may provide various graphical user interface (GUI) functions, advanced resource management, advanced location services, etc. Framework / middleware 1618 may provide a wide range of other APIs that may be used by applications 1620 or other software components / modules, some of which may be specific to a particular operating system 1614 or platform.

[0137] The application 1620 includes a built-in application 1640, a third-party application 1642, an LKAS module 1660, and a positioning module 1662. In some aspects, the LKAS module 1660 may include suitable circuits, logic, interfaces, or codes, and is configured to perform the same as Figure 6 The LKAS module 600 is associated and combined Figure 1-Figure 15 The positioning module 1662 may include suitable circuits, logic, interfaces or codes and be used to perform the Fig.12 The positioning module 1200 is associated and combined Figure 10-Figure 14 One or more of the vehicle positioning functions described.

[0138] Examples of representative built-in applications 1640 may include, but are not limited to, a contact application, a browser application, a reader application, a location application, a media application, a messaging application, or a game application. Third-party applications 1642 may include any of the built-in applications 1640 as well as a wide variety of other applications. In a specific example, third-party applications 1642 (e.g., applications developed by entities other than the vendor of a particular platform using Android TM or iOS TM Software development kit (SDK) applications can be developed on platforms such as iOS TM 、Android TM , Mobile software running on a mobile operating system such as Windows Phone or other mobile operating systems. In this example, third-party applications 1642 can call API calls 1624 provided by a mobile operating system (e.g., operating system 1614) to facilitate the functions described herein.

[0139] Applications 1620 may utilize built-in operating system functionality (e.g., kernel 1628, services 1630, and drivers 1632), libraries (e.g., system libraries 1634, API libraries 1636, and other libraries 1638), and framework / middleware 1618 to create a user interface to interact with a user of the system. Alternatively or additionally, in some systems, interaction with the user may occur through a presentation layer (e.g., presentation layer 1644). In these systems, the application / module "logic" may be separate from aspects of the application / module that interact with the user.

[0140] Some software architectures use virtual machines. Fig.16 In the example of FIG. 16 , a virtual machine is shown by virtual machine 1648. A virtual machine creates a software environment in which an application / module can be run as if it were on a hardware machine (e.g., Fig.17 The virtual machine 1648 is hosted by a host operating system (e.g., operating system 1614) and typically (although not always) has a virtual machine monitor 1646. The virtual machine monitor 1646 is used to manage the operation of the virtual machine 1648 and the connection with the host operating system (i.e., operating system 1614). The software architecture 1602 is executed in the virtual machine 1648, such as an operating system 1650, a library 1652, a framework / middleware 1654, an application 1656, or a presentation layer 1658. These software architecture layers executed in the virtual machine 1648 can be the same as the corresponding layers described above, or they can be different.

[0141] Fig.17is a block diagram of circuitry of devices implementing algorithms and execution methods provided by some example embodiments. Not all components need to be used in each embodiment. For example, a client, a server, and a cloud-based network device may each use a different set of components, or, in the case of a server, a larger storage device.

[0142] An example computing device in the form of a computer 1700 (also referred to as computing device 1700, computer system 1700, or computer 1700) may include a processor 1705, memory 1710, removable storage 1715, non-removable storage 1720, input interface 1725, output interface 1730, and communication interface 1735, all connected via a bus 1740. Although the example computing device is shown and described as being a computer 1700, the computing device may take different forms in different embodiments.

[0143] Memory 1710 may include volatile memory 1745 and nonvolatile memory 1750, and may store programs 1755. Computing device 1700 may include or have access to a computing environment that includes various computer-readable media, such as volatile memory 1745, nonvolatile memory 1750, removable memory 1715, and non-removable memory 1720. Computer memory includes random-access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), and electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD ROM), digital versatile disk (DVD) or other optical disk storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium capable of storing computer-readable instructions.

[0144] Computer-readable instructions stored in a computer-readable medium (e.g., a program 1755 stored in memory 1710) can be executed by a processor 1705 of the computing device 1700. Hard disk drives, CD-ROMs, and RAM are some examples of articles of manufacture that include non-transitory computer-readable media such as storage devices. The terms "computer-readable medium" and "storage device" do not include carrier waves that are considered too transient. "Computer-readable non-transitory medium" includes all types of computer-readable media, including magnetic storage media, optical storage media, flash memory media, and solid-state storage media. It should be understood that the software can be installed in a computer and sold with the computer. Alternatively, the software can be obtained and loaded into the computer, including obtaining the software through a physical medium or distribution system, including (for example) obtaining the software from a server owned by the software creator or from a server that the software creator does not own but uses. For example, the software can be stored in a server for distribution over the Internet. The terms "computer-readable medium" and "machine-readable medium" used herein are interchangeable.

[0145] The program 1755 may use a customer preference structure using the modules described herein, such as the LKAS module 1760 and the location module 1765, which may be respectively associated with Fig.16 The LKAS module 1660 and the positioning module 1662 are the same.

[0146] Any one or more modules described herein may be implemented using hardware (e.g., a processor of a machine, an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or any suitable combination thereof). In addition, any two or more of these modules may be combined into a single module, and the functionality of a single module described herein may be subdivided between multiple modules. In addition, according to various example embodiments, the modules described herein as being implemented within a single machine, database, or device may be distributed to multiple machines, databases, or devices.

[0147] In some aspects, the LKAS module 1760, the positioning module 1765, and one or more other modules as part of the program 1755 can be integrated into a single module to perform the corresponding functions of the integrated module.

[0148] Although several embodiments are described in detail above, other modifications may be made. For example, the logic flow described in the figure does not require the specific order or sequential order shown to achieve the desired result. Other steps may be provided to the described flow or steps may be deleted from the described flow, and other components may be added to the described system or components may be removed from the described system. Other embodiments may be within the scope of the appended claims.

[0149] In an example embodiment, the computer 1700 includes: an extraction module for extracting multiple features from multiple image frames obtained by a camera in a moving vehicle; a sequence module for generating a correlation sequence between multiple features; a reduction module for performing dimensionality reduction of the correlation sequence to generate a modified correlation sequence; a road parameter module for generating road parameter estimation data associated with the moving vehicle based on the modified correlation sequence; and a warning module for providing a lane keep assist system (LKAS) warning based on the road parameter estimation data. In some embodiments, the computer 1700 may include other modules or additional modules for performing any step or combination of steps described in the embodiments. In addition, any additional or alternative embodiments or aspects of the method as shown in any of the figures or described in any of the claims are also expected to include similar modules.

[0150] In an example embodiment, a computer system 1700 is provided to perform autonomous level functions associated with a moving vehicle. The computer system 1700 includes a memory 1715 storing instructions, and one or more processors 1705 in communication with the memory 1715. The one or more processors 1705 execute instructions to extract multiple features from multiple image frames obtained by a camera in a moving vehicle using a convolutional layer of a convolutional neural network; generate a correlation sequence between multiple features using a first fully connected (FC) layer of the convolutional neural network; perform dimensionality reduction of the correlation sequence using a spatial long short-term memory (LSTM) to generate a modified correlation sequence; generate road parameter estimation data associated with the moving vehicle based on the modified correlation sequence using a second FC layer of the convolutional neural network; and provide a lane keep assist system (LKAS) warning based on the road parameter estimation data.

[0151] It should also be understood that software including one or more computer executable instructions that facilitate the processes and operations described above in connection with any or all of the steps of the present invention may be installed in and sold with one or more computing devices consistent with the present invention. Alternatively, the software may be acquired and loaded into one or more computing devices, including acquiring the software via physical media or a distribution system, including, for example, acquiring the software from a server owned by the creator of the software or from a server not owned by the creator of the software but used by the creator of the software. For example, the software may be stored in a server for distribution via the Internet.

[0152] In addition, it will be appreciated by those skilled in the art that the present invention is not limited in its application to the construction and arrangement details of the components set forth in the specification or illustrated in the drawings. The embodiments herein can be applied to other embodiments and can be practiced or executed in various ways. In addition, it will be appreciated that the wording and terminology used herein are for descriptive purposes and should not be considered restrictive. The use of "include", "comprise" or "have" and its variants herein is intended to cover the items listed thereafter and their equivalents and other items. Unless otherwise limited, the terms "connect", "couple" and "install" and their variants are widely used herein and cover direct and indirect connections, couplings and installations. In addition, the terms "connect" and "couple" and their variants are not limited to physical or mechanical connections or couplings. In addition, terms such as "upper", "lower", "bottom" and "top" are relative and are used to help illustrate, but are not restrictive.

[0153] The components of the illustrative devices, systems, and methods used in accordance with the described embodiments may be implemented at least in part in digital electronic circuitry, analog electronic circuitry, or in computer hardware, firmware, software, or a combination thereof. For example, these components may be implemented as a computer program product (e.g., a computer program, program code, or computer instructions) tangibly embodied in an information carrier, or in a machine-readable storage device, for execution by a data processing apparatus (e.g., a programmable processor, a computer, or multiple computers), or for controlling the operation of a data processing apparatus.

[0154] The computer program can be written in any form of programming language (including compiled or interpreted languages) and can be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. The computer program can be deployed to be executed in one computer or in multiple computers at one site, or it can be distributed at multiple sites and interconnected through a communication network. In addition, the functional programs, codes, and code segments used to implement the technology described herein are easily understood by programmers in the field of the technology described herein as being within the scope of the claims. The method steps associated with the illustrative embodiments can be performed by one or more programmable processors, thereby executing computer programs, codes, or instructions to perform functions (e.g., operating on input data or generating output). For example, the method steps can also be performed by a dedicated logic circuit (e.g., a field programmable gate array (FPGA) or an application-specific integrated circuit (ASIC)), and the device for performing the method can be implemented as the dedicated logic circuit.

[0155] The various illustrative logical blocks, modules, and circuits described in conjunction with the embodiments disclosed herein may be implemented or executed using a general purpose processor, a digital signal processor (DSP), an ASIC, an FPGA, or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination thereof designed to perform the functions described herein. A general purpose processor may be a microprocessor, and optionally, the processor may also be any conventional processor, controller, microcontroller, or state machine. A processor may also be implemented as a combination of computing devices, such as a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors in combination with a DSP core, or any other similarly configured combination.

[0156] For example, processors suitable for executing computer programs include general and special microprocessors, and any one or more processors of any type of digital computer. Typically, the processor receives instructions and data from read-only memory or random access memory or both. The essential elements of a computer are a processor for executing instructions and one or more memory devices for storing instructions and data. Typically, the computer also includes one or more mass storage devices (such as magnetic disks, magneto-optical disks, or optical disks) for storing data, or is operably coupled to one or more mass storage devices for storing data to receive data from the mass storage device and / or transmit data to the mass storage device. Information carriers suitable for embodying computer program instructions and data include various forms of non-volatile memory, for example, semiconductor memory devices, such as electrically programmable read-only memory or electrically programmable ROM (electrically programmable read-only memory, EPROM), electrically erasable programmable ROM (electrically erasable programmable ROM, EEPROM), flash memory devices, or data storage disks (e.g., magnetic disks, built-in hard disks, removable disks, magneto-optical disks, or CD-ROM / DVD-ROM disks). The processor and memory can be supplemented by or incorporated into a dedicated logic circuit.

[0157] Those skilled in the art will appreciate that any of a variety of different technologies and technical methods may be used to represent information and signals. For example, data, instructions, commands, information, signals, bits, symbols, and chips that may be cited in the above description may be represented by voltage, current, electromagnetic waves, magnetic fields or magnetic particles, optical fields or optical particles, or any combination thereof.

[0158] As used herein, "machine-readable media" (or "computer-readable media") includes devices capable of temporarily or permanently storing instructions and data, and may include, but are not limited to, random-access memory (RAM), read-only memory (ROM), buffer memory, flash memory, optical media, magnetic media, cache memory, other types of memory (e.g., erasable programmable read-only memory (EEPROM)), or any suitable combination thereof. The term "machine-readable medium" should be understood to include a single medium or multiple media (e.g., a centralized or distributed database, or associated caches and servers) that can store processor instructions. The term "machine-readable medium" should also be understood to include any medium or combination of multiple media that can store instructions executed by one or more processors, and when one or more processors execute the instructions, one or more processors perform any one or more of the methods described herein. Accordingly, "machine-readable medium" refers to a single storage device or device, as well as a "cloud-based" storage system or storage network that includes multiple storage devices or devices. The term "machine-readable medium" as used herein does not include the signal itself.

[0159] In addition, without departing from the scope of the present invention, the techniques, systems, subsystems and methods described and illustrated as discrete or separate in various embodiments may be combined or integrated with other systems, modules, techniques or methods. Other items shown or described as coupled to each other, or directly coupled, or communicating with each other may be indirectly coupled or communicated electrically, mechanically or otherwise through some interface, device or intermediate component. Other examples of changes, substitutions and alterations may be determined by those skilled in the art and may be exemplified without departing from the scope disclosed herein.

[0160] Although the present invention has been described with reference to specific features and embodiments of the present invention, it is apparent that various modifications and combinations of the present invention may be formulated without departing from the present invention. For example, other components may be added to or removed from the described system. Therefore, the specification and the accompanying drawings are only considered as an illustration of the present invention as defined by the appended claims and any and all modifications, variants, combinations or equivalents falling within the scope of the present invention are contemplated. Other aspects may be within the scope of the appended claims. Finally, unless otherwise specifically stated, the conjunction "or" used herein refers to a non-exclusive "or".

Claims

1. A computer-implemented method for performing autonomous level functions associated with a mobile vehicle, characterized in that The method comprises: extracting a plurality of features from a plurality of image frames obtained by a camera within the moving vehicle using a convolutional layer of a convolutional neural network; Generate a correlation sequence between the multiple features using a first fully connected (FC) layer of the convolutional neural network; performing dimensionality reduction of the correlation sequence using a spatial long-short-term memory (LSTM) to generate a modified correlation sequence; generating, using a second FC layer of the convolutional neural network, road parameter estimation data associated with the moving vehicle based on the modified correlation sequence; providing a lane keep assist system (LKAS) warning based on the road parameter estimation data; The method further comprises: detecting one or more time series data patterns in the road parameter estimation data using a temporal LSTM; Using the temporal LSTM, the road parameter estimate data is modified according to the detected one or more patterns of the time series data to generate modified road parameter estimate data.

2. The computer-implemented method of claim 1, wherein: The modified road parameter estimation data includes lane parameter information and lane context information.

3. The computer-implemented method of claim 2, wherein: The lane parameter information includes one or more of the following: Lane marking heading angle; Lane marking offset; Lane marking curvature; Lane marking curvature derivative; The lane marking type.

4. The computer-implemented method of claim 2, wherein: The lane context information includes one or more of the following: a relative road distance, representing a distance traveled by the mobile vehicle between two consecutive frames of the plurality of image frames; a lane number indication of the road lane on which the mobile vehicle is traveling; The lane number associated with the road on which the mobile vehicle is traveling.

5. The computer-implemented method according to any one of claims 1 to 4, characterized in that: The method also includes: using a convolutional layer of a second convolutional neural network to extract geographic location information of the moving vehicle using the multiple image frames obtained by the camera.

6. The computer-implemented method of claim 5, wherein: The method further comprises: applying one or more spatial constraints to the geographic location information using Bayesian filtering to generate updated geographic location information for the mobile vehicle, the one or more spatial constraints being based on the road parameter estimate data; The updated geographic location information of the moving vehicle is output.

7. A system for performing autonomous level functions associated with a moving vehicle, characterized in that The system comprises: a memory for storing instructions; one or more processors in communication with the memory, the one or more processors executing the instructions to: extracting a plurality of features from a plurality of image frames obtained by a camera within the moving vehicle using a convolutional layer of a convolutional neural network; Generate a correlation sequence between the multiple features using a first fully connected (FC) layer of the convolutional neural network; performing dimensionality reduction of the correlation sequence using a spatial long-short-term memory (LSTM) to generate a modified correlation sequence; generating, using a second FC layer of the convolutional neural network, road parameter estimation data associated with the moving vehicle based on the modified correlation sequence; providing a lane keep assist system (LKAS) warning based on the road parameter estimation data; The one or more processors also execute the instructions to: detecting one or more time series data patterns in the road parameter estimation data using a temporal LSTM; Using the temporal LSTM, the road parameter estimate data is modified according to the detected one or more patterns of the time series data to generate modified road parameter estimate data.

8. The system according to claim 7, characterized in that The modified road parameter estimation data includes lane parameter information and lane context information.

9. The system according to claim 8, characterized in that The lane parameter information includes one or more of the following: lane marking heading angle; Lane marking offset; Lane marking curvature; Lane marking curvature derivative; The lane marking type.

10. The system according to claim 8, characterized in that The lane context information includes one or more of the following: a relative road distance, representing a distance traveled by the mobile vehicle between two consecutive frames of the plurality of image frames; a lane number indication of the road lane on which the mobile vehicle is traveling; The lane number associated with the road on which the mobile vehicle is traveling.

11. The system according to any one of claims 7 to 10, characterized in that The one or more processors also execute the instructions to: Using a convolutional layer of a second convolutional neural network, the geographic location information of the moving vehicle is extracted using the multiple image frames obtained by the camera.

12. The system according to claim 11, characterized in that The one or more processors also execute the instructions to: applying one or more spatial constraints to the geographic location information using Bayesian filtering to generate updated geographic location information for the mobile vehicle, the one or more spatial constraints being based on the road parameter estimate data; The updated geographic location information of the moving vehicle is output.

13. A computer readable medium, characterized in that storing computer instructions for executing autonomous level functions associated with a mobile vehicle, wherein the instructions, when executed by one or more processors of a computing device, cause the one or more processors to perform operations including: extracting a plurality of features from a plurality of image frames obtained by a camera within the moving vehicle using a convolutional layer of a convolutional neural network; Generate a correlation sequence between the multiple features using a first fully connected (FC) layer of the convolutional neural network; performing dimensionality reduction of the correlation sequence using a spatial long-short-term memory (LSTM) to generate a modified correlation sequence; generating, using a second FC layer of the convolutional neural network, road parameter estimation data associated with the moving vehicle based on the modified correlation sequence; providing a lane keep assist system (LKAS) warning based on the road parameter estimation data; The instructions further cause the one or more processors to: detecting one or more time series data patterns in the road parameter estimation data using a temporal LSTM; Using the temporal LSTM, the road parameter estimate data is modified according to the detected one or more patterns of the time series data to generate modified road parameter estimate data.

14. The computer-readable medium of claim 13, wherein: The modified road parameter estimation data includes lane parameter information and lane context information.

15. The computer-readable medium of claim 14, wherein: The lane parameter information includes one or more of the following: Lane marking heading angle; Lane marking offset; Lane marking curvature; Lane marking curvature derivative; The lane marking type.

16. The computer-readable medium of claim 14, wherein: The lane context information includes one or more of the following: a relative road distance, representing a distance traveled by the mobile vehicle between two consecutive frames of the plurality of image frames; a lane number indication of the road lane on which the mobile vehicle is traveling; The lane number associated with the road on which the mobile vehicle is traveling.

17. The computer-readable medium according to any one of claims 13 to 16, wherein: The instructions further cause the one or more processors to: Using a convolutional layer of a second convolutional neural network, the geographic location information of the moving vehicle is extracted using the multiple image frames obtained by the camera.

18. The computer-readable medium of claim 17, wherein: The instructions further cause the one or more processors to: applying one or more spatial constraints to the geographic location information using Bayesian filtering to generate updated geographic location information for the mobile vehicle, the one or more spatial constraints being based on the road parameter estimate data; The updated geographic location information of the moving vehicle is output.

19. A system for performing autonomous level functions associated with a mobile vehicle, characterized in that The system comprises: an extraction module for extracting a plurality of features from a plurality of image frames obtained by a camera in the mobile vehicle; A correlation module, used to generate a correlation sequence between the multiple features; a dimensionality reduction module for performing dimensionality reduction of the correlation sequence to generate a modified correlation sequence; an estimation module for generating road parameter estimation data associated with the moving vehicle based on the modified correlation sequence; a notification module, configured to provide a lane keep assist system (LKAS) warning based on the road parameter estimation data; The system further comprises: means for detecting one or more time series data patterns in the road parameter estimate data; Means for modifying the road parameter estimate data based on the detected one or more time series data patterns to generate modified road parameter estimate data.

Images