Vehicle control systems for camera-based vehicle navigation

Abstract

A vehicle control system for camera-based vehicle navigation includes at least one vehicle camera configured to capture an image of a forward view of a vehicle, a global positioning system (GPS) receiver configured to receive a current location of the vehicle, a vehicle user interface including a display, and a vehicle control module.The vehicle control module is configured to obtain the current location of the vehicle via the GPS receiver, identify a sequence of vehicle navigation steps to a destination, capture the image via the at least one vehicle camera, process the image with a machine learning model to detect a plurality of objects in the image, rank the plurality of objects according to landmark evaluation criteria that indicate a probability of object detection by a driver of the vehicle, and display a highest ranked one of the plurality of objects in association with a next vehicle navigation step.

Description

introduction

[0001] The present invention relates generally to a vehicle control system according to the preamble of claim 1 and a method according to the preamble of claim 8 for camera-based vehicle navigation, as essentially known from US 2021 / 0 247 201 A1.

[0002] US 2017 / 0 314 954 A1 describes a system that is essentially comparable in nature.

[0003] Further prior art can be found in US 2023 / 0 137 744 A1.

[0004] Vehicle navigation systems generate a sequence of turns for a vehicle to travel from a current location to a destination. The sequence of turns can be displayed on a user interface to alert a driver to upcoming turns based on the vehicle's current location, as determined by a GPS system. The upcoming turns are generally associated with the name of a street on which the vehicle should make the next left or right turn. Summary

[0005] According to the invention, a vehicle control system for camera-based vehicle navigation is presented, which is characterized by the features of claim 1.

[0006] In other features, the vehicle control module is configured to record a turn action of the vehicle, compare a location of the turn action to the next vehicle navigation step to determine a turn conformity score indicating whether the highest ranked one of the plurality of objects was an accurate guidance landmark, and update the machine learning model via supervised learning according to the turn conformity score.

[0007] In other features, the vehicle control module is configured to obtain a distance between the current location of the vehicle and the next vehicle navigation step and process the image with a depth estimation model to generate a region of interest within the image, wherein the distance between the current location of the vehicle and the next vehicle navigation step is within the region of interest of the image.

[0008] In other features, the vehicle control module is configured to detect the multiple objects in the image only within the area of ​​interest.

[0009] In other features, the vehicle control module is configured to access a database to obtain multiple points of interest corresponding to the current location of the vehicle and detect objects corresponding to the points of interest within the area of ​​interest of the image.

[0010] In other features, the vehicle control module is configured to obtain an associated popularity score from the database for each of the plurality of points of interest, the associated popularity score indicating a degree of recognition of the point of interest, and provide each associated popularity score to the machine learning model to facilitate ranking the objects corresponding to the points of interest.

[0011] In other features, the vehicle control module is configured to divide the image into multiple segments via the machine learning model, crop a bounding box for each of the multiple segments, and transform each of the multiple segments into a text output via a large machine learning language model.

[0012] In other features, the vehicle control module is configured to generate a histogram of a plurality of terms according to the text output corresponding to each of the plurality of segments and to select one of the plurality of terms with the lowest frequency for display on the vehicle user interface in association with a next one of the sequence of vehicle navigation steps.

[0013] In other features, the vehicle control module is configured to perform optical character recognition for each of the plurality of objects to identify text associated with the object, compare the object to a database of stored logo data to determine if the object has a matching logo, and obtain a confidence score from the machine learning model indicating a detection confidence for the object.

[0014] In other features, the vehicle control module is configured to generate a landmark score for each of the plurality of objects as a combination of a visibility score for the object, an intuitiveness score for the object, and a uniqueness score for the object.

[0015] According to the invention, a method for camera-based vehicle navigation is further presented, which is characterized by the features of claim 8.

[0016] In other features, the method includes recording a turn action of the vehicle, comparing a location of the turn action to the next vehicle navigation step to determine a turn conformity score indicating whether the highest ranked one of the plurality of objects was an accurate guidance landmark, and updating the machine learning model through supervised learning according to the turn conformity score.

[0017] In other features, the method includes obtaining a distance between the current location of the vehicle and the next vehicle navigation step and processing the image with a depth estimation model to generate a region of interest within the image, wherein the distance between the current location of the vehicle and the next vehicle navigation step is within the region of interest of the image.

[0018] In other features, detecting the multiple objects includes detecting the multiple objects in the image only within the region of interest.

[0019] In other features, the method includes accessing a database to obtain a plurality of points of interest corresponding to the current location of the vehicle, and detecting objects corresponding to the points of interest within the region of interest of the image.

[0020] In other features, the method includes, for each of the plurality of points of interest, obtaining an associated popularity score from the database, the associated popularity score indicating a degree of recognition of the point of interest, and providing each associated popularity score to the machine learning model to facilitate ranking the objects corresponding to the points of interest.

[0021] Further areas of applicability of the present invention will become apparent from the detailed description, claims, and drawings. The detailed description and specific examples are intended for illustrative purposes only. Brief description of the drawings

[0022] The present invention will be more fully understood from the detailed description and the accompanying drawings, in which: Fig. 1 is a functional block diagram of an exemplary embodiment of a vehicle control system for camera-based vehicle navigation; Fig. 2 is a flowchart illustrating an exemplary process for camera-based vehicle navigation, including image object detection for context-aware vehicle navigation guidance; Fig. 3 is a flowchart illustrating an exemplary process for using a depth estimation neural network to extract a relevant region of interest in a vehicle front camera image; Fig. 4 is a flowchart illustrating an exemplary process for detecting objects in a vehicle front camera image; Fig. 5 is a flowchart illustrating an exemplary process for classifying detected objects in a vehicle front camera image; Fig. 6 is a flowchart illustrating an exemplary process for identifying patches in an image region of interest with an object probability above a specified threshold; Fig. 7 is a diagram illustrating an exemplary vehicle user interface displaying guidance for contextual vehicle navigation; Fig. 8A and Fig. 8B are graphical representations of exemplary recurrent neural networks for processing images from the vehicle's front camera; Fig. 9 is a graphical representation of layers of an exemplary long-term memory machine learning (LSTM) model; and Fig. 10 is a flowchart illustrating an example process for training a machine learning model.

[0023] Reference numbers may be reused in the drawings to identify similar and / or identical elements. Detailed description

[0024] Turn-by-turn navigation systems guide or direct the driver to a desired destination by providing a simple set of instructions that usually include information about whether to turn right or left, a street name for the next turn, and a distance to the next turn.

[0025] In some example embodiments, a computer vision-based approach assists users in correctly implementing a navigation instruction by associating the navigation instruction with spatial visual cues. The cues may be visual landmarks detected in a scene ahead of the vehicle, such as by leveraging CV techniques with vehicle-mounted cameras (e.g., "turn right after the [national fast-food chain] sign").

[0026] Machine learning techniques can be used to determine which objects are the most visible and intuitive visual landmarks to associate with a navigation instruction. Additionally, some implementations can help drivers better identify a destination by associating it with distinguishable elements currently identified in the scene (e.g., "Your destination is next to the [national home improvement chain] truck"). Example systems can be configured to use feedback from driver turn actions to improve models for identifying the best landmark objects for contextual vehicle navigation guidance.

[0027] Referring to Fig. 1, a vehicle 10 comprises front wheels 12 and rear wheels 13. In Fig. 1, a drive unit 14 selectively delivers torque to the front wheels 12 and / or the rear wheels 13 via drivetrains 16 and 18, respectively. The vehicle 10 may include various types of drive units. For example, the vehicle may be an electric vehicle such as a battery electric vehicle (BEV), a hybrid vehicle or a fuel cell vehicle, a vehicle including an internal combustion engine (ICE), or another type of vehicle.

[0028] Some examples of the drive unit 14 may include any suitable electric motor, a power inverter, and a motor controller configured to control power switches within the power inverter to adjust motor speed and torque during propulsion and / or regeneration. A battery system provides power to the electric motor of the drive unit 14 via the power inverter during propulsion or receives power from it during regeneration.

[0029] While the vehicle 10 in Fig. 1 has a drive unit 14, the vehicle 10 may have other configurations. For example, two separate drive units may drive the front wheels 12 and the rear wheels 13, one or more separate drive units may drive individual wheels, etc. As can be understood, other vehicle configurations and / or drive units may be used.

[0030] The vehicle control module 20 may be configured to control the operation of one or more vehicle components, such as the drive unit 14 (e.g., by commanding torque settings of an electric motor of the drive unit 14). The vehicle control module 20 may receive inputs to control components of the vehicle, such as signals received from a steering wheel, an accelerator pedal, etc. The vehicle control module 20 may monitor vehicle telematics for safety purposes, such as vehicle speed, vehicle location, vehicle braking and acceleration, etc.

[0031] The vehicle control module 20 may receive signals from any suitable components for monitoring one or more aspects of the vehicle, including one or more vehicle sensors (such as cameras, microphones, pressure sensors, wheel position sensors, location sensors such as global positioning system (GPS) antennas, etc.). Some sensors may be configured to monitor the current motion of the vehicle, vehicle acceleration, steering torque, etc.

[0032] As in Fig. 1, the vehicle 10 includes a front-facing vehicle camera 24 and an optional side-facing vehicle camera 26. In various implementations, the vehicle 10 may include more or fewer of one or more of these vehicle cameras (such as a rear-facing vehicle camera, etc.). Each vehicle camera may include any suitable camera lens, laser, lidar sensor, etc., used to capture images around the vehicle 10 (such as a front-facing vehicle camera image depicting a scene in front of the vehicle, etc.). In some example embodiments, a vehicle object detector may be configured to detect a closest in-path vehicle (CIPV) (e.g., another vehicle in front of a current path of travel of the vehicle 10), a vulnerable road user (VRU) (e.g., a pedestrian or cyclist), etc.

[0033] The vehicle control module 20 may communicate with another device or apparatus via a wireless communication interface, which may include one or more wireless antennas for transmitting and / or receiving wireless communication signals. For example, the wireless communication interface may communicate via any suitable wireless communication protocol, including, but not limited to, vehicle-to-everything (V2X) communication, Wi-Fi communication, wireless area network (WAN) communication, cellular communication, personal area network (PAN) communication, short-range wireless communication (e.g., Bluetooth), etc.The wireless communication interface may communicate with a remote computing device over one or more wireless and / or wired networks. With regard to vehicle-to-vehicle (V2X) communication, the vehicle 10 may include one or more V2X transceivers (e.g., V2X signal transmitting and / or receiving antennas).

[0034] The vehicle 10 also includes a user interface 22. The user interface 22 may include any suitable display (such as on a dashboard, console, or elsewhere), a touchscreen or other input device, speakers for generating audio signals, etc. In some example embodiments, the vehicle control module 20 may be configured to display vehicle navigation guidance on the user interface 22.

[0035] Fig. 2 is a flowchart illustrating an exemplary process for camera-based vehicle navigation, including image object detection for context-aware vehicle navigation guidance. The process may be implemented using, for example, the vehicle control module 20 of Fig. 1. At 204, the method begins by obtaining vehicle navigation instructions to a destination, such as a sequence of turns and distances between turns, for the vehicle 10 to travel from a current location to the destination as determined by a GPS vehicle navigation system.

[0036] The vehicle control module is configured to receive an image from the vehicle's front camera, such as the front camera 24 of the vehicle 10, at 208 Fig. 1. Although some exemplary embodiments refer to images captured by a front-facing camera of the vehicle, in some implementations, images may be captured using a side-facing camera of the vehicle, a combination of one or more front-facing and side-facing cameras of the vehicle, etc. The vehicle control module then uses a depth estimate at 212 to extract a relevant region of interest (ROI) based on the navigation instructions. Further details on the use of the depth estimate are described below with reference to Fig. 3 described.

[0037] The vehicle control module is configured to detect objects detected in the extracted region of interest at 216. Further details regarding the detection of objects in the ROI are provided below with reference to Fig. 4 described.

[0038] The control module is configured to retrieve 220 recognized landmarks at the vehicle's current location from a database. For example, one or more databases may be configured to store various known landmarks at different locations. The vehicle control module may receive a current location of the vehicle and then access stored recognizable landmarks at the vehicle's current location from the database(s).

[0039] The vehicle control module is configured to receive 224 points of interest (POIs) from a vehicle location map. For example, a vehicle navigation system may have stored points of interest associated with different locations, and the vehicle control module may receive the points of interest corresponding to the vehicle's current location.

[0040] The vehicle control module is configured to receive, at 228, popularity ratings of points of interest based on external metrics. For example, various ranking systems, databases, etc., may store scores indicating which points of interest are more popular with drivers, have higher ratings or reviews for drivers, are visited more frequently by drivers, etc.

[0041] The vehicle control module is configured to rank the detected objects at 232 using image processing and additional metrics. For example, the vehicle control module may be configured to determine which objects detected in the vehicle's front-facing camera image are most likely to provide a most recognizable or identifiable landmark for displaying contextual driving navigation to the driver. Further details regarding an exemplary ranking algorithm for detected objects are provided below with reference to Fig. 5 described.

[0042] The vehicle control module is configured to determine at 236 whether any relevant objects above a predetermined threshold have been detected. A predetermined threshold for classification may be defined, for example, based on a landmark rating determined for each object, such as object visibility, object detectability, etc., that indicates whether the object in the image would be a good candidate for display to a driver for contextual navigation guidance. If a detected object has a landmark rating above a predetermined threshold, control proceeds 248 to display the object, or one or more highest-ranked objects of a plurality of objects above the threshold, on a vehicle user interface to provide contextual driving navigation to the driver.

[0043] If the controller determines at 236 that no detected objects have a landmark or relevance score above a specified threshold, the controller proceeds to 240 to identify locations in a region of interest of the image that have a high object probability. For example, if it is detected that no objects have a sufficient match to known points of interest or known recognized landmarks, such as fast-food restaurants or other widely known businesses, etc., the controller may perform image processing to identify common objects in the region of interest of the image. Further details regarding identifying locations in the image are described below with reference to Fig. 6 described.

[0044] The vehicle control module is configured to generate image captions for identified probable objects at 244. For example, a text transformation model can be applied to generate text descriptions such as car, tree, sign, etc. for portions of the image corresponding to the detected objects. The control then displays the highest-ranked objects and / or associated captions at 248.

[0045] The vehicle control module is configured to determine a driver action related to the displayed objects at 252. For example, the controller may determine whether the driver followed a suggested turn prompt in contextual driving navigation, whether the driver passed the identified landmark or turned too early, etc. The controller then provides feedback to update the object classification algorithm at 256 based on the driver action.

[0046] In various implementations, each vehicle navigation instruction is followed by a driver action. If the driver performs an incorrect action (e.g., turns incorrectly at the correct location), the system can use the information to improve a landmark evaluation algorithm, such as by updating the weights of a landmark selection model through supervised learning. This allows the algorithm to learn and continuously update the algorithm to identify which landmarks are more likely to be recognized and followed by drivers.

[0047] In some example embodiments, personal information learned from the driver's previous navigation destinations may be used to update a landmark scoring model for the driver. For example, if there is an object associated with a location the driver has frequently visited, the object may receive a higher score as a candidate landmark compared to locations the driver has visited less frequently.

[0048] Fig. 3 is a flowchart illustrating an example process for using a depth estimation neural network to extract a relevant region of interest from a vehicle's front-facing camera image. The vehicle control module is configured to receive an image from the vehicle's front-facing camera at 304.

[0049] The vehicle control module then performs a monocular depth estimation on the obtained image, such as via a trained neural network. The neural network may, for example, identify locations in the image that are estimated to be within a specified distance range from a current location of the vehicle, such as between 75 meters and 125 meters in front of the vehicle. This may be defined by focal points, such as identifying points in the image that lie between a first focal point f1 and a second focal point f2. Further details regarding example machine learning models are provided below with reference to Fig. 8-10 described.

[0050] The controller is configured to receive a next vehicle navigation action and a current vehicle location at 312. For example, the controller may identify a next left or right turn in the sequence of navigation steps to the destination and also receive a current location of the vehicle via a GPS system.

[0051] The vehicle control module is configured to determine the distance to the next navigation action based on the current vehicle position at 316. For example, the controller may receive a distance to a road where the next left or right turn occurs. At 320, the controller may generate a window in the image corresponding to the location of the next turn, such as the window 50 meters away, 100 meters away, 500 meters ahead, etc.

[0052] For example, based on the monocular depth estimate and the distance to the next navigation action, the controller may perform depth masking of the obtained image, such as by excluding areas of the image having a depth "D" greater than a first distance or focal point f1 corresponding to a distance past the location of the next turn, and excluding areas of the image having a depth "D" less than a second distance or focal point f2 before the location of the next turn.

[0053] Fig. 4 is a flowchart illustrating an example process for detecting objects in a vehicle's front-facing camera image. The vehicle control module is configured to obtain a bounding box for a region of interest in an image captured by a vehicle's front-facing camera at 404. The controller then identifies segmented objects in the region of interest using a machine learning model at 408. Further details for example machine learning models are provided below with reference to Fig. 8-10 described.

[0054] At 412, the controller compares the objects in the area of ​​interest to logos stored in one or more databases. For example, the controller may determine whether any of the particular objects are buildings or vehicles with logos corresponding to known companies that would be identifiable to the driver.

[0055] The vehicle control module is configured to store matching logos identified in the region of interest of the image at 416. Control proceeds to 420 to perform optical character recognition (OCR) on objects identified in the images of interest. For example, control may perform the OCR process to determine text associated with the detected objects to identify landmark names.

[0056] At 424, the controller provides the segmented objects, the matching logos, and the OCR results to an object classification algorithm. The object classification algorithm may use the object segmentation, the matching logos, the OCR results, etc., to determine which objects in the vehicle's front-facing camera image may provide the best landmark for displaying contextual vehicle navigation guidance to the driver.

[0057] Fig. Figure 5 is a flowchart illustrating an example process for ranking detected objects in a vehicle's front-facing camera image. For example, the vehicle control module may select a best visual landmark from multiple candidate objects to provide context-aware vehicle navigation guidance to the driver. An example algorithm for evaluating and ranking multiple detected candidate objects is described below.

[0058] The vehicle control module is configured to receive, at 504, a detection confidence score and other data for each detected object in the region of interest of the image. For example, the object detection model may provide a confidence score for each detected object, which is an indicator of the probability that the model correctly identified the detected object. The controller may also receive any matching logos, OCR data, etc., that match each detected object.

[0059] At 508, the controller processes the image to generate a pixel saliency map. Salient objects are, for example, objects that are more noticeable or conspicuous to the driver according to their visual or optical characteristics. The visibility of an object can be determined largely according to the object's saliency level. If an object, such as a crosswalk sign, has a higher saliency than a parked black car (e.g., as identified in an image processing saliency map in which brighter areas are more conspicuous), a contextual navigation indicator may indicate "Turn right before the crosswalk sign" as opposed to "Turn just before the black parked car."

[0060] In some example embodiments, a saliency map may be generated by combining a set of known salient features. For example, a saliency map may be generated by combining separate image processing maps configured to emphasize red color in the image, emphasize blue color, emphasize intensity levels in the image, and emphasize Gabor levels (e.g., detection of straight lines in the image), etc.

[0061] The vehicle control module is configured to assign a visibility score to each object at 512 based on the saliency map, the detection confidence, and the object occlusion. The visibility score may indicate the object's ability to attract the driver's attention. For example, the controller may use the saliency map, the confidence scores for detected objects, an indication of whether an object is partially or fully occluded from the driver's view, etc., to determine how visible the object will be to the driver while traveling on the road.

[0062] The vehicle control module is configured to receive text and logo data associated with the detected objects at 516. Control then proceeds 520 to assign an intuitiveness score to each object based on the text and / or logo data associated with the object. The intuitiveness score may indicate whether there are elements in the detected object that facilitate understanding by the driver, such as text, familiar logos, traffic lights, road signs, etc. For example, if a familiar logo or text is associated with or associated with an object, it may receive a higher score because it is intuitively recognizable to the driver.

[0063] The vehicle control module is configured to determine, at 524, a relative occurrence of each object in an image ROI compared to objects in the entire image. The controller may determine whether there are multiple versions of a similar type in the image or whether an object is unique compared to other object types in the scene.

[0064] At 528, the controller assigns a uniqueness score to each object based on the determined relative occurrence in the image. The uniqueness score may be an indicator of whether the object type (e.g., a black car) occurs elsewhere in the scene. For example, if a building type is identified as the only building in the scene, a high uniqueness score may be assigned compared to a detected black car when there are multiple other black cars in the scene.

[0065] At 532, the controller assigns a landmark score to each object. The landmark score may be a weighted combination of the visibility score, the uniqueness score, and the intuitiveness score previously determined for an object. The landmark score may be an indicator of the likelihood that the driver will be able to clearly and quickly identify the object as a landmark for use in contextual vehicle navigation guidance.

[0066] Fig. 6 is a flowchart illustrating an exemplary process for identifying locations within an image region of interest with an object probability above a specified threshold. The vehicle control module is configured to divide the image region of interest into segments at 604. The controller then crops a bounding box for each segment within the image region of interest at 608.

[0067] At 612, the controller transforms each image segment into text using a large language model (e.g., a CLIP model). For example, a machine learning module may be configured to determine text words corresponding to common objects such as a tree, sidewalk, road, no-right-turn sign, wall, post, etc. Text embeddings may be obtained by processing a specified number of the most common nouns in English (or another desired language).

[0068] The vehicle control module is configured to generate a histogram of nouns in the scene at 616 and normalize each category based on its area in the image. The controller can remove non-informative objects (e.g., "road") to focus the list on unique objects in the scene that the driver would be able to more easily identify.

[0069] The controller selects an object with a smallest drop count and a largest area at 620. For example, the controller may identify an object with a large perimeter in the image if the object is also the only one of its kind in the image (e.g., the only traffic sign). Selecting a large and unique object makes it easier for the driver to recognize the selected object to provide better context-aware navigation guidance. The controller then provides the identified object for display on a vehicle user interface at 624 so that the driver knows which landmark object to look for in order to know when to make the next turn.

[0070] Fig. 7 is a diagram illustrating an example of a vehicle user interface 700 for displaying guidance for context-based vehicle navigation. As in Fig. 7, the vehicle user interface 700 displays contextual navigation guidance 706 that includes a right turn arrow indicator to identify a landmark object 702, such as a fast food restaurant, for performing a next navigation turn (e.g., to reach a destination).

[0071] The textual navigation guidance may include a textual prompt 704 that alerts the driver to the upcoming landmark object 702 for the next navigation turn, such as the statement "turn right after the fast food restaurant" (e.g., when a fast food restaurant with a familiar logo is identified as the landmark object 702, etc.).

[0072] Fig. 8A and Fig. 8B shows an example of a recurrent neural network used to generate models such as those described above using machine learning techniques. Machine learning is a process used to design and develop complex models and algorithms capable of prediction (for example, patient-provider match predictions). Models generated using machine learning, such as those described above, can produce reliable, repeatable decisions and outcomes and uncover hidden insights by learning from historical relationships and trends in the data.

[0073] The purpose of using a recurrent neural network-based model and training the model using machine learning as described above can be to directly predict dependent variables without casting relationships between the variables into a mathematical form. A neural network model contains a large number of virtual neurons operating in parallel and arranged in layers. The first layer is the input layer and receives unprocessed, or raw, input data. Each subsequent layer modifies the output of a previous layer and sends it to the next layer. The final layer is the output layer and generates the system's output.

[0074] Fig. Figure 8A shows a fully connected neural network, where every neuron in a given layer is connected to every neuron in the next layer. In the input layer, each input node is assigned a numerical value, which can be any real number. In each layer, each connection emanating from an input node has an associated weight, which can also be any real number (see Fig. 8B). In the input layer, the number of neurons is equal to the number of features (columns) in a dataset. The output layer can have multiple continuous outputs.

[0075] The layers between the input and output layers are hidden layers. The number of hidden layers can be one or more (for most applications, one hidden layer may be sufficient). A neural network without hidden layers can represent linear separable functions or decision functions. A neural network with one hidden layer can perform a continuous mapping from one finite space to another. A neural network with two hidden layers can approximate any smooth mapping to arbitrary accuracy.

[0076] The number of neurons can be optimized. At the beginning of training, a network configuration is more likely to have too many nodes. Some nodes can be removed from the network during training, which does not noticeably affect the network's performance. For example, nodes with weights approaching zero after training can be removed (a process called pruning). The number of neurons can cause underfitting (inability to adequately capture signals in the dataset) or overfitting (insufficient information to train all neurons; the network performs well on the training set but not on the test set).

[0077] Various methods and criteria can be used to measure the performance of a neural network model. For example, the root mean squared error (RMSE) measures the average distance between observed values ​​and model predictions. A coefficient of determination (R2) measures the correlation (not the accuracy) between observed and predicted outcomes. This method is not necessarily reliable if the data has a large variance. Other measures of performance include irreducible noise, model bias, and model variance. High model bias for a model is an indicator that the model is unable to capture a true relationship between the predictors and the outcome. Model variance can be an indicator of whether a model is stable (a small perturbation in the data will significantly change the model fit).The neural network can receive inputs, such as vectors, that can be used to generate models that can be used with provider matching, risk model processing, or both, as described here.

[0078] Fig. 9 illustrates an example long short-term memory (LSTM) neural network 902 used to generate models such as those described above using machine learning techniques, although other example embodiments may include other types of machine learning models, including transformer layers, other model topologies, etc. The generic example LSTM neural network 902 may be used to implement a machine learning model, and various implementations may utilize other types of machine learning networks (such as transformer layers, other model topologies or architectures, etc.). The LSTM neural network 902 includes an input layer 904, a hidden layer 908, and an output layer 912. The input layer 904 includes inputs 904a, 904b... 904n. The hidden layer 908 contains neurons 908a, 908b... 908n.The output layer 912 includes the outputs 912a, 912b... 912n.

[0079] Each neuron of hidden layer 908 receives an input from input layer 904 and outputs a value to the corresponding output in output layer 912. For example, neuron 908a receives an input from input 904a and outputs a value to output 912a. Each neuron, except neuron 908a, also receives an output from a previous neuron as input. For example, neuron 908b receives inputs from input 904b and output 912a. In this way, the output of each neuron is passed to the next neuron in hidden layer 908. The last output 912n in output layer 912 outputs a probability associated with inputs 904a-904n. Although input layer 904, hidden layer 908, and output layer 912 are each depicted as containing three elements, each layer may contain any number of elements.

[0080] In various implementations, each layer of the LSTM neural network 902 must contain the same number of elements as each of the other layers of the LSTM neural network 902. In some example embodiments, a convolutional neural network may be implemented. Similar to LSTM neural networks, convolutional neural networks include an input layer, a hidden layer, and an output layer. However, in a convolutional neural network, the output layer contains one fewer output than the number of neurons in the hidden layer, and each neuron is connected to each output. Furthermore, each input in the input layer is connected to every neuron in the hidden layer. In other words, the input 904a is connected to each of the neurons 908a, 908b, ... 908n.

[0081] In various implementations, each input node in the input layer can be assigned a numerical value, which can be any real number. In each layer, each connection emanating from an input node has an associated weight, which can also be any real number. In the input layer, the number of neurons is equal to the number of features (columns) in a dataset. The output layer can have multiple continuous outputs.

[0082] As mentioned above, the layers between the input and output layers are hidden layers. The number of hidden layers can be one or more (one hidden layer may be sufficient for many applications). A neural network without hidden layers can represent linear separable functions or decisions. A neural network with one hidden layer can perform a continuous mapping from one finite space to another. A neural network with two hidden layers can approximate any smooth mapping with arbitrary accuracy. The neural network of Fig. 9 may receive inputs, e.g., vectors, that may be used to generate models that may be used with vehicle camera images to generate outputs for predicting a pedestrian's crossing intent, as described herein.

[0083] Fig. 10 shows an exemplary process for generating a machine learning model. At 1007, the controller obtains data from a database 1002 (e.g., a data warehouse). The data may include any suitable data for developing machine learning models.

[0084] At 1011, the controller separates the data obtained from the database 1002 into training data 1015 and test data 1019. The training data 1015 is used to train the model at 1023, and the test data 1019 is used to test the model at 1027. Typically, depending on the desired parameters for model development, the set of training data 1015 is selected to be larger than the set of test data 1019. For example, the training data 1015 may comprise about seventy percent of the data obtained from the database 1002, about eighty percent of the data, about ninety percent, etc. The remaining thirty percent, twenty percent, or ten percent is then used as the test data 1019.

[0085] Separating a portion of the acquired data as test data 1019 enables testing of the trained model against the actual output data to facilitate more accurate training and development of the model at 1023 and 1027. The model may be trained at 1023 using any suitable techniques for machine learning models, including those described herein such as random forest, generalized linear models, decision trees, and neural networks.

[0086] At 1031, the controller evaluates the model test results. For example, at 1027, the trained model may be tested using the test data 1019, and the results of the output data of the tested model may be compared to actual outputs of the test data 1019 to determine an accuracy level. The model results may be evaluated using any suitable analysis of a machine learning model, such as the example techniques described below.

[0087] After evaluating the model test results in 1031, the model can be deployed in 1035 if the model test results are satisfactory. Deploying the model may involve using the model to generate predictions for a large input dataset with unknown outputs. If the evaluation of the model test results in 1031 is not satisfactory, the model can be further developed using different parameters, using different modeling techniques, using different model types, etc. The procedure for a machine learning model of Fig. 10 can receive inputs, e.g., vectors, that can be used to generate models that can be used in image processing, object detection, etc., as described here.

Claims

[1] Vehicle control system for camera-based vehicle navigation, the vehicle control system comprising: at least one vehicle camera configured to capture an image of a forward view of a vehicle; a global positioning system (GPS) receiver configured to obtain a current location of the vehicle; a vehicle user interface including a display; and a vehicle control module configured to: receives the current location of the vehicle via the GPS receiver; identifies a sequence of vehicle navigation steps from the current location of the vehicle to a destination; records the image of the view to the front of the vehicle via at least one vehicle camera; processes the image with a machine learning model to detect multiple objects in the image; classifies the plurality of objects according to landmark evaluation criteria that indicate a probability of object detection by a driver of the vehicle; and displays on the vehicle user interface a highest ranked of the plurality of objects in association with a next vehicle navigation step; characterized by , that the vehicle control module is configured to: processes the image to generate a saliency map by a red color saliency map is generated that highlights pixels in the image that correspond to a red color; a blue color saliency map is generated that highlights pixels in the image that correspond to a blue color; an intensity saliency map is generated that indicates an intensity level for each pixel in the image; a Gabor saliency map is generated according to a detection of straight lines in the image; and the red color conspicuity map, the blue color conspicuity map, the intensity conspicuity map and the Gabor conspicuity map are combined, where the saliency map indicates a saliency level for each pixel of the image; and provides the machine learning model with the saliency map to facilitate classification of the multiple detected objects. [2] The vehicle control system of claim 1, wherein the vehicle control module is configured to: records a turning action of the vehicle; compares a location of the turn action with the next vehicle navigation step to determine a turn conformity score indicating whether the highest ranked of the plurality of objects was an accurate guidance landmark; and the machine learning model is updated through supervised learning according to the turn conformity assessment. [3] The vehicle control system of claim 1, wherein the vehicle control module is configured to: obtains a distance between the current location of the vehicle and the next vehicle navigation step; and the image is processed with a depth estimation model to generate a region of interest within the image, where the distance between the current location of the vehicle and the next vehicle navigation step lies within the region of interest of the image. [4] The vehicle control system of claim 3, wherein the vehicle control module is configured to detect the plurality of objects in the image only within the region of interest. [5] The vehicle control system of claim 3, wherein the vehicle control module is configured to: accesses a database to obtain multiple points of interest corresponding to the current location of the vehicle; and Objects corresponding to the points of interest within the region of interest of the image are detected. [6] The vehicle control system of claim 5, wherein the vehicle control module is configured to: obtaining an associated popularity score from the database for each of the plurality of points of interest, the associated popularity score indicating a degree of recognition of the point of interest; and providing each associated popularity score to the machine learning model to facilitate ranking the objects according to the points of interest. [7] The vehicle control system of claim 1, wherein the vehicle control module is configured to: the image is divided into several segments using the machine learning model; crops a bounding box for each of the multiple segments; and Each of the multiple segments is transformed into a text output using a large machine learning language model. [8] A method for camera-based vehicle navigation, the method comprising: Obtaining a current location of a vehicle via a global positioning system (GPS) receiver of the vehicle; Identifying a sequence of vehicle navigation steps from the current location of the vehicle to a destination; Capturing an image of a forward view of the vehicle via at least one vehicle camera; Processing the image with a machine learning model to detect multiple objects in the image; Classifying the plurality of objects according to landmark evaluation criteria that indicate a probability of object detection by a driver of the vehicle; and Displaying, on a vehicle user interface, a highest ranked one of the plurality of objects in association with a next vehicle navigation step; characterized by , that processing the image to generate a saliency map, wherein the saliency map indicates a degree of saliency for each pixel of the image, and providing the saliency map to the machine learning model to facilitate ranking of the multiple detected objects, where generating the abnormality map comprises: generating a red color saliency map that highlights pixels in the image that correspond to a red color, generating a blue color saliency map that highlights pixels in the image that correspond to a blue color, generating an intensity saliency map that indicates an intensity level for each pixel in the image, generating a Gabor saliency map according to the detection of straight lines in the image, and combining the red color saliency map, the blue color saliency map, the intensity saliency map, and the Gabor saliency map.

Images