Context recommendation and control

Abstract

A system for providing contextual recommendations within a vehicle includes a system controller in communication with a plurality of on-board sensors adapted to collect real-time data relating to a location of the vehicle and an operating state of the vehicle; a database in communication with the system controller, adapted to store data relating to past actions and data relating to the position of the vehicle and the operating state of the vehicle when such past actions occur, the system controller includes a driver-specific machine learning model adapted to predict a desired action based on real-time data relating to a location and an operating state of the vehicle and data from a database, the system controller further adapted to initiate the predicted desired action, receive input from an occupant within the vehicle, and updating the driver-specific machine learning model.

Description

Technical Field

[0001] The present disclosure relates to systems and methods for providing recommended actions to occupants within a vehicle. Background Art

[0002] Vehicles are equipped with many sensors to monitor the environment around the vehicle and provide warnings when objects are approaching, etc. In many cases, occupants within the vehicle take specific actions, such as opening a glove box or closing a window whenever the occupant reaches a specific location or when certain conditions exist. Furthermore, in modern vehicles, the ability to activate certain actions often requires occupants within the vehicle to navigate through multiple menus on a display screen.

[0003] Therefore, while current systems and methods achieve their intended purposes, a new and improved system and method is needed for providing recommendations to occupants in a vehicle by receiving input from sensors and predicting desired actions, and initiating the desired actions automatically or by prompting the occupants in the vehicle. Summary of the invention

[0004] According to several aspects of the present disclosure, a method for providing contextual recommendations within a vehicle includes collecting real-time data related to the location of the vehicle and the operating status of the vehicle through multiple on-board sensors that communicate with a system controller, accessing a database of stored data related to past actions and data related to the location of the vehicle and the operating status of the vehicle through the system controller, predicting a desired action based on the real-time data related to the location of the vehicle and the operating status of the vehicle and the data from the database through a driver-specific machine learning model within the system controller when such past actions occur, initiating the predicted desired action, receiving input from an occupant within the vehicle, and updating the driver-specific machine learning model.

[0005] According to another aspect, the method also includes training a general machine learning model by collecting data from multiple different vehicles located in areas and climates similar to the vehicle, uploading the general machine learning model to the vehicle, and creating a driver-specific machine learning model by updating the general machine learning model.

[0006] According to another aspect, initiating the desired action further includes prompting an occupant of the vehicle for a recommendation of the predicted desired action.

[0007] According to another aspect, prompting an occupant in the vehicle to recommend a predicted desired action also includes providing an audible recommendation of the predicted desired action via a speaker connected to a human-machine interface (HMI), and displaying the recommendation of the predicted desired action on a touch screen display of the HMI, and receiving input from the occupant in the vehicle via a system controller includes at least one of receiving verbal input from the occupant in the vehicle via a microphone connected to the HMI and receiving input from the occupant in the vehicle through the touch screen display.

[0008] According to another aspect, displaying recommendations for predicted desired actions on a touch screen display of the HMI further comprises considering icons for predicted desired actions within a plurality of predefined menus suitable for display on the touch screen display of the HMI, and displaying only icons for predicted desired actions on the touch screen display.

[0009] According to another aspect, initiating the desired action further comprises automatically initiating, by the system controller, the desired action.

[0010] According to another aspect, updating the driver-specific machine learning model further includes receiving new data from other vehicles, selecting training data from the new data, and updating the driver-specific machine learning model with the selected training data.

[0011] According to another aspect, updating the driver-specific machine learning model also includes comparing input received from an occupant within the vehicle with the predicted desired action, and updating the driver-specific machine learning model when the input from the occupant within the vehicle does not match the predicted desired action.

[0012] According to another aspect, the method further includes receiving data related to an action being taken by an occupant in the vehicle via communication between the system controller and an onboard system in the vehicle, comparing the action being taken by the occupant in the vehicle with a predicted desired action, and at least one of: prompting the occupant in the vehicle to verify that the occupant in the vehicle wants to proceed when the action being taken by the occupant does not match the predicted desired action, and prompting the occupant in the vehicle with a warning message when the action being taken by the occupant is identified by the system controller as an inherently unsafe action.

[0013] According to several aspects of the present disclosure, a system for providing contextual recommendations within a vehicle includes: a system controller in communication with a plurality of on-board sensors, the plurality of on-board sensors being adapted to collect real-time data related to the location of the vehicle and the operating status of the vehicle; a database in communication with the system controller being adapted to store data related to past actions and data related to the location of the vehicle and the operating status of the vehicle when such past actions occurred, the system controller including a driver-specific machine learning model adapted to predict desired actions based on the real-time data related to the location and operating status of the vehicle and data from the database, the system controller being further adapted to initiate predicted desired actions, receive input from occupants within the vehicle, and update the driver-specific machine learning model.

[0014] According to another aspect, a driver-specific machine learning model is created by training a general machine learning model with data collected from multiple different vehicles located in areas and climates similar to the vehicle, uploading the general machine learning model to the vehicle, and creating a driver-specific machine learning model by updating the general machine learning algorithm.

[0015] According to another aspect, when the desired action is initiated, the system controller is further adapted to prompt an occupant within the vehicle with a recommendation for the predicted desired action.

[0016] According to another aspect, when prompting an occupant in the vehicle to recommend a predicted desired action, the system controller is further adapted to provide at least one recommendation for the predicted desired action via a speaker connected to a human machine interface (HMI) and display the recommendation for the predicted desired action on a touch screen display of the HMI, and when receiving input from an occupant in the vehicle, the system controller is further adapted to receive at least one of verbal input from the occupant in the vehicle via a microphone connected to the HMI and input from the occupant in the vehicle through the touch screen display.

[0017] According to another aspect, when displaying recommendations for predicted desired actions on the touch screen display of the HMI, the system controller is further adapted to identify icons for the predicted desired actions within a plurality of predefined menus adapted to be displayed on the touch screen display of the HMI, and to display only the icons for the predicted desired actions on the touch screen display.

[0018] According to another aspect, when the desired action is initiated, the system controller is further adapted to automatically initiate the desired action.

[0019] According to another aspect, when updating the driver-specific machine learning model, the system controller is further adapted to receive new data from other vehicles, select training data from the new data, and update the driver-specific machine learning model by the selected training data.

[0020] According to another aspect, when updating the driver-specific machine learning model, the system controller is also adapted to compare input received from an occupant within the vehicle with the predicted desired action, and to update the driver-specific machine learning model when the input from the occupant within the vehicle does not match the predicted desired action.

[0021] According to another aspect, the system controller is further adapted to receive data related to actions being taken by an occupant in the vehicle via communications between the system controller and an onboard system in the vehicle, compare the actions being taken by the occupant in the vehicle with predicted desired actions, and at least one of: prompting the occupant in the vehicle to verify that the occupant in the vehicle wants to proceed with the action when the action being taken by the occupant does not match the predicted desired action, and prompting the occupant in the vehicle with a warning message when the action being taken by the occupant is identified by the system controller as an inherently unsafe action.

[0022] Further areas of applicability will become apparent from the description provided herein.It should be understood that the description and specific examples are intended for purposes of illustration only and are not intended to limit the scope of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] The drawings described herein are for illustration purposes only and are not intended to limit the scope of the present disclosure in any way.

[0024] Figure 1 is a schematic diagram of a vehicle having a system for providing a recommended action to an occupant according to an exemplary embodiment;

[0025] Figure 2 is a schematic diagram of a system according to an exemplary embodiment;

[0026] Figure 3 is a schematic diagram showing how to create and update a driver-specific machine learning model;

[0027] Figure 4 is a schematic diagram showing a plurality of menus displayed on a touch screen display;

[0028] Figure 5 yes Figure 1 a schematic diagram of the interior of the vehicle shown;

[0029] Figure 6 is a front view of the touch screen display showing an icon and message for releasing the glove box; and

[0030] Figure 7 is a flow chart illustrating a method of providing recommendations to an occupant in a vehicle according to an exemplary embodiment.

[0031] The drawings are not necessarily drawn to scale, and some features may be enlarged or reduced, such as to illustrate details of a particular component. In some cases, well-known components, systems, materials, or methods are not described in detail to avoid obscuring the present disclosure. Therefore, the specific structural and functional details disclosed herein should not be interpreted as limiting, but merely as a basis for the claims and as a representative basis for teaching those skilled in the art to use the present disclosure in various ways. DETAILED DESCRIPTION

[0032] The following description is essentially only exemplary and is not intended to limit the present disclosure, application or use. In addition, it is not desired to be bound by any express or implied theory presented in the aforementioned technical field, background technology, summary of the invention or the following specific embodiments. It should be understood that in all drawings, the corresponding reference numerals represent the same or corresponding parts and features. As used herein, the term module refers to any hardware, software, firmware, electronic control component, processing logic and / or processor device individually or in any combination, including but not limited to: specific integrated circuit (ASIC), electronic circuit, processor (shared, specific or grouped) and memory, combinational logic circuit and / or other suitable components providing the function that execute one or more software or firmware programs. Although the drawings shown in this article describe examples with certain element arrangements, additional intermediate elements, devices, features or components may be present in actual embodiments. It should also be understood that the drawings are merely illustrative and may not be drawn to scale.

[0033] As used herein, the term "vehicle" is not limited to automobiles. Although the present technology is described herein primarily in conjunction with automobiles, the present technology is not limited to automobiles. These concepts can be used in a variety of applications, such as those related to aircraft, ships, other vehicles, and consumer electronic components.

[0034] According to an exemplary embodiment, Figure 1 An electric vehicle 10 is shown having an associated system 11 for providing recommendations to occupants within the vehicle 10 according to various embodiments. Generally, the system 11 works with other systems within the vehicle 10 to display various information and infotainment content to the passengers. The vehicle 10 generally includes a chassis 12, a body 14, front wheels 16, and rear wheels 18. The body 14 is disposed on the chassis 12 and generally surrounds the elements of the vehicle 10. The body 14 and the chassis 12 may together form a frame. The front wheels 16 and the rear wheels 18 are each rotatably coupled to the chassis 12 near respective corners of the body 14.

[0035] In various embodiments, the vehicle 10 is an autonomous vehicle, and the system 11 is incorporated into the autonomous vehicle 10. The autonomous vehicle 10 is, for example, a vehicle 10 that is automatically controlled to transport passengers from one location to another. The vehicle 10 is depicted as a passenger car in the illustrated embodiment, but it should be understood that any other vehicle including motorcycles, trucks, sport utility vehicles (SUVs), recreational vehicles (RVs), etc. may also be used. In an exemplary embodiment, the vehicle 10 is equipped with a so-called level 4 or level 5 automation system. A level 4 system represents "high automation" and refers to the driving mode-specific performance exhibited by the autonomous driving system in all aspects of dynamic driving tasks even when the human driver does not respond appropriately to intervention requests. A level 5 system represents "full automation" and refers to the full-time performance of the autonomous driving system in all aspects of dynamic driving tasks under all road and environmental conditions that can be managed by a human driver. The novel aspects of the present disclosure are also applicable to non-autonomous vehicles.

[0036] As shown, the vehicle 10 generally includes a propulsion system 20, a transmission system 22, a steering system 24, a braking system 26, a sensor system 28, an actuator system 30, at least one data storage device 32, a vehicle controller 34, and a wireless communication system 36. In embodiments where the vehicle 10 is an electric vehicle, there may be no transmission system 22. In various embodiments, the propulsion system 20 may include an internal combustion engine, an electric machine such as a traction motor, and / or a fuel cell propulsion system. The transmission system 22 is used to transmit power from the propulsion system 20 to the front wheels 16 and rear wheels 18 of the vehicle according to a selectable speed ratio. According to different embodiments, the transmission system 22 may include a stepped ratio automatic transmission, a continuously variable transmission, or an appropriate transmission. The braking system 26 is used to provide braking torque to the front wheels 16 and rear wheels 18 of the vehicle. In various embodiments, the braking system 26 may include friction brakes, line brakes, regenerative braking systems such as electric machines, and / or other appropriate braking systems. The steering system 24 affects the position of the front wheels 16 and rear wheels 18. Although depicted as including a steering wheel for purposes of illustration, in some embodiments contemplated within the scope of the present disclosure, the steering system 24 may not include a steering wheel.

[0037] The sensor system 28 includes one or more sensing devices 40a-40n that sense observable conditions of the external environment and / or the internal environment of the autonomous vehicle 10. The sensing devices 40a-40n may include, but are not limited to, radar, lidar, global positioning system, optical camera, thermal imager, ultrasonic sensor, and / or other sensors. The camera may include two or two or more digital cameras at a selected distance, wherein the two or more digital cameras are used to obtain stereo images of the surrounding environment in order to obtain a three-dimensional image or map. The plurality of sensing devices 40a-40n are used to determine information about the surrounding environment of the vehicle 10. In an exemplary embodiment, the plurality of sensing devices 40a-40n include at least one of a motor speed sensor, a motor torque sensor, an electric drive motor voltage and / or current sensor, an accelerator pedal position sensor, a coolant temperature sensor, a cooling fan speed sensor, and a transmission oil temperature sensor. In another exemplary embodiment, the plurality of sensing devices 40a-40n also include sensors for determining information about the surrounding environment of the vehicle 10, for example, an ambient air temperature sensor, an atmospheric pressure sensor, and / or a photo and / or video camera positioned to view the environment in front of the vehicle 10. In another exemplary embodiment, at least one of the plurality of sensing devices 40 a - 40 n is capable of measuring distances in the environment surrounding the vehicle 10 .

[0038] In a non-limiting example where the plurality of sensing devices 40a-40n include cameras, the plurality of sensing devices 40a-40n measure distance using an image processing algorithm for processing images from the cameras and determining distances between objects. In another non-limiting example, the plurality of vehicle sensors 40a-40n include a stereo camera with distance measurement capabilities. In one example, at least one of the plurality of sensing devices 40a-40n is secured to the interior of the vehicle 10, such as in a roof lining of the vehicle 10, with a view through the windshield of the vehicle 10. In another example, at least one of the plurality of sensing devices 40a-40n is secured to the exterior of the vehicle 10, such as on the roof of the vehicle 10, with a view of the environment surrounding the vehicle 10. It should be understood that various additional types of sensing devices, such as DAR sensors, ultrasonic ranging sensors, radar sensors, and / or time-of-flight sensors are within the scope of the present disclosure. The actuator system 30 includes one or more actuator devices 42 a - 42 n that control one or more features of the vehicle 10 , such as, but not limited to, the propulsion system 20 , the transmission system 22 , the steering system 24 , and the braking system 26 .

[0039] The vehicle controller 34 includes at least one processor 44 and a computer readable storage device or medium 46. The at least one data processor 44 can be any custom or commercially available processor, a CPU (central processing unit), a GPU (graphics processing unit), an auxiliary processor of several processors associated with the vehicle controller 34, a semiconductor-based microprocessor (in the form of a microchip or chipset), a macroprocessor, any combination thereof, or any device generally used to execute instructions. The computer readable storage device or medium 46 can include volatile and non-volatile storage in, for example, read-only memory (ROM), random access memory (RAM), and keep-alive memory (KAM). KAM is a persistent or non-volatile memory that can be used to store various operating variables when the at least one data processor 44 is powered off. The computer readable storage device or medium 46 can be implemented using any of a variety of known memory devices, such as a programmable read-only memory (PROM), an electrical PROM (EPROM), an electrically erasable PROM (EEPROM), flash memory, or any other electrical, magnetic, optical, or combination memory device capable of storing data, some of which represents executable instructions used by the controller 34 in controlling the vehicle 10.

[0040] The instructions may include one or more separate programs, each of which includes an ordered list of executable instructions for implementing logical functions. When executed by at least one processor 44, the instructions receive and process signals from the sensor system 28, execute logic, calculations, methods and / or algorithms for automatically controlling components of the vehicle 10, and generate control signals to the actuator system 30 based on the logic, calculations, methods and / or algorithms to automatically control the components of the vehicle 10. Although in Figure 1 Only one controller 34 is shown, but embodiments of vehicle 10 may include any number of controllers 34 that communicate via any suitable communication medium or combination of communication mediums and cooperate to process sensor signals, execute logic, calculations, methods and / or algorithms, and generate control signals to automatically control features of autonomous vehicle 10.

[0041] In various embodiments, one or more instructions of the vehicle controller 34 are included in the trajectory planning system and, when executed by at least one data processor 44, generate trajectory outputs that solve the kinematic and dynamic constraints of the environment. For example, the instructions receive process sensor and map data as input. The instructions execute a graph-based approach with a customized cost function to handle different road scenarios in urban and highway roads.

[0042] The wireless communication system 36 is used to wirelessly communicate with other remote entities 48, such as, but not limited to, other vehicles ("V2V" communication), infrastructure ("V2I" communication), remote systems, remote servers, cloud computers and / or personal devices. In an exemplary embodiment, the communication system 36 is a wireless communication system for communicating via a wireless local area network (WLAN) using the IEEE 802.11 standard or by using cellular data communications. However, other or alternative communication methods such as a dedicated short-range communication (DSRC) channel are also considered to be within the scope of the present disclosure. A DSRC channel refers to a one-way or two-way short-range to medium-range wireless communication channel dedicated to the vehicle and a corresponding set of protocols and standards.

[0043] The vehicle controller 34 is a non-universal electronic control device having a pre-programmed digital computer or processor, a memory or non-transitory computer readable medium for storing data such as control logic, software applications, instructions, computer code, data, look-up tables, etc., and a transceiver [or input / output port]. Computer readable media include any type of media that can be accessed by a computer, such as read-only memory (ROM), random access memory (RAM), hard disk drive, compact disk (CD), digital video disk (DVD), or any other type of memory. "Non-transitory" computer readable media do not include wired, wireless, optical or other communication links that transmit temporary electrical or other signals. Non-transitory computer readable media include media that can permanently store data and media that can store and subsequently rewrite data, such as rewritable optical disks or erasable memory devices. Computer code includes any type of program code, including source code, object code, and executable code.

[0044] refer to Figure 2 , a schematic diagram of the system 11 is shown. The system 11 includes a system controller 34A in communication with a plurality of sensing devices (on-board sensors) 40a-40n. The system controller 34A may be the vehicle controller 34, or the system controller 34A may be a separate controller in communication with the vehicle controller 34. In addition to the plurality of on-board sensors 40a-40n, the system controller 34A also communicates with a human machine interface (HMI) 50, a vehicle control module (VCM) 52, a database 54, and a wireless communication module 36.

[0045] The system controller 34A is adapted to collect real-time data related to the location of the vehicle 10 and the operating conditions of the vehicle 10 via communication with a plurality of onboard sensors 40a-40n. The database 54 is in communication with the system controller 34A and is adapted to store data related to past actions and data related to the location of the vehicle 10 and the operating conditions of the vehicle 10 when such past actions occurred. Past actions may include actions taken by an occupant 56, such as opening a window, closing a window, opening a glove box, etc. The database 54 stores the occurrence of such actions and data related to the location of the vehicle 10 and the operating conditions of the vehicle 10 (such as weather, speed, other occupants in the vehicle, etc.) when such past actions occurred.

[0046] The system controller 34A includes a driver-specific machine learning model 58 adapted to predict desired actions based on real-time data related to the location and operating state of the vehicle 10 and data from the database 54. Various techniques are employed to extract meaningful features from the sensor readings and data, including time series analysis, frequency domain analysis, and space-time patterns. The driver-specific machine learning model may be, but is not limited to, one of the following: long short-term memory (LSTM), gated recurrent unit (GRU), decision tree, random forest, support vector machine (SVM), neural network (NN), nearest neighbor method (KNN), gradient boosting, and recurrent neural network (RNN).

[0047] Therefore, the system controller 34A uses the driver-specific machine learning model 58 and machine learning techniques to predict the desired action that the occupant 56 will take based on analyzing real-time data of the location of the vehicle 10 and the operating state of the vehicle 10 based on data received from the database 54, including past actions and the location and operating state of the vehicle 10 when such past actions occurred.

[0048] Once the system controller 34A predicts the desired action using the driver-specific machine learning model 58 , the system controller 34A is adapted to initiate the predicted desired action, receive input from an occupant 56 within the vehicle 10 , and update the driver-specific machine learning model 58 .

[0049] Occupants in a vehicle often behave in repetitive patterns. These patterns may include, for example, opening the glove box whenever the occupant 56 arrives at a particular shopping center in the vehicle 10, or rolling up the windows whenever it rains or when the vehicle 10 approaches a car wash. Observing such patterns allows the driver-specific machine learning model 58 to establish behavioral patterns and predict future behaviors based on such patterns.

[0050] refer to Figure 3, in order to create the driver-specific machine learning model 58, a general machine learning model 60 is first trained by collecting data (offline dataset 62) from multiple different vehicles located in areas and climates similar to the vehicle 10. Different datasets are collected from vehicles equipped with sensors such as GPS, accelerometers, cameras, radars, and LIDARs. The data contains various driving scenarios, including urban, highway, and off-road driving. Before the data is fed into the machine learning model, preprocessing steps are taken to remove noise, handle missing values, and standardize features. The basic step in driving behavior classification is to extract relevant features from the raw data. As described above, various techniques are used to extract meaningful features from sensor readings, including time series analysis, frequency domain analysis, and space-time patterns. Different types of machine learning algorithms can be used for probabilistic recognition of patterns, including but not limited to long short-term memory (LSTM), gated recurrent unit (GRU), decision tree, random forest, support vector machine (SVM), neural network (NN), nearest neighbor method (KNN), gradient enhancement, and recurrent neural network (RNN). A general machine learning model 60 is trained on a labeled dataset and evaluated using various performance metrics such as accuracy, precision, recall, F1 score, and confusion matrix. The hyperparameters of the model are adjusted to achieve optimal results. The data is trained on the general machine learning model 60 and learns to map input features to corresponding pattern (action) probabilities.

[0051] The generic machine learning model 60 is uploaded to the system controller 34A within the vehicle 10 (transfer learning 64). The generic machine learning model 60 provides the basis for creating the driver-specific profile and the driver-specific machine learning model 58, and the upload of the generic machine learning model 60 can be via a subscription-based service from a third-party provider or the manufacturer of the vehicle 10. The upload of the generic machine learning model 60 can be via a subscription-based service from a third-party provider or the manufacturer of the vehicle 10.

[0052] The driver-specific machine learning model 58 is ultimately created by updating the general machine learning model 60 (online learning 66). Once the general machine learning model 60 is uploaded, data is collected as the occupant 56 uses the vehicle 10 day after day. As the occupant 56 uses the vehicle 10, the general machine learning model 60 is updated to personalize the general machine learning model 60 to the specific occupant 56 of the vehicle 10, thereby creating a driver-specific machine learning model 58 that is continuously updated at the same time, as described below.

[0053] In an exemplary embodiment, the system controller 34A is further adapted to prompt the occupant 56 in the vehicle 10 with a recommended predicted desired action when the desired action is initiated. For example, if the occupant 56 and the vehicle 10 are approaching the aforementioned specific shopping mall, the driver-specific machine learning model 58 accesses data from the database 54 and detects a pattern that the occupant 56 has a high probability of opening the glove box shortly after parking the vehicle 10 in a parking space when located in a shopping mall. Figure 4 In order to open the glove box, the occupant 56 must navigate through multiple menus on the touch screen display 68 of the HMI 50, however, the driver-specific machine learning model 58 predicts the desired action (when located in a shopping mall, the occupant 56 will want to open the glove box shortly after the vehicle 10 is parked in a parking space) and prompts the occupant 56 to check whether the occupant 56 wants to open the glove box.

[0054] refer to Figure 5 In one exemplary embodiment, when prompting the occupant 56 in the vehicle 10 with a recommendation for the predicted desired action, the system controller 34A is also adapted to provide an audible recommendation for the predicted desired action via a speaker 70 connected to the HMI 50. Thus, the system controller 34A broadcasts an audible message to the occupant 56 via the speaker 70, such as “We are at the shopping center, do you want to open the glove box?”.

[0055] In another exemplary embodiment, the system controller 34A is further adapted to display the recommendation for the predicted desired action on the touch screen display 68 of the HMI 50 when prompting the occupant 56 in the vehicle 10 for the recommendation for the predicted desired action. Thus, the system controller 34A may display a text message on the touch screen display 68 of the HMI 50, “We are at the shopping center, do you want to open the glove box?”.

[0056] In another exemplary embodiment, the system controller 34 identifies icons 72 for predicted desired actions within a plurality of predefined menus suitable for display on the touch screen display 68 of the HMI 50 and displays only the icons 72 for predicted desired actions. Figure 4 In this example, typically to open the glove box, the occupant 56 would need to navigate through multiple menus on the touch screen display 68 of the HMI 50 to find the icon 72 representing the glove box release, where the occupant 56 would touch the icon 72 to open the glove box. Figure 6 , in this case, the system controller 34A recognizes the glove box icon 72 and clears the touch screen display 68 of the occupant 56 by only displaying the glove box release icon 72 and the message “We are at the shopping center, do you want to open the glove box?” This is an example of location-based dynamic and personalized adaptation of the HMI interaction experience.

[0057] Reference again Figure 5 When receiving input from an occupant 56 in the vehicle 10, the system controller 34A is also adapted to receive verbal input from the occupant 56 in the vehicle 10 via a microphone 74 connected to the HMI 50. Thus, when the system controller 34A provides a voice prompt with a predicted desired action, such as "We are at the shopping center, do you want to open the glove box?", the occupant 56 can respond, such as saying "yes" or "no".

[0058] In another exemplary embodiment, when receiving input from an occupant 56 in the vehicle 10, the system controller 34A is further adapted to receive input from an occupant 56 in the vehicle 10 via the touch screen display 68. Thus, when a message is prompted on the touch screen display 68, the occupant 56 can choose whether to take the predicted desired action. For example, referring to Figure 6 , the system controller 34A prompts the occupant 56 by displaying an icon 72 for releasing the glove box and a message “We are at the shopping center, do you want to open the glove box?”, wherein if the occupant 56 wishes to open the glove box, the occupant 56 can touch the icon 72 on the display 68 to open the glove box. If the occupant 56 does not want to open the glove box, the occupant 56 does nothing, and after a certain time, the system controller 34A recognizes the lack of response of the occupant 56 as a negative response.

[0059] In another exemplary embodiment, the system controller 34A is further adapted to automatically initiate the desired action when the desired action is initiated. Thus, in the above example, when the occupant 56 arrives at a shopping mall in the vehicle 10, the driver-specific machine learning model 58 accesses data from the database 54 and detects the following pattern: when located at the shopping mall, there is a high probability that the occupant 56 will open the glove box shortly after parking the vehicle 10 in a parking space. Therefore, once the occupant 56 parks the vehicle 10 in a parking space, the system controller 34A automatically opens the glove box. Such automatic actions may be authorized in advance by the occupant 56 and may be limited to basic actions, such as opening / closing windows, where actions like parking the vehicle 10 in a parking space cannot be automatically initiated. In all cases, the system controller 34A will not automatically or in response to input from the occupant 56 initiate any action that would violate the safety protocol.

[0060] Furthermore, automatic initiation of the predicted desired action may occur only when the probability calculated by the driver-specific machine learning model 58 that the occupant 56 will take the predicted action exceeds a certain threshold. For example, the driver-specific machine learning model 58 accesses data from the database 54 and detects a pattern in which there is a high probability that the occupant 56 will open the glove box shortly after parking the vehicle 10 in a parking space when located at a shopping center. In an exemplary embodiment, if the probability calculated by the driver-specific machine learning model 58 that the occupant 56 will open the glove box exceeds 50%, the system controller 34A will prompt the occupant 56 to recommend the predicted desired action (open the glove box), and if the probability calculated by the driver-specific machine learning model 58 that the occupant 56 will open the glove box exceeds 90%, the system controller 34A will automatically open the glove box.

[0061] In another exemplary embodiment, when prompting an occupant 56 in the vehicle 10 for a recommendation regarding the predicted desired action, the system controller 34A is further adapted to provide the recommendation regarding the predicted desired action via a personal device of the occupant 56. For example, if the occupant has left the vehicle 10 and the weather conditions indicate that it is raining, or the sensors 40a-40n in the vehicle 10 detect that it is raining, the system controller 34A may prompt the occupant 56 via a personal device such as a smart phone or smart watch to recommend closing the windows of the occupant 56. In another example, the system controller 34A identifies a pattern through the machine learning model 58, wherein whenever the occupant 56 arrives at a particular destination, the occupant 56 locks the doors of the vehicle 10 when leaving the vehicle 10, and there are no other passengers in the vehicle 10. Therefore, when the occupant 56 leaves the vehicle 10, the system controller 34A prompts the occupant 56 via a personal device to recommend that the occupant 56 should lock the doors of the vehicle 10.

[0062] Reference again Figure 3 In one exemplary embodiment, when updating the driver-specific machine learning model 58, the system controller 34A is further adapted to receive new data 76 from other vehicles, select training data from the new data 76, and update the driver-specific machine learning model 58 with the selected training data. The new data 76 from other vehicles may be received via upload from a subscription service or a database 78 supported by the vehicle manufacturer, or by receiving data from other vehicles in similar locations.

[0063] In another exemplary embodiment, when updating the driver-specific machine learning model 58, the system controller 34A is also suitable for comparing the input received from the occupant 56 in the vehicle 10 with the predicted desired action at box 80, and updating the driver-specific machine learning model 58 (online learning 66) when the input from the occupant 56 in the vehicle 10 does not match the predicted desired action.

[0064] For example, when the occupant 56 arrives at a shopping mall in the vehicle 10, the driver-specific machine learning model 58 accesses data from the database 54 and detects a pattern that there is a high probability that the occupant 56 will open the glove box shortly after parking the vehicle 10 in a parking space when located at a shopping mall. Therefore, the system controller 34A prompts the occupant 56 to recommend the predicted desired action (opening the glove box).

[0065] If in Figure 3 At box 80 of the HMI 50, if the occupant 56 in the vehicle 10 accepts the recommendation and opens the glove box either verbally or by touching the display screen 68 of the HMI 50, the system controller 34A will take no further action (box 82) because the occupant 56 acted in accordance with the prediction of the driver-specific machine learning model 58, and therefore no update is required. However, if the occupant 56 in the vehicle 10 does not accept the recommendation and does not open the glove box, the system controller 34A will update the driver-specific machine learning model 58 (online learning 66). The situation in which the occupant 56 does not follow the recommendation of the predicted desired action will result in a lower calculated probability that the occupant 56 will open the glove box the next time he arrives at the shopping mall. The system controller 34A pushes data related to the occurrence to the database 54, and may also push data to a (cloud-based) database 78 supported by a subscription service or vehicle manufacturer, or receive data from other vehicles that are similarly located and provide new data to update to other vehicles.

[0066] In another exemplary embodiment, the system controller 34A is further adapted to receive data related to actions taken by an occupant 56 within the vehicle 10 via communications between the system controller 34A and an onboard system within the vehicle 10, compare the actions taken by the occupant 56 within the vehicle 10 with the predicted desired actions, and when the actions taken by the occupant 56 do not match the predicted desired actions, prompt the occupant 56 within the vehicle 10 to verify that the occupant 56 within the vehicle 10 wishes to continue the action.

[0067] For example, when the occupant 56 arrives at a shopping mall in the vehicle 10, the driver-specific machine learning model 58 accesses data from the database 54 and detects the following pattern: when located at a shopping mall, there is a high probability that the occupant 56 will open the glove box shortly after parking the vehicle 10 in a parking space. Therefore, the system controller 34A prompts the occupant 56 to recommend that the predicted desired action (opening the glove box) be taken. However, instead of opening the glove box, the occupant 56 in the vehicle 10 opens the door to exit the vehicle 10. The system controller 34A compares the action taken (opening the door) to the recommended predicted desired action (opening the glove box), and because they do not match, the occupant 56 of the vehicle 10 is prompted to verify that the occupant 56 does not wish to open the glove box but wishes to continue to open the door.

[0068] In another example, the system controller 34A detects that the occupant 56 has taken an unusual action. For example, the occupant 56 accidentally presses a button to open the trunk of the vehicle 10 while the vehicle 10 is traveling along a road at a high rate of speed. The database 54 includes instances where the occupant 56 did not open the trunk while the vehicle 10 was moving, and therefore the driver-specific machine learning model 58 identifies the action as abnormal and prompts the occupant 56 to verify that the occupant 56 actually wants to open the trunk. In addition, with or without any historical pattern, the system controller 34A may identify the action as inherently unsafe, and therefore the system controller 34A prompts the occupant 56 to verify that the occupant 56 actually wants to perform the action. This inherently unsafe action will not be driver-specific and will be part of the general machine learning model 60 regardless of any personalized settings of the driver-specific machine learning model 58. In the above example, the general machine learning model 60 identifies opening the trunk at a speed exceeding, for example, 10 MPH as inherently unsafe. The prompt provided by the system controller 34A may provide a warning message informing the occupant that the action is unsafe in addition to verifying that the occupant 56 wants to open the trunk.

[0069] The onboard safety system of vehicle 10 may prevent certain identified inherently unsafe actions, such as opening the trunk of vehicle 10 while traveling at a high speed. In this case, the system controller 34A takes no action and may provide a prompt to the occupant that simply informs occupant 56 that the action cannot be taken. In other cases, the action may be undesirable but is identified as abnormal or somewhat unsafe. For example, occupant 56 accidentally presses a button to open the trunk of vehicle 10 while vehicle 10 is in a car wash. Database 54 includes instances where occupant 56 did not open the trunk while vehicle 10 was in a car wash, and further, general machine learning model 60 is programmed that the trunk should not be opened while vehicle 10 is in a car wash. Therefore, driver-specific machine learning model 58 identifies the action as abnormal and prompts occupant 56 to verify that occupant 56 actually wants to open the trunk while vehicle 10 is being washed. The prompt may include a message indicating that it may be unwise to open the trunk while the vehicle 10 is being washed, but if the occupant 56 insists and provides input verifying that the occupant 56 wants to open the trunk, the system controller 34A will open the trunk.

[0070] refer to Figure 7, a method 100 for providing contextual recommendations within a vehicle 10 includes starting at box 102, collecting real-time data related to the location of the vehicle 10 and the operating status of the vehicle 10 through multiple on-board sensors 40a-40n in communication with the system controller 34A, moving to box 104, accessing stored data related to past actions and a database 54 related to the location of the vehicle 10 and the operating status of the vehicle 10 through the system controller 34A, when such past actions occur, moving to box 106, predicting a desired action based on the real-time data related to the location of the vehicle 10 and the operating status of the vehicle 10 and data from the database 54 through the driver-specific machine learning model 58 within the system controller 34A, moving to box 108, initiating the predicted desired action, moving to box 110, receiving input from an occupant 56 within the vehicle 10, and moving to box 112, updating the driver-specific machine learning model 58.

[0071] In an exemplary embodiment, method 100 also includes moving to box 114, training the general machine learning model 60 by collecting data from multiple different vehicles located in areas and climates similar to vehicle 10, moving to box 116, uploading the general machine learning model 60 to vehicle 10, and moving to box 118, creating a driver-specific machine learning model 58 by updating the general machine learning model 60.

[0072] In one exemplary embodiment, initiating the desired action at block 108 also includes moving to block 120 and prompting the occupant 56 within the vehicle 10 to recommend the predicted desired action.

[0073] In an exemplary embodiment, prompting the occupant 56 in the vehicle 10 at box 120 to recommend a predicted desired action also includes at least one of: moving to box 122, providing an audible recommendation for the predicted desired action through a speaker 70 connected to the human-machine interface (HMI) 50; and moving to box 124, displaying the recommendation for the predicted desired action on the touch screen display 68 of the HMI 50.

[0074] In another exemplary embodiment, receiving input from an occupant 56 within the vehicle 10 via the system controller 34A at box 110 includes at least one of: moving to box 126, receiving verbal input from the occupant 56 within the vehicle 10 via the microphone 74 connected to the HMI 50, and moving to box 128, receiving input from the occupant 56 within the vehicle 10 via the touch screen display 68.

[0075] In another exemplary embodiment, displaying recommendations for predicted desired actions on the touch screen display 68 of the HMI 50 at box 124 also includes moving to box 130, identifying icons 72 for predicted desired actions within a plurality of predefined menus suitable for display on the touch screen display 68 of the HMI 50, and moving to box 132, displaying only the icons 72 for the predicted desired actions on the touch screen display 68.

[0076] In another exemplary embodiment, initiating the desired action at block 108 further includes automatically moving to block 134 , initiating the desired action, by the system controller 34A.

[0077] In another exemplary embodiment, updating the driver-specific machine learning model 58 at box 112 also includes moving to box 136, receiving new data 76 from other vehicles, moving to box 138, selecting training data from the new data 76, and moving to box 140, updating the driver-specific machine learning model 58 with the selected training data.

[0078] In another exemplary embodiment, updating the driver-specific machine learning model 58 at box 112 also includes: moving to box 142, comparing the input received from the occupant 56 in the vehicle 10 with the predicted desired action; and moving to box 144, when the input from the occupant 56 in the vehicle 10 does not match the predicted desired action, moving to box 146, updating the driver-specific machine learning model 58. When at box 144, the input from the occupant 56 in the vehicle 10 matches the predicted desired action, moving to box 148, the system controller 34A takes no action.

[0079] In another exemplary embodiment, at box 106, after the desired action is predicted by the driver-specific machine learning model 58 within the system controller 34A based on real-time data related to the location of the vehicle 10 and the operating state of the vehicle 10 and data from the database 54, the method 100 includes moving to box 150, receiving data related to the action taken by the occupant 56 within the vehicle 10 via communication between the system controller 34A and the onboard system within the vehicle 10, moving to box 152, comparing the action taken by the occupant 56 within the vehicle 10 to the predicted desired action, and moving to box 154, when the action taken by the occupant 56 does not match the predicted desired action, moving to box 156, prompting the occupant 56 within the vehicle 10 to verify that the occupant 56 within the vehicle 10 wants to proceed. When the action taken by the occupant 56 at box 154 matches the predicted desired action, moving to box 158, the system controller 34A takes no action.

[0080] The systems and methods of the present disclosure provide the advantage of predicting a desired action of an occupant within a vehicle and prompting the occupant to easily and streamlined select the predicted desired action or automatically initiating the predicted desired action.

[0081] The description of the present disclosure is merely exemplary in nature, and variations that do not depart from the gist of the present disclosure are intended to be within the scope of the present disclosure. These variations should not be regarded as departing from the scope and scope of the present disclosure.

Claims

1. A method for providing contextual recommendations in a vehicle, comprising: collecting real-time data related to the location of the vehicle and the operating status of the vehicle via a plurality of onboard sensors in communication with a system controller; accessing, by the system controller, a database of stored data relating to past actions and data relating to the location of the vehicle and the operating state of the vehicle when such past actions occurred; predicting, by a driver-specific machine learning model within the system controller, a desired action based on the real-time data related to the position of the vehicle and the operating state of the vehicle and data from the database; Initiate the predicted desired action; receiving input from an occupant of the vehicle; and The driver-specific machine learning model is updated.

2. The method according to claim 1, further comprising: training a general machine learning model using data collected from a plurality of different vehicles located in regions and climates similar to the vehicle; uploading the general machine learning model to the vehicle; as well as The driver-specific machine learning model is created by updating the general machine learning model.

3. The method according to claim 2, wherein: Initiating the desired action also includes prompting the occupant of the vehicle with a recommendation for the predicted desired action.

4. The method according to claim 3, wherein: Prompting the occupant in the vehicle with a recommendation for the predicted desired action also includes at least one of the following: providing an audible recommendation of the predicted desired action via a speaker connected to a human machine interface (HMI); as well as displaying a recommendation for the predicted desired action on a touch screen display of the HMI; and Receiving input from an occupant in the vehicle by the system controller includes at least one of the following: receiving verbal input from the occupant within the vehicle via a microphone connected to the HMI; and Input is received from the occupant in the vehicle via the touch screen display.

5. The method according to claim 4, wherein: Displaying a recommendation for the predicted desired action on a touch screen display of the HMI further comprises: identifying an icon for the predicted desired action within a plurality of predefined menus adapted for display on the touch screen display of the HMI; and Only the icons for the predicted desired actions are displayed on the touch screen display.

6. The method according to claim 2, wherein: The initiating the desired action further comprises automatically initiating the desired action with the system controller.

7. The method according to claim 1, wherein: The updating of the driver-specific machine learning model further comprises: Receive new data from other vehicles; selecting training data from the new data; and The driver-specific machine learning model is updated by the selected training data.

8. The method according to claim 1, wherein: The updating of the driver-specific machine learning model further comprises: comparing the input received from the occupant in the vehicle to the predicted desired action; and The driver-specific machine learning model is updated when the input from the occupant within the vehicle does not match the predicted desired action.

9. The method according to claim 1, further comprising: receiving data related to actions being taken by the occupant in the vehicle via communications between the system controller and an onboard system in the vehicle; comparing an action taken by the occupant of the vehicle to the predicted desired action; And at least one of the following: prompting the occupant of the vehicle to verify that the occupant of the vehicle wants to proceed when the action being taken by the occupant does not match the predicted desired action; as well as When the action taken by the occupant is identified by the system controller as an inherently unsafe action, the occupant in the vehicle is alerted by a warning message.

10. A system for providing contextual recommendations in a vehicle, comprising: a system controller in communication with a plurality of onboard sensors adapted to collect real-time data related to the location of the vehicle and the operating status of the vehicle; a database in communication with the system controller, the database being adapted to store data relating to past actions and data relating to the location of the vehicle and the operating state of the vehicle when such past actions occurred; as well as the system controller comprising a driver-specific machine learning model adapted to predict a desired action based on the real-time data related to the position and operating state of the vehicle and data from the database; The system controller is further adapted to: initiating said predicted desired action; receiving input from an occupant of the vehicle; and The driver-specific machine learning model is updated.

Images