Intelligent travel companion

Abstract

A system for providing intent-based infotainment within a vehicle includes a system controller. The system controller is adapted to: acquire data relating to the intentions of an occupant within the vehicle; generate a request based on the data relating to the intentions of the occupant within the vehicle; acquire data from multiple onboard sensors relating to the request; receive data from a remote source relating to the request via a wireless communication module; formulate a response using a Large Language Model (LLM) communicating with the system controller via the wireless communication module; and drive the system within the vehicle to automatically perform at least one of the following: providing infotainment content to the occupant within the vehicle via a Human-Machine Interface (HMI); and controlling the operation of the vehicle via an Advanced Driver Assistance System (ADAS).

Description

Technical Field

[0001] This disclosure relates to a system and method for providing intent-based infotainment within a vehicle. Background Technology

[0002] Current in-vehicle infotainment systems are suitable for providing various communication, notification, and entertainment aspects for occupants. However, current systems are limited to providing static infotainment content that is specifically requested.

[0003] Therefore, while current systems and methods achieve their intended purpose, there is still a need for new and improved systems and methods for delivering infotainment content based on occupant intent, which is derived by analyzing data using large language models and machine learning algorithms to automatically deliver dynamic and interactive infotainment content. Summary of the Invention

[0004] According to several aspects of this disclosure, a method for providing intent-based infotainment within a vehicle is provided. The method includes: utilizing a system controller in communication with multiple onboard sensors within the vehicle; acquiring data relating to the intents of occupants within the vehicle; generating a request based on the data relating to the intents of the occupants within the vehicle; acquiring data from the multiple onboard sensors relating to the request; receiving data from a remote source relating to the request via a wireless communication module; formulating a response via the wireless communication module using a Large Language Model (LLM) communicating with the system controller; and driving a system within the vehicle to automatically perform at least one of: providing infotainment content to occupants within the vehicle via a Human-Machine Interface (HMI); and controlling the operation of the vehicle via an Advanced Driver Assistance System (ADAS).

[0005] According to another aspect, the collection of data related to the intentions of occupants in the vehicle also includes at least one of the following: collecting data related to what the occupant is observing, the occupant's gestures, and the occupant's facial expressions through an occupant monitoring system; and collecting the occupant's voice expressions and manual data inputs from the occupant through an HMI.

[0006] According to another aspect, collecting data related to the intentions of occupants in the vehicle also includes using machine learning models to access data stored in a database related to past events of occupants using the vehicle.

[0007] According to another aspect, generating a request based on data related to the intentions of the occupants in the vehicle also includes using a system controller to detect one of the following: triggering language from the occupants via an HMI; or triggering an event via multiple onboard sensors.

[0008] According to another aspect, generating requests based on data related to the intentions of occupants in the vehicle also includes: using a machine learning model to predict the occupant's intentions based on data stored in a database; using an LLM to analyze data related to the occupant's intentions in the vehicle; using a machine learning model and an LLM to identify keywords and qualifiers; and using a system controller to quantify keywords and qualifiers using an LLM, a machine learning model, real-time data collected by multiple onboard sensors, and data received from a remote source.

[0009] According to another aspect, collecting data from multiple vehicle-mounted sensors related to the request and receiving data from remote sources related to the request via a wireless communication module also includes: using a system controller to identify a geographical area in which relevant data related to the request will be collected; collecting relevant data related to the request within the geographical area; and filtering the collected data based on keywords and qualifiers.

[0010] According to another aspect, acquiring data from multiple vehicle-mounted sensors related to the request and receiving data from remote sources related to the request via a wireless communication module also includes using a system controller and a machine learning model to identify unspecified keywords and unspecified qualifiers related to the request but not included in the request.

[0011] According to another aspect, collecting data from multiple vehicle-mounted sensors related to the request and receiving data from a remote source related to the request via a wireless communication module also includes: using a system controller to identify an extended geographical area in which relevant data related to the request will be collected, wherein the relevant data related to the request includes unspecified keywords and unspecified qualifiers; collecting relevant data related to the request within the extended geographical area; and filtering the collected data based on keywords, qualifiers, unspecified keywords, and unspecified qualifiers.

[0012] According to another aspect, the system utilizes a large language model (LLM) communicating with the system controller via a wireless communication module to formulate responses and drive the system within the vehicle to automatically provide infotainment content to occupants via an HMI, further comprising: using the LLM to formulate natural language responses for occupants; and performing at least one of the following: displaying the text language response on the touchscreen display of the HMI; and broadcasting the voice response to occupants via a speaker associated with the HMI.

[0013] According to another aspect, the system utilizes a large language model (LLM) to formulate responses via a wireless communication module and drives the system within the vehicle to automatically control the operation of the vehicle via an advanced driver assistance system (ADAS). This also includes: using LLM and machine learning models to identify the required vehicle operation based on a request; and automatically executing the required vehicle operation via ADAS.

[0014] According to several aspects of this disclosure, a system for providing intent-based infotainment within a vehicle is provided. The system includes: a system controller communicating with a plurality of onboard sensors within the vehicle, and the system controller is adapted to: acquire data relating to the intents of occupants within the vehicle; generate a request based on the data relating to the intents of the occupants within the vehicle; acquire data from the plurality of onboard sensors relating to the request; receive data from a remote source relating to the request via a wireless communication module; formulate a response via the wireless communication module using a Large Language Model (LLM) communicating with the system controller; and drive the system within the vehicle to automatically perform at least one of the following operations: providing infotainment content to occupants within the vehicle via a Human-Machine Interface (HMI); and controlling the operation of the vehicle via an Advanced Driver Assistance System (ADAS).

[0015] According to another aspect, when collecting data related to the intentions of occupants in the vehicle, the system controller is also adapted to perform at least one of the following: collecting data related to what the occupant is observing, the occupant's gestures, and the occupant's facial expressions via the occupant monitoring system; and collecting the occupant's voice expressions and manual data inputs from the occupant via the HMI.

[0016] According to another aspect, when collecting data related to the intentions of occupants in the vehicle, the system controller is also adapted to: utilize machine learning models to access data stored in a database related to past events of occupants using the vehicle.

[0017] According to another aspect, when a request is generated based on data relating to the intentions of the occupants in the vehicle, the system controller is also adapted to detect one of the following: triggering language from the occupants via the HMI, or triggering an event via multiple onboard sensors.

[0018] According to another aspect, when generating a request based on data related to the intentions of occupants in the vehicle, the system controller is also adapted to: use a machine learning model to predict the occupant's intentions based on data stored in a database; use an LLM to analyze data related to the occupant's intentions in the vehicle; use a machine learning model and an LLM to identify keywords and qualifiers; and use the system controller to quantify keywords and qualifiers using an LLM, a machine learning model, real-time data collected by multiple onboard sensors, and data received from remote sources.

[0019] According to another aspect, when data is collected from multiple onboard sensors related to the request and data is received from a remote source related to the request via a wireless communication module, the system controller is also adapted to: identify a geographical area in which relevant data related to the request will be collected; collect relevant data related to the request within the geographical area; and filter the collected data based on keywords and qualifiers.

[0020] According to another aspect, when acquiring data from multiple onboard sensors related to the request and receiving data from a remote source related to the request via a wireless communication module, the system controller is also adapted to: use a machine learning model to identify unspecified keywords and unspecified qualifiers related to the request but not included in the request.

[0021] According to another aspect, when data is collected from multiple onboard sensors related to the request and data is received from a remote source related to the request via a wireless communication module, the system controller is also adapted to: identify an extended geographical area in which relevant data related to the request will be collected, wherein the relevant data related to the request includes unspecified keywords and unspecified qualifiers; collect relevant data related to the request within the extended geographical area; and filter the collected data based on keywords, qualifiers, unspecified keywords, and unspecified qualifiers.

[0022] According to another aspect, when a response is formulated using a Large Language Model (LLM) communicating with the system controller via a wireless communication module, and when a system within the vehicle is driven to automatically provide infotainment content to occupants via an HMI, the system controller is also adapted to: formulate natural language responses for occupants using the LLM; and perform at least one of the following: displaying the text language response on the touchscreen display of the HMI; and broadcasting the voice response to occupants via a speaker associated with the HMI; and when a response is formulated using a Large Language Model (LLM) communicating with the system controller via a wireless communication module, and when a system within the vehicle is driven to automatically control the operation of the vehicle via an Advanced Driver Assistance System (ADAS), the system controller is also adapted to: identify the required vehicle operation based on a request using the LLM and machine learning models; and automatically execute the required vehicle operation via ADAS.

[0023] Further applicability will become apparent from the description provided herein. It should be understood that the description and specific examples are for illustrative purposes only and are not intended to limit the scope of this disclosure. Attached Figure Description

[0024] The accompanying drawings described herein are for illustrative purposes only and are not intended to limit the scope of this disclosure in any way.

[0025] Figure 1This is a schematic diagram of a vehicle having a system for providing intent-based infotainment content, according to an exemplary embodiment.

[0026] Figure 2 This is a schematic diagram of a system according to an exemplary embodiment;

[0027] Figure 3 This is a schematic diagram illustrating the vehicle carrying passengers traveling on a road.

[0028] Figure 4 This is a schematic diagram showing the geographic area and extended geographic area in which the system will search for data related to the request; and

[0029] Figure 5 This is a flowchart illustrating a method according to an exemplary embodiment.

[0030] The drawings are not necessarily drawn to scale, and some features may be exaggerated or minimized, for example, to show details of specific components. In some cases, well-known components, systems, materials, or methods have not been described in detail to avoid obscuring the contents of this disclosure. Therefore, the specific structural and functional details disclosed herein should not be construed as limiting, but merely as a representative basis for the claims and for instructing those skilled in the art to apply the contents of this disclosure in various ways. Detailed Implementation

[0031] The following description is merely exemplary in nature and is not intended to limit the scope, application, or use of this disclosure. Furthermore, this disclosure is not intended to be bound by any express or implied theory set forth in the foregoing technical fields, background art, summary of the invention, or the following detailed description. It should be understood that throughout the drawings, corresponding reference numerals indicate the same or corresponding parts and features. As used herein, the term "module" refers to any hardware, software, firmware, electronic control components, processing logic, and / or processor device, individually or in any combination, including, but not limited to: application-specific integrated circuits (ASICs), electronic circuits, processors (shared, dedicated, or grouped) and memories executing one or more software or firmware programs, combinational logic circuits, and / or other suitable components providing the said functionality. Although the drawings shown herein depict examples with certain element arrangements, additional intermediate elements, devices, features, or components may be present in actual embodiments. It should also be understood that the drawings are merely illustrative and may not be drawn to scale.

[0032] As used herein, the term "vehicle" is not limited to automobiles. Although this article primarily describes the technology in relation to automobiles, the technology is not limited to automobiles. These concepts can be broadly applied to a variety of applications, such as those relating to aircraft, ships, other vehicles, and consumer electronics components.

[0033] Example embodiments are provided to make this disclosure more detailed and to fully communicate its scope to those skilled in the art. Numerous specific details, such as examples of specific compositions, components, apparatuses, and methods, are set forth to provide a thorough understanding of embodiments of this disclosure. It will be apparent to those skilled in the art that the example embodiments may be embodied in many different forms without requiring the specific details, and none of these forms should be construed as limiting the scope of this disclosure. In some example embodiments, well-known processes, well-known apparatus structures, and well-known techniques are not described in detail.

[0034] The terminology used herein is for the purpose of describing particular exemplary embodiments only and is not intended to be limiting. As used herein, the singular forms “a,” “an,” and “the” may also be intended to include the plural forms unless the context clearly indicates otherwise. The terms “comprising,” “containing,” “including,” and “having” are inclusive and thus specify the presence of the stated features, elements, components, steps, integrals, operations, and / or parts, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, parts, and / or groups thereof. Although the open-ended term “comprising” should be understood as a non-limiting term used to describe and claim the various embodiments described herein, in some respects it may also be understood as a more restrictive and binding term, such as “consisting of” or “substantially consisting of.” Therefore, for any given embodiment in which compositions, materials, components, elements, features, integrals, operations, and / or process steps are described, this disclosure also specifically includes embodiments consisting of or substantially consisting of the stated compositions, materials, components, elements, features, integrals, operations, and / or process steps. In the case of “consisting of…”, the alternative embodiments exclude any additional compositions, materials, parts, elements, features, integrals, operations and / or process steps, while in the case of “consisting substantially of…”, any additional compositions, materials, parts, elements, features, integrals, operations and / or process steps that have a substantial effect on the basic and novel features are excluded from such embodiments, but any compositions, materials, parts, elements, features, integrals, operations and / or process steps that do not have a substantial effect on the basic and novel features may be included in the embodiments.

[0035] Any methods, procedures, and operations described herein should not be construed as requiring performance in a specific order as discussed or described, unless explicitly indicated as such. It should also be understood that additional or alternative steps may be employed unless otherwise stated.

[0036] When a component, element, or layer is referred to as “on,” “joined to,” “connected to,” or “coupled to,” another component or layer, it may be directly on, joined to, or directly coupled to another component, element, or layer, or there may be intermediate elements or layers present. Conversely, when an element is referred to as “directly on,” “directly joined to,” “directly connected to,” or “directly coupled to,” another component or layer, there may be no intermediate elements or layers present. Other terms used to describe relationships between elements should be interpreted in a similar manner (e.g., “between” vs. “directly between,” “adjacent” vs. “directly adjacent,” etc.). As used herein, the term “and / or” includes any and all combinations of one or more of the listed associated items.

[0037] Although the terms first, second, third, etc., may be used herein to describe various steps, elements, components, regions, layers, and / or portions, these steps, elements, components, regions, layers, and / or portions should not be limited by these terms unless otherwise stated. These terms first, second, third, etc., are used only to distinguish one step, element, component, region, layer, or portion from another. Terms such as “first,” “second,” and other numerical terms used herein do not imply a sequence or order unless the context clearly indicates otherwise. Therefore, a first step, element, component, region, layer, or portion discussed below may be referred to as a second step, element, component, region, layer, or portion without departing from the teachings of the exemplary embodiments.

[0038] For ease of description, this document may use spatial or temporal relative terms, such as “before,” “after,” “inside,” “outside,” “below,” “below,” “down,” “above,” “above,” etc., to describe the relationship between an element or feature and another element(s) shown in the accompanying drawings. Spatial or temporal relative terms may be intended to cover different orientations of the device or system in use or operation other than those shown in the accompanying drawings.

[0039] Throughout this disclosure, numerical values ​​represent approximate measurements or range limits to cover minor deviations from a given value and embodiments having approximately the stated value as well as embodiments having the precise stated value. Except for the working examples provided at the end of the detailed specification, all numerical values ​​of parameters (e.g., quantities or conditions) in this specification (including the appended claims) should be understood to be modified in all cases by the term “about,” regardless of whether “about” actually appears before the numerical value. “About” indicates that the stated numerical value allows for slight inaccuracies (to some extent numerically close to the precise value; approximately or reasonably close to the value; approximate). If the inaccuracy provided by “about” cannot be otherwise understood in the art for its usual meaning, then “about” as used herein at least indicates a variation that may be caused by the usual methods of measuring and using these parameters. For example, in terms of percentages, “about” includes a variation of ±5%, in terms of temperature, “about” includes a variation of ±5 degrees, and in terms of distance (width, height, length), “about” includes ±10%. Furthermore, the disclosure of ranges includes disclosing all values ​​throughout the range and further subdivisions of the range, including endpoints and subranges given for the range. Furthermore, the disclosure of a range includes disclosing all values ​​within the entire range and further subdivisions of the range, including the endpoints and subranges given for that range.

[0040] According to an exemplary embodiment, Figure 1 The vehicle 10 is shown, which has a related system 50 for providing intent-based infotainment within the vehicle 10. Generally, system 50 works in conjunction with other systems within the vehicle 10 to display various information and provide infotainment content to the occupants of the vehicle 10. The vehicle 10 generally includes a chassis 12, a body 14, front wheels 16, and rear wheels 18. The body 14 is arranged on the chassis 12 and substantially surrounds the components of the vehicle 10. The body 14 and the chassis 12 can together form a frame. The front wheels 16 and the rear wheels 18 are rotatably connected to the chassis 12 and to corresponding corners of the body 14.

[0041] In various embodiments, vehicle 10 is an autonomous vehicle, and system 50 is incorporated into autonomous vehicle 10 and communicates with Advanced Driver Assistance Systems (ADAS) 52. Autonomous vehicle 10 is, for example, a vehicle 10 that is automatically controlled to transport occupants from one location to another. In the illustrated embodiment, vehicle 10 is depicted as a passenger car, but it should be understood that any other vehicle may be used, including secondary cars, motorcycles, trucks, sport utility vehicles (SUVs), recreational vehicles (RVs), aircraft, boats, etc. In exemplary embodiments, vehicle 10 is equipped with a so-called Level 4 or Level 5 automation system. Level 4 system means “high automation,” referring to the performance of the autonomous driving system in a specific driving mode, performing all aspects of a dynamic driving task, even if the human driver does not respond appropriately to an intervention request. Level 5 system means “full automation,” referring to the all-weather performance of the autonomous driving system in performing all aspects of a dynamic driving task under all road and environmental conditions manageable by a human driver. Novel aspects of this disclosure also apply to non-autonomous vehicles.

[0042] As shown in the figure, vehicle 10 generally includes a propulsion system 20, a drivetrain 22, a steering system 24, a braking system 26, a sensor system 28, an actuator system 30, at least one data storage device 32, a vehicle controller 34, and a wireless communication module 36. In embodiments where vehicle 10 is an electric vehicle, the drivetrain 22 may be absent. In various embodiments, the propulsion system 20 may include an internal combustion engine, an electric motor (e.g., a traction motor), and / or a fuel cell propulsion system. The drivetrain 22 is configured to transmit power from the propulsion system 20 to the front wheels 16 and rear wheels 18 of the vehicle according to an optional speed ratio. According to various embodiments, the drivetrain 22 may include a stepped automatic transmission, a continuously variable transmission (CVT), or other suitable transmission. The braking system 26 is configured to provide braking torque to the front wheels 16 and rear wheels 18 of the vehicle. In various embodiments, the braking system 26 may include friction brakes, brake-by-wire brakes, regenerative braking systems (e.g., electric motors), and / or other suitable braking systems. The steering system 24 influences the position of the front wheels 16 and rear wheels 18. Although depicted as including a steering wheel for illustrative purposes, in some embodiments contemplated within the scope of this disclosure, the steering system 24 may not include a steering wheel.

[0043] Sensor system 28 includes one or more vehicle-mounted sensors 40a-40n for sensing observable conditions of the external and / or internal environment of vehicle 10. Vehicle-mounted sensors 40a-40n may include, but are not limited to, radar, lidar, global positioning system, optical camera, thermal imaging camera, ultrasonic sensor, and / or other sensors. Cameras may include two or more digital cameras spaced apart from each other at a selected distance, wherein the two or more digital cameras are used to acquire stereoscopic images of the surrounding environment to obtain a three-dimensional image or map. Multiple vehicle-mounted sensors 40a-40n are used to determine information about the environment surrounding vehicle 10. In an exemplary embodiment, multiple vehicle-mounted sensors 40a-40n include at least one of a motor speed sensor, a motor torque sensor, an electric drive motor voltage and / or current sensor, an accelerator pedal position sensor, a coolant temperature sensor, a cooling fan speed sensor, and a transmission oil temperature sensor. In another exemplary embodiment, multiple vehicle-mounted sensors 40a-40n also include sensors for determining information about the environment surrounding vehicle 10, such as an ambient air temperature sensor, a barometric pressure sensor, and / or a photographic and / or video camera for observing the environment in front of vehicle 10. In another exemplary embodiment, at least one of the plurality of vehicle-mounted sensors 40a-40n is capable of measuring distances in the environment surrounding the vehicle 10.

[0044] In a non-limiting example, the plurality of vehicle-mounted sensors 40a-40n include cameras, and the plurality of vehicle-mounted sensors 40a-40n measure distances by using image processing algorithms configured to process images from the cameras and determine distances between objects. In another non-limiting example, the plurality of vehicle-mounted sensors 40a-40n include stereo cameras with distance measurement capabilities. In one example, at least one of the plurality of vehicle-mounted sensors 40a-40n is fixed inside the vehicle 10, for example, fixed in the roof lining of the vehicle 10, and can be observed through the windshield of the vehicle 10. In another example, at least one of the plurality of vehicle-mounted sensors 40a-40n is fixed outside the vehicle 10, for example, fixed on the roof of the vehicle 10, and can see the environment around the vehicle 10. It should be understood that various additional types of sensing devices, such as lidar sensors, ultrasonic ranging sensors, radar sensors, cameras and / or time-of-flight sensors, are all within the scope of this disclosure. The actuator system 30 includes one or more actuator devices 42a-42n that control one or more features of the vehicle 10, such as, but not limited to, the propulsion system 20, the transmission system 22, the steering system 24, and the braking system 26.

[0045] System 50 includes an occupant monitoring system 54 that receives data from at least one camera among a plurality of onboard sensors 40a-40n. The occupant monitoring system 54 is adapted to detect head and eye movements of occupants within the vehicle 10 to determine the direction and object being gazed at by the occupant. The occupant monitoring system 54 is also adapted to monitor gestures made by the occupant through head movements (e.g., nodding) or hand gestures.

[0046] The vehicle controller 34 includes at least one processor 44 and a computer-readable storage device or medium 46. The at least one data processor 44 can be any custom or commercially available processor, central processing unit (CPU), graphics processing unit (GPU), auxiliary processor among several processors associated with the vehicle controller 34, semiconductor-based microprocessor (in the form of a microchip or chipset), macroprocessor, any combination thereof, or generally any means for executing instructions. The computer-readable storage device or medium 46 can include volatile and non-volatile memory, such as read-only memory (ROM), random access memory (RAM), and keep-alive memory (KAM). KAM is persistent or non-volatile memory that can be used to store various operational variables when at least one data processor 44 is powered off. The computer-readable storage device or medium 46 may be implemented using any of a variety of known memory devices, such as PROM (programmable read-only memory), EPROM (electric PROM), EEPROM (electrically erasable PROM), flash memory, or any other electrical, magnetic, optical, or combined memory device capable of storing data, some of which represents executable instructions used by the vehicle controller 34 to control the vehicle 10 and the system 50.

[0047] These instructions may include one or more separate programs, each comprising an ordered list of executable instructions for implementing logical functions. When executed by at least one processor 44, these instructions receive and process signals from the sensor system 28, execute logic, calculations, methods, and / or algorithms for automatically controlling components of the vehicle 10, and generate control signals to the actuator system 30 to automatically control components of the vehicle 10 based on logic, calculations, methods, and / or algorithms. Although Figure 1 Only one vehicle controller 34 is shown, but embodiments of vehicle 10 may include any number of controllers 34 that communicate via any suitable communication medium or combination of communication media and cooperate in processing sensor signals, executing logic, calculations, methods and / or algorithms, and generating control signals to automatically control the features of autonomous vehicle 10.

[0048] In various embodiments, one or more instructions from the vehicle controller 34 are embodied in the trajectory planning system and, when executed by at least one data processor 44, generate a trajectory output that addresses the kinematic and dynamic constraints of the environment. For example, the instructions receive process sensor and map data as input. The instructions employ a graph-based approach, using a customized cost function to handle different road scenarios, including urban and highway scenarios.

[0049] The wireless communication module 36 is configured to wirelessly transmit information to and from other remote entities 48, such as, but not limited to, other vehicles (“V2V” communication), infrastructure (“V2I” communication), remote systems, remote servers, cloud computers, and / or personal devices. In an exemplary embodiment, the wireless communication module 36 is a wireless communication system configured to communicate using a wireless local area network (WLAN) of the IEEE 802.11 standard or using cellular data communication. However, additional or alternative communication methods (e.g., dedicated short-range communication (DSRC) channels) are also considered to be within the scope of this disclosure. A DSRC channel refers to a one-way or two-way short- to medium-range wireless communication channel designed specifically for automotive applications, along with a corresponding set of protocols and standards.

[0050] The vehicle controller 34 is a non-general-purpose electronic control device that includes a pre-programmed digital computer or processor, memory or non-transitory computer-readable medium for storing data (e.g., control logic, software applications, instructions, computer code, data, lookup tables, etc.), and a transceiver (or input / output port). Computer-readable medium includes any type of media accessible by a computer, such as read-only memory (ROM), random access memory (RAM), hard disk drive, optical disc (CD), digital video disc (DVD), or any other type of memory. "Non-transitory" computer-readable medium does not include wired, wireless, optical, or other communication links that transmit transient electrical or other signals. Non-transitory computer-readable medium includes media that can permanently store data and media that can store data and later overwrite it, such as rewritable optical discs or erasable storage devices. Computer code includes any type of program code, including source code, object code, and executable code.

[0051] refer to Figure 2 A schematic diagram of system 50 is shown. System 50 includes a system controller 34A that communicates with multiple onboard sensors 40a-40n, ADAS 52, occupant monitoring system 54, human-machine interface (HMI) 56, database 58, and wireless communication module 36. System controller 34A may be vehicle controller 34, or system controller 34A may be a separate controller that communicates with vehicle controller 34.

[0052] refer to Figure 3 The HMI 56 may include a touchscreen display 60 on which the system controller 34A can display infotainment content. Occupant 66 can interact with the system 50 through interaction with the touchscreen display 60 and / or through voice input picked up by a microphone 62 associated with the HMI 56. In an exemplary embodiment, a speaker 64 associated with the HMI 56 is adapted to allow the system to broadcast voice-activated infotainment content to occupant 66 within the vehicle 10. In another exemplary embodiment, the HMI 56 is associated with a head-up display within the vehicle 10 and communicates with the system controller 34A, which, in addition to displaying infotainment content on the touchscreen display 60 of the HMI 56, can also display infotainment content on the inner surface of the windshield of the vehicle 10 using the head-up display.

[0053] In an exemplary embodiment, system controller 34A is adapted to acquire data relating to the intentions of occupant 66 within vehicle 10. In acquiring such data, system controller 34A communicates with multiple sensors among multiple on-board sensors 40a-40n and occupant monitoring system 54 to acquire data relating to the intentions of occupant 66 within vehicle 10. System controller 34A utilizes occupant monitoring system 54 to acquire data relating to what occupant 66 is observing, gestures made by occupant 66, and facial expressions of occupant 66, and utilizes HMI 54 to acquire voice expressions made by occupant 66 and manual data input from occupant 66. System controller 34A also acquires basic data relating to credentials and subscriptions that occupant 66 may possess to determine which third-party service providers occupant 66 can access, and to determine vehicle operating parameters acquired by the multiple on-board sensors 40a-40n.

[0054] For example, the system controller 34A picks up voice data from the occupant 66 via the microphone 62 associated with the HMI 54, where the occupant 66 says, “Hey vehicle. Looking for a good hotel that offers optional breakfast and a gym, is about 30 to 45 minutes away, and not too far from the highway.”

[0055] In an exemplary embodiment, system controller 34A is adapted to drive system 50 when a trigger language is detected from occupant 66 via an HMI or when a trigger event is detected via multiple onboard sensors. In the example above, system controller 34A detects the trigger language “Hey, vehicle,” which triggers system controller 34A. Alternatively, system controller 34A may be triggered by an event, wherein system controller 34A drives the system to provide infotainment content based on the detection of an impending vehicle event (such as a left turn or approach to an intersection). A trigger event can be any ongoing or impending situation that system controller 34A recognizes as potentially requiring specific infotainment content from occupant 66. System controller 34A may utilize machine learning models and algorithms discussed in detail below to analyze and quantify potential trigger events.

[0056] Once triggered, system controller 34A recognizes the voice input from occupant 66: “Looking for a good hotel that offers optional breakfast and a gym, is about 30 to 45 minutes away, and not too far from the highway.” It identifies occupant 66’s intent to find a good hotel and establishes a request based on the voice input. In this example, the occupant provides a clear voice command to define the request. System controller 34A can also recognize occupant 66’s intent by utilizing machine learning model 68 to access data stored in database 58 related to past events of occupant 66’s use of vehicle 10. By using machine learning model 68 and machine learning algorithms, system controller 34A probabilistically predicts occupant 66’s intent based on past behavior.

[0057] Various techniques are employed to extract meaningful features from sensor readings and data, including time series analysis, frequency domain analysis, and spatiotemporal patterns. The machine learning model 68 can be one of, but is not limited to, Long Short-Term Memory (LSTM), Gated Recurrent Unit (GRU), Decision Tree, Random Forest, Support Vector Machine (SVM), Neural Network (NN), K-Nearest Neighbor (KNN), Gradient Boosting, and Recurrent Neural Network (RNN).

[0058] Therefore, the system controller 34A uses machine learning model 68 and machine learning technology to predict the current intention of occupant 66 based on real-time data of the location of vehicle 10, the operating conditions of vehicle 10 (date, time, weather conditions, speed, etc.) and occupant aspects (alertness, fatigue, distraction), and combined with data received from database 58 (including past events of occupant movement within vehicle 10, and occupant 66 preferences and actions when the location of vehicle 10, the operating conditions of vehicle 10, and occupant 66 aspects are the same as or substantially similar to the real-time data of the location of vehicle 10, the operating conditions of vehicle 10, and occupant 66 aspects).

[0059] Occupants inside a vehicle often exhibit repetitive patterns of behavior. By observing these patterns, machine learning model 68 can establish behavioral patterns and predict future behavior based on these patterns. This enables machine learning model 68 to predict the intentions of occupant 66.

[0060] To create the machine learning model 68, a general machine learning model was first trained using data collected from multiple different vehicles located in areas and climates similar to those of vehicle 10. Diverse datasets were collected from vehicles equipped with sensors such as GPS, accelerometers, cameras, radar, and lidar. The data covered various driving scenarios, including urban, highway, and off-road driving. Preprocessing steps were required before feeding the data into the machine learning model to remove noise, handle missing values, and standardize features. A crucial step in driving behavior classification is extracting relevant features from the raw data. As mentioned above, various techniques can be used to extract meaningful features from sensor readings, including time series analysis, frequency domain analysis, and spatiotemporal pattern analysis. Different types of machine learning algorithms were used for probabilistic pattern recognition, including but not limited to Long Short-Term Memory (LSTM), Gated Recurrent Unit (GRU), Decision Tree, Random Forest, Support Vector Machine (SVM), Neural Network (NN), K-Nearest Neighbors (KNN), Gradient Boosting, and Recurrent Neural Network (RNN). The general machine learning model was trained on a labeled dataset and evaluated using various performance metrics such as accuracy, precision, recall, F1 score, and confusion matrix. The model's hyperparameters were tuned to obtain optimal results. General machine learning models are trained on training data and learn to map input features to corresponding pattern (action) probabilities.

[0061] A generic machine learning model is uploaded to the system controller 34A within the vehicle 10. This generic machine learning model provides the foundation for creating driver-specific profiles and machine learning models 68 for specific occupants 66 of the vehicle 10. Uploading the generic machine learning model can be done through a subscription service from a third-party provider or the vehicle 10 manufacturer. Machine learning model 68 is ultimately created by updating the generic machine learning model. Once uploaded, data is collected during the daily use of the vehicle 10. As occupant 66 drives within the vehicle 10, the generic machine learning model is updated to personalize it for the specific occupant 66, thus creating machine learning model 68, which is customized for that specific occupant 66 and is continuously updated. The system controller 34A can store multiple machine learning models, each customized for a specific occupant, and whenever the system controller 34A identifies a new occupant of the vehicle 10 through the occupant monitoring system 54, the system controller 34A begins customizing a copy of the generic machine learning model to create a unique machine learning model for that occupant 66.

[0062] Referring to the example provided above, occupant 66 may not provide explicit commands, but system controller 34A can predict that occupant 66 might want to stay at a hotel soon based on the time of day, occupant 66's driving hours, and detection of occupant 66's fatigue. Furthermore, system controller 34A can predict that the occupant might want to eat breakfast and go to a gym based on previous data indicating that occupant 66 sometimes eats breakfast at hotels and always chooses hotels with gyms. Finally, system controller 34A can predict that the occupant wants a hotel near a highway based on navigation data showing that occupant 66 will be driving all day again the next day to reach the final destination and therefore wants an easy return to the highway. Therefore, system controller 34A can determine occupant 66's intention by directly translating speech or by manually inputting data, or by using machine learning model 68 and machine learning algorithms to infer occupant 66's intention using probabilistic prediction.

[0063] After collecting data related to the intentions of occupant 66, system controller 34A is adapted to generate a request based on the data related to the intentions of occupant 66 within vehicle 10. System controller 34A uses Large Language Model (LLM) 70 to analyze the data related to the intentions of occupant 66 within vehicle 10, and utilizes machine learning model 68 and LLM 70 to identify keywords and qualifying words for the request.

[0064] Large language models are artificial intelligence algorithms that apply neural network techniques with many parameters to process and understand human language or text using self-supervised learning. Tasks such as text generation, machine translation, summarizing, generating images from text, machine coding, chatbots, or conversational AI are all applications of large language models.

[0065] Large language models (MLMs) are purely based on deep learning methods, capable of efficiently capturing complex entity relationships within given text and generating text using the semantics and syntax of a specific language. Operating on deep learning principles, MLMs utilize neural network architectures to process and understand human language. These models employ self-supervised learning techniques for training on massive datasets. Their core functionality lies in learning complex patterns and relationships from diverse language data during training. MLMs consist of multiple layers, including feedforward layers, embedding layers, and attention layers. They employ attention mechanisms (such as self-attention) to measure the weight of different tokens in a sequence, allowing the model to capture dependencies and relationships.

[0066] The architecture of a Large Language Model (LLM) is determined by many factors, such as the specific model design goals, available computational resources, and the type of language processing task the LLM is intended to perform. Transformer-based models have revolutionized natural language processing tasks. These models typically follow a general architecture that includes components such as input embeddings, where the input text is labeled into smaller units, such as words or subwords, and each label is embedded in a continuous vector representation that captures the semantic and syntactic information of the input. Positional encoding is added to the input embeddings to provide information about the position of the labels, since the transformer does not naturally encode the order of the labels. This allows the Large Language Model to process labels while considering their order. The encoder uses neural network techniques to analyze the input text and creates many hidden states to preserve the context and meaning of the textual data. Multiple encoder layers form the core of the transformer architecture. Self-attention mechanisms and feedforward neural networks are two fundamental sub-components of each encoder layer. The self-attention mechanism allows the model to measure the weight of different labels in the input sequence by calculating attention scores. It allows the model to consider dependencies and relationships between different labels in a context-aware manner. After the self-attention step, the feedforward neural network is applied independently to each label. This network includes fully connected layers with non-linear activation functions, allowing the model to capture complex interactions between tags. In some transducer-based models, a decoder component is included in addition to the encoder. The decoder layer supports autoregressive generation, where large language models can generate sequential outputs by focusing on previously generated tags. Transducers typically employ multi-head attention mechanisms, where self-attention occurs simultaneously with different learned attention weights. This enables large language models to capture different types of relations and focus on various parts of the input sequence simultaneously. Layer normalization is applied after each sub-component or layer in the transducer architecture. It helps stabilize the learning process and improves the model's generalization ability across different inputs. The output layer of a transducer model can vary depending on the specific task. For example, in language modeling, a probability distribution for generating the next tag is often used, following a linear projection with SoftMax activation.

[0067] Large language models can generate accurate code based on user instructions for specific tasks, and assist in identifying code errors, suggesting fixes, and even automating project documentation. Users can ask large language models arbitrary and complex questions and receive detailed context-aware responses. Large language models can translate text between different languages ​​and correct grammatical errors. Large language models perform exceptionally well in one-shot and zero-shot learning scenarios.

[0068] The use of the large language model 70 enables the system controller 34A to engage in natural dialogue with the occupant 66. The large language model 70 enables the system controller 34A to understand and interpret voice input and text input (requests) and provide text or voice responses.

[0069] Therefore, the system controller 34A receives the request "Looking for a good hotel that offers optional breakfast and a gym, is about 30 to 45 minutes away, and not too far from the highway" using LLM 70 and machine learning model 68, and parses the request into keywords and qualifiers. From the request example above, the keywords include "hotel," "breakfast," "gym," and "highway." The qualifiers include "good," "optional," "30 to 45 minutes away," and "not too far."

[0070] Then, the system controller 34A uses LLM 70, machine learning model 68, real-time data collected by multiple onboard sensors 40a-40n, and data received from remote sources 48 (including crowdsourced databases, the Internet, and other data sources) to quantify keywords and qualifying terms.

[0071] For example, for the qualifier "good," system controller 34A interprets and translates it using LLM 70, understands passenger 66's personal preferences using machine learning model 68, and quantifies the meaning of "good" for passenger 66 by using data from remote sources to understand conventional views on what constitutes a "good" hotel, thus establishing standards. Therefore, system controller 34A determines that a "good" hotel must have a four-star rating or higher and at least 100 published reviews.

[0072] The interpretation and quantification of "optional" breakfast means that breakfast may or may not be included in the hotel room price, both of which are acceptable to the passengers. The interpretation and quantification of "approximately 30 to 45 minutes' drive" means that, calculated by system controller 34A using current vehicle speed, traffic conditions, etc., the hotel should be 40 to 60 miles from the current location of vehicle 10. Finally, the interpretation and quantification of "not too far" from the highway means that the hotel should be no more than 5 miles from the highway. Again, these determinations are based on probabilistic predictions made by machine learning models 68 and LLM 70, utilizing data related to the past behavior and preferences of passenger 66, as well as data related to traditional norms based on crowdsourced data.

[0073] The keywords and qualifiers of this request have now been quantified. The request will be transformed from:

[0074] "Looking for a good hotel that offers optional breakfast and a gym, is about 30 to 45 minutes away, and isn't too far from the highway."

[0075] Convert to:

[0076] "Looking for a hotel with at least 100 published reviews, a rating of at least four stars, a gym, and located within 60 miles of the vehicle's current location and within 5 miles of a highway."

[0077] After quantifying the keywords and qualifiers in the request, the system controller 34A is adapted to acquire data from multiple on-board sensors 40a-40n related to the request and receive data from a remote source 48 related to the request via the wireless communication module 36.

[0078] In an exemplary embodiment, system controller 34A is adapted to identify a geographical area in which relevant data relating to a request will be collected; to collect the relevant data relating to the request within the geographical area; and to filter the collected data based on keywords and qualifiers. (See reference) Figure 4 Using the example above, system controller 34A will identify circle 72, where the center 74 of circle 72 is located at the current position of vehicle 10, and the radius 76 of circle 72 is sixty miles, and will identify strip 78, which extends five miles on either side of highway 80. The geographic area 82 in which relevant data related to the request will be collected is defined as the area where circle 72 and strip 78 overlap. Therefore, system controller 34A will search within this geographic area 82 to find a suitable hotel for occupant 66.

[0079] In another exemplary embodiment, when data is acquired from multiple onboard sensors 40a-40n related to the request, and data is received from a remote source 48 related to the request via the wireless communication module 36, the system controller 34A is further adapted to: identify unspecified keywords and unspecified qualifiers related to the request but not included in the request by using a machine learning model 68. The machine learning model 68 will use real-time data on vehicle location, date, and occupant 66 preferences to identify the unspecified keywords and unspecified qualifiers.

[0080] For example, using the above request, occupant 66 did not mention that the hotel had a swimming pool. Therefore, the original request did not contain keywords or qualifiers related to the existence of a swimming pool or specific information about the pool. However, machine learning model 68 identified a pattern in database 58 where occupant 66 stays at hotels with swimming pools most of the time, and there are multiple instances where occupant 66 uses system 50 and explicitly requests that the hotel has a swimming pool. Therefore, using machine learning model 68, system controller 34A will include the unspecified word "swimming pool" in the request. Furthermore, using data from multiple weather / temperature related sensors and data from remote sources, machine learning model 68 will identify that a forecast is predicting cold weather, and using data from database 58, machine learning model 68 will identify a pattern where occupant 66 stays at hotels with indoor swimming pools when it is cold outside. Therefore, using machine learning model 68, system controller 34A will include the unspecified qualifier "indoor" together with the unspecified keyword "swimming pool" and include "indoor swimming pool" in the request.

[0081] In another example, using data from a remote source, system controller 34A identifies a new hotel with an indoor water park. This hotel was not yet built when passenger 66 passed through the area on previous occasions, and therefore was not an option. Therefore, system controller 34A identifies potential keywords that passenger 66 might want to include in their request. Subsequently, system controller 34A can either automatically include "indoor water park" as an unspecified keyword, or prompt passenger 66 about their preference for an indoor water park. If this is the first time, database 58 will not contain any data related to an indoor water park, so system controller 34A can prompt passenger 66 not only to enter their preference for an indoor water park, but also to enter their preference for ranking indoor water parks compared to other keywords.

[0082] In this way, even if the collected data and the behavior displayed by occupant 66 do not indicate the occupant's intentions to system controller 34A, including unspecified keywords and unspecified qualifiers, system controller 34A can fine-tune the request based on occupant 66's preferences. This allows system 50 to more accurately interpret the occupant's intentions and take into account new data that was previously unavailable, thereby providing a more acceptable response to occupant 66.

[0083] In another exemplary embodiment, when data is collected from multiple vehicle-mounted sensors 40a-40n related to the request, and data is received from a remote source 48 related to the request via the wireless communication module 36, the system controller 34A is further adapted to identify an extended geographic area 84, in which relevant data related to the request will be collected, wherein the relevant data related to the request includes unspecified keywords and unspecified qualifying terms. (See again) Figure 4 Using navigation data and machine learning model 68, system controller 34A predicts that after spending the night at a hotel, occupant 66 will travel along the highway in vehicle 10, as indicated by arrow 86. System controller 34A will define an extended geographic area 84, located further than the qualifier "forty to sixty miles away," but potentially including better hotel options that meet the requirements of other keywords, qualifiers, unspecified keywords, or unspecified qualifiers. Furthermore, despite being further away, this extended geographic area 84 is located on occupant 66's expected future route and is therefore likely to be more appealing to occupant 66.

[0084] In an exemplary embodiment, by using LLM 70 and machine learning model 68, system controller 34A is adapted to rank keywords and qualifiers by predicting which keywords and qualifiers are more important to occupant 66. Thus, referring again to the example above, the request includes the words "gym" and "within forty to sixty miles," however, based on past data, occupant 66 exhibits a pattern of always choosing hotels that offer gyms. Therefore, system controller 34A ranks the presence of gyms above distance items and uses machine learning model 68 to predict how many miles occupant 66 is willing to drive to reach a hotel that offers a gym. Therefore, if no hotel offering a gym is found in the search within geographic region 80, system controller 34A will search within extended geographic region 84 to see if hotels further away offer gyms. It should be understood that the ranking of keywords and qualifiers may be based on occupant 66's preferences identified by machine learning model 68, or on environmental or vehicle operating parameters, such as a preference for paved highways over scenic driving along rural roads when weather conditions make driving less than ideal.

[0085] After the system controller 34A collects relevant data related to geographic region 82 and extended geographic region 84, the system controller collectively filters all data based on keywords, qualifying words, unspecified keywords, and unspecified qualifying words. The system controller 34A uses LLM 70 via wireless communication module 36 to formulate a response and drive the system in vehicle 10 to automatically perform at least one of the following operations: providing infotainment content to occupants 66 in vehicle 10 via HMI 56; and controlling the operation of vehicle 10 via Advanced Driver Assistance System (ADAS).

[0086] In an exemplary embodiment, when the LLM 70 is used to formulate a response and drive the system within vehicle 10 to automatically provide infotainment content to occupant 66 via HMI 56, system controller 34A is also adapted to perform one of the following operations: formulate a natural language response for occupant 66 using the LLM 70, display a text-based language response at least on the touchscreen display 60 of HMI 56, and broadcast a voice response to occupant 66 via speaker 64 associated with HMI 56. For example, in response to a request “finding a good hotel that offers optional breakfast and a gym, is about 30 to 45 minutes away, and is not too far from the highway,” system controller 34A can identify multiple matching hotels located within geographic region 82 and meeting occupant 66’s preferences for identified keywords, qualifiers, unspecified keywords, and unspecified qualifiers. The response could be a list of identified matching hotels displayed to occupant 66 on the touchscreen display 60 of the HMI, which occupant 66 can view and select from. Optionally or in combination, the system controller 34A formulates a natural language response using the LLM 70, wherein the list of matching hotels is “read aloud” to the occupant 66 via the speaker 64 of the HMI 56. At any time while the hotel list is being “read aloud,” the occupant 66 can interrupt by selecting one of the matching hotels via voice.

[0087] In another exemplary embodiment, when the LLM 70 is used to formulate a response and drive the system within vehicle 10 to automatically control the operation of vehicle 10 via ADAS, system controller 34A is also adapted to utilize the LLM 70 and machine learning model 68 to identify the required vehicle operation based on the request and to automatically execute the required vehicle operation via ADAS. For example, in response to the request “finding a good hotel that offers optional breakfast and a gym, is about 30 to 45 minutes away, and is not too far from the highway,” system controller 34A can identify multiple matching hotels located within geographic region 82 and satisfying the preferences of occupant 66 for identified keywords, qualifiers, unspecified keywords, and unspecified qualifiers. By using machine learning model 68 and machine learning techniques, system controller 34A can rank the identified matching hotels and automatically select the highest-ranked hotel among the multiple matching hotels, wherein system controller 34A automatically drives the autonomous driving function of ADAS 52 to provide autonomous driving navigation for vehicle 10, navigating to the highest-ranked hotel among the multiple matching hotels. Simultaneously, the system controller 34A can display a list of multiple identified matching hotels on the touchscreen display 60 of the HMI 56, allowing the occupant 66 to view the list and see the highest-ranked hotel among the identified matching hotels. If the occupant 66 wishes, the occupant 66 can provide manual or voice commands to go beyond the automatic selection made by the system controller 34A, and select different hotels among the multiple matching hotels by voice or manual. In this case, the system controller will drive the ADAS 52 to autonomously navigate the vehicle 10 to the hotel selected by the occupant 66.

[0088] By using LLM 70 and machine learning model 68, system controller 34A can provide a wide variety of infotainment content to occupants 66 or occupants within vehicle 10. As in the example above, system 50 can recognize a specific command given by occupant 66, interpret the command using machine learning and LLM 70, and provide content including options that occupant 66 can choose from. In other scenarios, by using multiple onboard sensors 40a-40n and the occupant monitoring system 54, the system controller 34A can identify data interpreted by the LLM 70 as that of an anxious child within the vehicle 10. Using the LLM 70, the system controller interprets this data as the intention of the occupant 66 and a request for entertaining informational entertainment for the child. The responses proposed by the system controller 34A include: accessing the occupant 66's (parent driver's) subscription to a book or movie service (remote entity 48) via the wireless communication module 36; and selecting a story or movie based on the child's age and interests (based on prior data, the machine learning model 68, and the database 58) using the machine learning model 68 and the LLM 70. By utilizing the LLM 70, the system controller 34A will automatically display a movie or "read aloud" a story to the child using the HMI 56.

[0089] By using LLM 70 and data collected from crowdsourced remote entity 48, system 50 can provide qualitative responses to occupant 66's requests. For example, occupant 66 can verbally generate a request, "Hey vehicle, compare my driving performance on this trip with my parents' driving performance and general behavior." System controller 34A, together with LLM 70 and machine learning model 68, will identify keywords such as "ranking," "my driving," "comparison," "parents," and "general behavior," as well as qualifiers such as "this trip." Specifically, system controller 34A uses data from LLM 70 and best behavior from crowdsourced remote entity 48 to quantify the occupant's ranking as good, average, or poor, or alternatively, from one to ten, where one is best and ten is poor, using data on occupant 66's driving characteristics, such as lane-changing behavior, accurate lane and / or route following, compliance with traffic rules, and obedience to traffic signals. This driving characteristic data is collected by multiple onboard sensors 40a-40n within vehicle 10. The system controller 34A will further compare the quantified data of occupant 66 with similar quantified data collected and stored in database 58 when occupant 66's parents are driving / driving vehicle 10. The system controller 34A will quantify the occupant's level as good, average, or poor, or rank it from one to ten, by comparing it to the behavior of occupant 66's parents. The qualifier "this trip" means that occupant 66 only wants to know their ranking of driving behavior in the current driving event (from the last drive until vehicle 10 is parked again). Therefore, the system controller 34A will only use the data of occupant 66 collected during the current driving event, and not the data of occupant 66 collected during the current driving event, nor the data from past driving events of occupant 66 extracted from database 58.

[0090] refer to Figure 5A method 100 for providing intent-based infotainment within a vehicle 10, the method comprising: utilizing a system controller 34A communicating with multiple on-board sensors 40a-40n within the vehicle 10; starting from block 102, acquiring data relating to the intent of an occupant 66 within the vehicle 10; moving to block 104, generating a request based on the data relating to the intent of the occupant 66 within the vehicle 10; moving to block 106, acquiring data from the multiple on-board sensors 40a-40n relating to the request; and moving to block 108, via a wireless communication module 36 Receive data from a remote source 48 in relation to the request; move to box 110 to formulate a response via wireless communication module 36 using a large language model (LLM) 70 communicating with system controller 34A; and move to box 112 to drive the system vehicle 10 within the vehicle 10 to automatically perform at least one of the following operations: move to box 114 to provide infotainment content to occupants 66 within the vehicle 10 via human-machine interface (HMI) 56, and move to box 116 to control the operation of the vehicle 10 via advanced driver assistance system (ADAS) 52.

[0091] In an exemplary embodiment, at block 102, the acquisition of data relating to the intentions of the occupant 66 within the vehicle 10 further includes at least one of the following: using the occupant monitoring system 54 to acquire data relating to what the occupant 66 is observing, gestures made by the occupant 66, and facial expressions of the occupant 66; and using the HMI 56 to acquire voice expressions made by the occupant 66 and manual data inputs from the occupant 66.

[0092] In another exemplary embodiment, at block 102, collecting data relating to the intentions of the occupant 66 in the vehicle 10 further includes using a machine learning model 68 to access data stored in a database 58 relating to past events of the occupant 66’s use of the vehicle 10.

[0093] In another exemplary embodiment, at block 104, generating a request based on data relating to the intentions of an occupant 66 within the vehicle 10 further includes using a system controller to detect one of the following: triggering a speech from the occupant 66 via an HMI 56; or triggering an event via multiple onboard sensors 40a-40n.

[0094] In another exemplary embodiment, at block 104, generating a request based on data relating to the intentions of an occupant 66 within the vehicle 10 further includes: using a machine learning model 68 to predict the intentions of the occupant 66 based on data stored in a database 58; using an LLM 70 to analyze data relating to the intentions of the occupant 66 within the vehicle 10; using the machine learning model 68 and the LLM 70 to identify keywords and qualifying words; and using a system controller 34A to quantify the keywords and qualifying words by using the LLM 70, the machine learning model 68, real-time data collected by multiple onboard sensors 40a-40n, and data received from a remote source 48.

[0095] In another exemplary embodiment, at block 106, request-related data is collected from multiple vehicle sensors 40a-40n, and at block 108, request-related data is received from a remote source 48 via a wireless communication module 36. The method also includes: using a system controller 34A to identify a geographic region 82 in which request-related data will be collected; collecting request-related data within geographic region 82; and filtering the collected data based on keywords and qualifiers.

[0096] In another exemplary embodiment, at block 106, request-related data is collected from multiple vehicle sensors 40a-40n, and at block 108, data is received from a remote source related to the request via wireless communication module 36. The method also includes using system controller 34A to identify unspecified keywords and unspecified qualifiers related to the request but not included in the request by using machine learning model 68.

[0097] In another exemplary embodiment, at block 106, request-related data is collected from multiple vehicle sensors 40a-40n, and at block 108, data is received from a request-related remote source 48 via wireless communication module 36. The method further includes: using system controller 34A to identify an extended geographic area 84 in which request-related data will be collected, wherein the request-related data includes unspecified keywords and unspecified qualifiers; collecting request-related data within the extended geographic area 84; and filtering the collected data based on keywords, qualifiers, unspecified keywords, and unspecified qualifiers.

[0098] In another exemplary embodiment, at block 110, a response is formulated using a Large Language Model (LLM) 70 communicating with a system controller 34A via a wireless communication module 36, and at block 114, driving a system within the vehicle 10 to automatically provide infotainment content to an occupant 66 within the vehicle 10 via an HMI 56 further includes: formulating a natural language response for the occupant 66 using the LLM 70; and performing at least one of the following operations: displaying the text language response on a touchscreen display 60 of the HMI 56; and broadcasting the voice response to the occupant 66 via a speaker 64 associated with the HMI 56.

[0099] In yet another exemplary embodiment, at block 110, a response is formulated using a Large Language Model (LLM) 70 communicating with a system controller 34A via a wireless communication module 36, and at block 116, a system driven within the vehicle 10 is used to automatically control the operation of the vehicle 10 via an Advanced Driver Assistance System (ADAS) 52, further including: using the LLM 70 and a machine learning model 68 to identify the desired vehicle operation based on a request; and automatically performing the desired vehicle operation via ADAS 52.

[0100] The system 50 and method 100 of this disclosure have the following advantages: interpreting the intention of the occupant 66 by using a large language model 70, and providing explicitly requested infotainment content based on the occupant 66's intentions by using a machine learning model 68 to predict the occupant 66's intentions based on past behavior, or inferring infotainment content based on collected data related to the occupant 66's intentions; and generating requests based on the occupant 66's intentions by using the large language model 70 and the machine learning model 68, and formulating responses to provide infotainment content to the occupant 66 and / or providing autonomous vehicle 10 control in response to the occupant 66's intentions.

[0101] The descriptions in this disclosure are merely exemplary in nature, and any modifications that do not depart from the spirit and scope of this disclosure are intended to fall within its scope. Such modifications should not be considered as departing from the spirit and scope of this disclosure.

Claims

1. A method for providing intent-based infotainment within a vehicle, comprising utilizing a system controller in communication with a plurality of onboard sensors within the vehicle: Collect data relating to the intentions of the occupants inside the vehicle; The request is generated based on the data relating to the intentions of the occupants inside the vehicle. Data is collected from the plurality of vehicle-mounted sensors associated with the request; Data is received from a remote source related to the request via a wireless communication module; The response is formulated using the Large Language Model (LLM) communicated with the system controller via the wireless communication module; and The system within the vehicle is driven to automatically perform at least one of the following: To provide infotainment content to the occupants of the vehicle via a human-machine interface (HMI); and The vehicle's operation is controlled by an Advanced Driver Assistance System (ADAS).

2. The method according to claim 1, wherein, The collection of data relating to the intentions of occupants within the vehicle also includes at least one of the following operations: The occupant monitoring system collects data related to what the occupant is observing, the occupant's gestures, and the occupant's facial expressions. as well as The HMI is used to collect the occupant's verbal expressions and manual data inputs.

3. The method according to claim 2, wherein, The collection of data relating to the intentions of the occupants in the vehicle also includes using a machine learning model to access data stored in a database relating to past events of the occupants' use of the vehicle.

4. The method according to claim 3, wherein, The process of generating a request based on the data relating to the intentions of the occupants within the vehicle further includes: The system controller is used to detect one of the following: The HMI is used to trigger language from the occupant; or Events are triggered by the aforementioned multiple vehicle-mounted sensors.

5. The method according to claim 4, wherein, The process of generating a request based on the data relating to the intentions of the occupants within the vehicle further includes: Using the machine learning model, the occupant's intention is predicted based on the data stored in the database; The LLM is used to analyze the data related to the intentions of the occupants inside the vehicle; The machine learning model and the LLM are used to identify keywords and qualifiers; and The system controller is used to quantify the keywords and qualifying terms using the LLM, the machine learning model, real-time data collected by the multiple onboard sensors, and data received from remote sources.

6. The method according to claim 5, wherein, The acquisition of data from the plurality of vehicle-mounted sensors in relation to the request and the receipt of data from a remote source in relation to the request via the wireless communication module further include: The system controller is used to identify the geographical area in which relevant data related to the request will be collected. Collect relevant data related to the request within the geographical area; and The collected data is filtered based on the keywords and qualifiers.

7. The method according to claim 6, wherein, The acquisition of data from the plurality of vehicle-mounted sensors in relation to the request and the receipt of data from a remote source in relation to the request via the wireless communication module further include: Using the system controller, the machine learning model is used to identify unspecified keywords and unspecified qualifiers that are related to the request but not included in the request.

8. The method according to claim 7, wherein, The acquisition of data from the plurality of vehicle-mounted sensors in relation to the request and the receipt of data from a remote source in relation to the request via the wireless communication module further include: The system controller is used to identify an extended geographical area in which relevant data relating to the request will be collected, wherein the relevant data relating to the request includes unspecified keywords and unspecified qualifiers; Collect relevant data related to the request within the extended geographical area; and The collected data is filtered based on the keywords, qualifiers, unspecified keywords, and unspecified qualifiers.

9. The method according to claim 8, wherein, The method of using the Large Language Model (LLM) to formulate responses via the wireless communication module and communicating with the system controller, and the method of driving the system within the vehicle to automatically provide infotainment content to the occupants within the vehicle via the HMI, further includes: Utilize the LLM to develop a natural language response for the occupant; and perform at least one of the following: The text language response is displayed on the touchscreen display of the HMI; and A voice response is broadcast to the occupants via a speaker associated with the HMI.

10. The method according to claim 9, wherein, The method of formulating responses using a Large Language Model (LLM) in communication with the system controller via the wireless communication module, and the system driving the vehicle to automatically control the operation of the vehicle via an Advanced Driver Assistance System (ADAS), further includes: Using the LLM and the machine learning model, the required vehicle operation is identified based on the request; and The required vehicle operations are performed automatically through the ADAS.

Images