Personalized advertising system for a vehicle

Abstract

Systems and methods are disclosed for predicting when to display an advertisement to a driver of a vehicle, based on environmental sensor data and driver status data from a driver monitoring system (DMS) of the vehicle during operation of the vehicle by a driver. In particular, the environmental sensor data and driver status data may be reformatted into a plurality of numerical scores associated with different groupings of elements of the environmental sensor data and driver status data, and inputted into a machine learning (ML) model that is trained to output a prediction of a suitability of conditions for displaying the advertisement. The ML model may be trained at a remote training system, based on supervised learning, where explicit driver input / feedback on the timing of the display of the ad is used as ground truth data.

Description

PERSONALIZED ADVERTISING SYSTEM FOR A VEHICLEFIELD

[0001] The disclosure relates to a customized deliver}’ of advertising content to a driver of a vehicle.BACKGROUND

[0002] In modem connected vehicles, in-vehicle infotainment (IVI) systems may be configured to display video content to a driver of a vehicle. For example, the video content may be displayed on a dashboard display screen of the vehicle. The video content may include information about one or more systems of the vehicle; information about environmental settings of a cabin of the vehicle, including information about music playing in the cabin, cabin temperature, etc.; information about an external environment of the vehicle, such as traffic, weather, or route information; and / or other types of information. Further, the IVI system may be connected to a wireless network, such as a cell phone network, that provides Internet access. As a result, the IVI system can be configured to display advertisements to the driver. The advertisements may include, for example, ads for subscription or on-demand services, such as streaming music or other audio content, roadside assistance sendees, or in-cabin features; ads for roadside services or attractions such as food, gas, hotels, etc.; or other types of sendees or merchandise. However, the advertisements may be distracting to the driver, and may cause the driver to look away from the road. Current methods for determining when it is appropriate to display an ad to the driver may be inadequate.SUMMARY

[0003] In various embodiments, the issues described above may be addressed by a machine-implemented method, comprising collecting environmental sensor data of a vehicle during operation of the vehicle by a driver; collecting driver status data from a driver monitoring system (DMS) of the vehicle; reformatting the collected environmental sensor data and the driver status data to reduce a size and / or dimensionality of the collected environmental sensor data and the driver status data; predicting a suitability’ of conditions for displaying an advertisement to the driver, based on the reformatted environmental sensor data and driver status data, using a machine learning (ML) model; and in response to the predicted suitability exceeding a threshold suitability, displaying the advertisement to the driver.

[0004] It should be understood that the summary above is provided to introduce in simplified form a selection of concepts that are further described in the detailed description. It is not meant to identify key or essential features of the claimed subject matter, the scope of which is defined uniquely by the claims that follow the detailed description. Furthermore, the claimed subject matter is not limited to implementations that solve any disadvantages noted above or in any part of this disclosure.BRIEF DESCRIPTION OF THE DRAWINGS

[0005] The disclosure may be better understood from reading the following description of non-limiting embodiments, with reference to the attached drawings, wherein below:

[0006] FIG. 1 is a schematic block diagram of a vehicle control system including an advertisement delivery7system, in accordance with one or more embodiments of the present disclosure;

[0007] FIG. 2 is a schematic block diagram that shows examples of data that may be received as input into an advertisement timing model of the advertisement delivery system, in accordance with one or more embodiments of the present disclosure;

[0008] FIG. 3 is a schematic block diagram showing inputs into the advertisement delivery system, in accordance with one or more embodiments of the present disclosure;

[0009] FIG. 4 is a diagram showing a vehicle in communication with a cloud-based ad timing training system used to train the advertisement timing model, in accordance with one or more embodiments of the present disclosure;

[0010] FIG. 5 is a schematic block diagram showing an exemplary7flow of data during training of the advertisement timing model, in accordance with one or more embodiments of the present disclosure;

[0011] FIG. 6 shows an exemplary dashboard of a vehicle including a plurality of controls, in accordance with one or more embodiments of the present disclosure;

[0012] FIG. 7 is a schematic block diagram that shows an in-vehicle computing sy stem and a control system of a vehicle, in accordance with one or more embodiments of the present disclosure;

[0013] FIG. 8 is a flowchart illustrating a machine-implemented method for determining when to display an advertisement to a driver of a vehicle using an advertisement timing model, in accordance with one or more embodiments of the present disclosure: and

[0014] FIG. 9 is a flowchart illustrating a machine-implemented method for training an advertisement timing model, in accordance with one or more embodiments of the present disclosure.DETAILED DESCRIPTION

[0015] In modem connected vehicles, in-vehicle infotainment (IVI) systems may be configured to display advertisements (also referred to herein as ads) to a driver of a vehicle while the driver is operating the vehicle. For example, the ads may be displayed on a dashboard display screen of the vehicle. However, the ads may be distracting to the driver, and may cause the driver to look away from the road at an inopportune moment. To display the ads to the driver at a suitable time, data about the ads. the driver, the vehicle, and the environment may be collected and analyzed by an advertising del i \ erx system of the vehicle. The advertising delivery system may display an ad if the data indicates that certain environmental and / or driving conditions are met, and / or if a psychological and / or emotional conditions of the driver are met.

[0016] The data may include dnver data collected from a driver monitoring system (DMS). which may be used to estimate a physiological state (such as a drowsiness or a distraction) of the driver. A DMS may be based on sensors including contactless sensors that analyze various aspects or parameters of driver behavior, such as, for example, head movement, eye gaze, and / or other similar aspects or parameters of driver behavior that may indicate or relate to the physiological state of the driver. For example, a first set of patterns in head movement and / or eye gaze data may indicate that the driver may be drowsy, a second set of patterns in the head movement and / or eye gaze data may indicate that the driver may be distracted, and so forth. Next generation DMS systems extend the functionality in considering cognitive aspects or moods of the driver based on sensor data.

[0017] For example, at a first time, data collected from vehicle systems, external cameras, and other vehicle sensors may indicate that the driver is operating the vehicle on a straight road without traffic, and driver data collected from a DMS of the vehicle may be used to estimate a low level of stress or anxiety of the driver. Consequently, the advertising delivery system may determine that conditions (e.g., driving conditions, vehicle conditions, driver conditions, weather conditions, traffic conditions, etc.) at the first time are suitable for displaying an ad, whereby the advertising delivery' system may display the advertisement to the driver on the dashboard display screen. At a second time, the collected data may indicate that the driver is operating the vehicle in traffic, and may be experiencing stress or anxiety', whereby theadvertising delivery' system may determine that conditions at the second time are not suitable for displaying an ad, and the advertising delivery system may not display the advertisement to the driver. At a third time, the collected data may indicate that the driver is following a route on an onboard navigation system and is periodically looking at a map of the route on the dashboard display, whereby the advertising delivery' system may determine that conditions at the third time are not suitable for displaying an ad, and the advertising delivery system may not display the advertisement to the driver. In this way, the collected data may be used to estimate the suitability' of conditions at a current time for displaying an ad based on a variety of different factors.

[0018] To determine whether the conditions at the current time are suitable for displaying an ad, the advertisement delivery system may rely on one or more artificial intelligence (Al) based models that analyze the data of a plurality of drivers, from a plurality of vehicles, in a plurality of driving conditions. In various embodiments, the one or more Al-based models may include a machine learning (ML) model, such as a convolutional neural network (CNN). However, given the wide variety’ of interacting factors influencing a prediction as to when to display an ad, training the ML model to make accurate predictions about when to display the ad may' rely on collecting, storing, and transmitting a large amount of training data to a training system (e.g., a cloud-based training system). Due to limitations on memory', processing, and bandwidth resources, developing a model with acceptable performance under a variety of driving conditions may be unfeasible.

[0019] A bigger problem is that training an ML model using supervised learning may rely on generating training pairs including ground truth data. Each training pair may include a set of input data (e.g., the collected driver and driving condition data), and a ground truth assessment of an appropriateness of displaying an ad based on the input data. The training pairs may include positive examples, when the input data suggests that an ad may be displayed, and negative examples, when the input data suggests that an ad should not be displayed. However, accurately assigning ground truth assessments to the input data may be difficult, since the positive and the negative examples may be based on the occurrence or non-occurrence of negative feedback to an ad display, which may be difficult to assess and objectively undesirable under driving conditions.

[0020] To address this issue, an advertisement delivery' system is disclosed herein that delivers ads to the driver in a manner such that explicit feedback regarding the ads may be provided by the driver. In various embodiments, the driver may be incentivized to provide feedback for the ad, for example, to earn discounts on services, points towards vehicle features,etc. After an ad has been displayed, the driver may provide feedback in a variety of ways, such as via a control element in a graphical user interface (GUI) of a dashboard display, via a control element such as a button on a steering wheel of the vehicle, by making verbal statements that are recorded by a microphone of the vehicle, or in a different manner. A training data pair for training the ML model may be generated in response to the feedback, where input data of the training data pair may include sensor data of the vehicle, environmental sensor data, DMS data, driver profile data, and the like. The training data pair may include an interpretation of the feedback as ground truth. For example, a numeric success score may be assigned to the ad based on the feedback. The training pair data may be used to train the ML model to learn when to predict when to display ads to drivers during operation of the vehicle.

[0021] Additionally, systems and methods are described for reformatting the input data to reduce a dimensionality and a size of the input data with respect to the raw environmental sensor data, DMS data and driver profile data. By reducing the dimensionality and size of the input data, the input data may be more efficiently stored and transmitted to an ad timing training system not located at the vehicle. The ML model may also be trained on the reformatted input data in a faster and more efficient manner than on the raw data.

[0022] Reformatting the input data may include grouping the environmental sensor data, the DMS data, and the driver profile data into different categories, and generating ratings or scores for the different categories that may be used as input into the ML model. In some examples, one or more models may be applied to the environmental sensor data, the DMS data, and / or the driver profile data to determine a relative contribution of different elements of the data in each group. The relative contributions may be used to assign numeric scores or ratings to each of the different groupings. For example, a first set of environmental sensor data may be assessed and interpreted to generate a first score characterizing weather and road conditions of the vehicle; a second set of environmental sensor data may be assessed and interpreted to generate a second score characterizing traffic conditions of the vehicle; a third set of DMS data may be assessed and interpreted to generate a third cognitive score characterizing a degree of distractedness of the driver; a fourth set of DMS data may be assessed and interpreted to generate a fourth cognitive score characterizing a degree of anxiety of the driver; and so on. The different scores generated for the different categories may be included in the input data, and the raw environmental sensor data and DMS data may be discarded and not stored, reducing an amount of memory' usage of the ad delivery system.

[0023] Further, in some embodiments, the advertisement delivery’ system may deliver ads to a queue, and driver may be notified when an ad is placed in the queue or with respect to anumber of ads in the queue. When driver conditions, environmental conditions, and driving conditions are met, the driver may wish to display the ad. The driver may select an ad for display, for example, via a control element of the GUI, or by voice, or via the button on the steering wheel, or in a different manner, and the ad may be displayed. In such cases, a training data pair for training the ML model may be generated in response to the feedback, where input data of the training data pair may include sensor data of the vehicle, environmental sensor data, DMS data, and the like, and a high success score may be assigned as ground truth, given that the driver has explicitly communicated that it is a good time to display the ad.

[0024] The training data pairs may be transmitted to the ad timing training system, where the ML model may be trained on a set of training data pairs collected from a plurality of drivers and a plurality of vehicles. For example, the training data pairs may be transmitted wirelessly to a cloud-based server hosting the ad timing training system. The training data pairs may be transmitted in real time, or in some embodiments, the training data pairs may be stored at the vehicle until a threshold amount of training data has been collected. When the threshold amount of training data has been collected, the stored training data pairs may be transmitted to the ad timing training system. After the ML model has been trained on the training data pairs, a copy of the ML model may be transmitted back to the vehicle. The local copy of the ML model at the vehicle may be used to predict when ads may be displayed to the driver.

[0025] An additional advantage of the disclosed system is that the ML model may additionally or alternatively be used to determine when to display other types of data to the driver.

[0026] Referring now to FIG. 1, a simplified vehicle control system 100 of a vehicle is shown, where vehicle control system 100 includes a controller 102 configured to display advertisements to a driver of the vehicle via an advertisement delivery system 110 stored in a memory 106 of controller 102. Vehicle control system 100 includes a plurality of sensors 120 of the vehicle, and a plurality of media output channels 130 of the vehicle. Various outputs of sensors 120 may be used by advertisement delivery' system 110 to display targeted ads to the driver via the media output channels 130. Advertisement delivery system 110 includes a local ad timing model 1 12, a training database 114, and a local ad success model 116, described below.

[0027] Sensors 120 may include a DMS 140, and one or more environmental sensors 122, which may be used to collect environmental sensor data pertaining to a driver of the vehicle. DMS 140 may monitor the driver to detect or measure aspects of a physiological state of the driver (such as drowsiness, distraction, anxiety, etc.), for example, via a dashboard camera ofthe vehicle, or via one or more sensors arranged in the cabin of the vehicle. Biometric data of the driver (e.g., vital signs, galvanic skin response, and so on) may be collected from a sensor of a driver’s seat of the vehicle, or a sensor on a steering wheel of the vehicle, or a different sensor in the cabin. DMS 140 may analyze dashboard camera data, biometric data, and other data of the driver to generate an output.

[0028] Sensors 120 may also include one or more driver feedback controls 142, which may be used to provide feedback to an advertisement delivery system of the vehicle about an ad displayed at a display of the vehicle, as described in greater detail below. The one or more driver feedback controls 142 may include, for example, control elements such as buttons arranged on a steering wheel of the vehicle and / or a dashboard of the vehicle.

[0029] Environmental sensors 122 may output environmental context information pertinent to the physiological state of the driver, such as a number of passengers in the vehicle, a noise level in the cabin of the vehicle, an amount of traffic surrounding the vehicle, and the like. Environmental sensors 122 may include, for example, sensors from an in-vehicle infotainment (I VI) system 124, a navigation system or navigational guidance system 126, one or more bus systems 128. and / or a V2X / Telematics module 129 of the vehicle. V2X / Telematics module 129 may include environmental sensors positioned outside the vehicle (e.g., at a Telematics box in an environment of the vehicle) via a wireless network (e.g., a cellular network). Controller 102 may include at least one processor 104, which may execute instructions stored on a memory 106 to control media output channels 130 based at least partly on output of sensors 120.

[0030] Media output channels 130 may include a dashboard display 132, on which visual content may be displayed to the driver and / or a passenger of the vehicle. Dashboard display 132 may be a part of the IVI system 124, and may include a display screen mounted in or on a dashboard of the vehicle. In some embodiments, dashboard display 132 may include a portion of a windshield of the vehicle, on which visual content may be projected, or a different type of display on which content directed at the driver may be displayed. The visual content may include visual information, such as notifications, messages, information about a system or technology of the vehicle, information about elements of the environment of the vehicle, information about music being played at the vehicle or in-cabin environmental information (e.g., temperature, etc.). The visual content may also include streaming video content (e.g., entertainment or infotainment).

[0031] In some embodiments, the display screen may be a touchscreen, and the visual content may also include interactive controls such as buttons, sliders, check boxes, and / or otherselectable control elements displayed on the touchscreen. Media output channels 130 may also include one or more rear-seat displays 134, on which the same or similar visual content may be displayed to passengers. Media output channels 130 may also include one or more front seat speakers 136 and / or one or more rear seat speakers 138, through which audio content may be played to the driver and / or other occupants of the vehicle. The audio content may be associated with the visual content displayed on dashboard display 132 and / or the one or more rear seat displays 134. or the audio content may be played independently from the visual content.

[0032] Additionally, the visual content may include advertisements directed at the driver and / or a passenger of the vehicle. The advertisements may be targeted to the driver or the passenger, and may be displayed by the advertisement delivery system 110. In particular, a timing with which the advertisements (also referred to herein as ads) are displayed may be controlled based on an output of local ad timing model 112. That is, local ad timing model 112 may predict whether and / or when an ad may be displayed to the driver (or the passenger), based on sensor data of environmental sensors 122 and DMS 140, as described in greater detail herein.

[0033] As an example, in a first scenario. DMS 140 may detect a pattern in video data of the driver received from a dashboard cam that may be associated with drowsiness. An output of DMS 140 may be received as input into local ad timing model 112, and as a result, local ad timing model 112 may output a prediction of an unsuitability of conditions at a current time for displaying an ad. In response to the output, advertisement delivery system 110 may not display an ad on dashboard display 132. DMS 140 may similarly detect a pattern that may be associated with distraction, anxiety, stress, or a different state of the driver, where local ad timing model 112 may output a prediction of the unsuitability' of the conditions at the current time for displaying an ad.

[0034] In a second scenario, based on inputs into local ad riming model 1 12 including data from DMS 140 and environmental sensors 122, local ad timing model 112 may determine that the driver is not distracted, anxious, stressed, etc., and as a result, local ad timing model 112 may output a prediction of a suitability of the conditions of the current time for displaying an ad. In response to the output, advertisement delivery system 110 may display an ad on dashboard display 132.

[0035] The local ad timing model 112 may be an artificial intelligence (Al) model such as a machine learning (ML) or deep learning (DL) model. For example, local ad timing model 112 may comprise a neural network, such as a convolutional neural network (CNN), a generative adversarial network (GAN), or a different kind of neural network model. In otherembodiments, local ad timing model 112 may include one or more different types of models, such as rules-based models, statistical models, hierarchical models, probabilistic models, etc. It should be appreciated that the examples provided herein are for illustrative purposes, and other types of models may be included in local ad timing model 112 without departing from the scope of this disclosure.

[0036] Memory 106 may also include a driver profile 108. Driver profile 108 may include driving style data of a driver of the vehicle. In some embodiments, the driving style data may be an input into advertisement delivery system 110 or local ad timing model 112, as described in greater detail below.

[0037] As discussed herein, memory 106 may include any non-transitory computer readable medium in which instructions are stored. For the purposes of this disclosure, the term "non-transilory computer readable medium” is expressly defined to include any type of computer readable storage, which in various embodiments may include a non-transitory computer readable medium such as a flash memory', a read only memory (ROM), a random access memory (RAM), a cache, or any other storage media (e.g., a tangible medium) in which information is stored for any duration (e.g., for extended period time periods, permanently, brief instances, for temporarily buffering, and / or for caching of the information). Computer memory' of computer readable storage mediums as referenced herein may include volatile and non-volatile or removable and non-removable media for a storage of electronic-formatted information such as computer readable program instructions or modules of computer readable program instructions, data, and the like that may be stand-alone or as part of a computing device. Examples of computer memory may include any other medium which can be used to store the desired electronic format of information and which can be accessed by the processor or processors or at least a portion of a computing device. Various methods and systems disclosed herein may be implemented using instructions (e.g., programming instructions, coded instructions, executable instructions, computer readable instructions, and the like) stored in a non-transitory' computer readable medium.

[0038] Memory 106 may include a training database 114, which may be used to store training data collected by advertising delivery system 110. The training data may include a plurality of training pairs that may be used to periodically re-train local ad timing model 112. More specifically, the plurality7of training pairs may be transmitted to an ad timing training system located in a cloud-based server, as described below in reference to FIGS. 4 and 5. The plurality of training pairs may be used to train a master ad timing model. A copy of the trainedmaster ad timing model may be transmitted back to the vehicle, and the trained master ad timing model may become / replace local ad timing model 112.

[0039] Each training pair of the plurality of training pairs may include ground truth data generated by local ad success model 116, where local ad success model 116 may predict a success score of an advertisement displayed to the driver. The success score may be an encoding of explicit feedback regarding the advertisement provided by the driver, for example, via the driver feedback controls 142. The success score may also be estimated or adjusted based on a degree of attention paid by the driver to the advertisement, or a probability of the degree of attention paid by the driver exceeding a threshold amount of attention. For example, the degree of attention may be estimated based on internal sensors of the vehicle, such as sensors of IVI system 124 and / or vehicle bus systems 128 (e.g., eye-tracking data, etc.). Local ad success model 116 may be a copy of a master ad success model stored on a cloud-based server, that is periodically updated and transmitted to controller 102.

[0040] Referring now to FIG. 2, a data schematic 200 shows examples of data that may be received as input into advertisement delivery system 110 of FIG. 1. by one or more environmental sensors of the vehicle control system and a by DMS of the vehicle control system 100 (e.g., environmental sensors 122 and DMS 140 of FIG. 1). Data schematic 200 includes an exemplary7set of environmental sensor data 202 and an exemplary7set of DMS data 204.

[0041] Environmental sensor data 202 may include IVI system data 210, navigation system data 224, vehicle bus data 250, and V2X / Telematics data 229. In some embodiments, IVI system data 210 may include, for example, a detection of an incoming phone call 212, and / or a detection of an outgoing phone call 214. For some embodiments, IVI system data 210 may include detection of a radio station selection 216 (e.g., a detection of a user of the vehicle selecting a radio station), and / or a media source selection 218 (e.g., a detection of a user of the vehicle selecting a source of infotainment media, such as radio, phone, an online audio service, a collection of stored content, and so on). In some embodiments, IVI system data 210 may include a media track selection 220 of the online service and / or collection of stored content, such as whether the driver or a user of the vehicle is searching for a specific track or browsing forward or backward through the online service and / or collection of stored content. For some embodiments, IVI system data 210 may include one or more infotainment settings 222 of the IVI system, such as for example, a language, font size, brightness, or a different setting.

[0042] In some embodiments. IVI system data 210 may include additional or alternative data not shown in FIG. 2. For example, IVI system data 210 may include data from one ormore devices coupled to the vehicle either via a wired or wireless connection and / or paired with the vehicle, such as a tablet and / or smart phone of an occupant of the vehicle paired to the vehicle via a Bluetooth® connection. (Bluetooth® is a registered trademark of Bluetooth SIG, Inc., Kirkland, WA.)

[0043] Navigation system data 224 of environmental sensor data 202 may include information from a global positioning system (GPS)-based navigation system that may aid the driver in navigating the vehicle through a surrounding environment of the vehicle. Navigation system data 224 may include, for example, a global position of the vehicle, a relative position of the vehicle with respect to one or more traffic features and / or physical features of the surrounding environment, a type or characterization of a kind of driving undertaken by the driver (e.g., city driving, high-traffic scenario, and so on), route information (e.g.. a destination of the driver), and / or other types of information produced by and / or made available by navigation systems.

[0044] In some embodiments, navigation system data 224 may include, for example, an active route start 226 (e.g., a detection that the driver has initiated a planned route), an active route end 228 (e.g., a detection that the driver has finished the planned route), an active route change 230 and / or a routing options change 232 (e g., a detection that the driver has changed a routing option or selected an alternative route), and / or an added route point 234 (e.g., a detection that the driver has included an intermediate or additional destination on the planned route). For some embodiments, navigation system data 224 may include an interactive mode selection 236 of the navigation system, if the driver sw itches a map viewer of the navigation system to an interactive map mode, and / or zoom data 238, if the driver is zooming in or zooming out of a map displayed on the map viewer.

[0045] Vehicle bus data 250 of environmental sensor data 202 may be received from one or more bus systems of the vehicle, such as a controller area network (CAN) bus of the vehicle, a local interconnect network (LIN) bus of the vehicle, and / or an automotive Ethernet netw ork of the vehicle. In some embodiments, vehicle bus data 250 may include exterior camera data 252 (e.g., images) outputted by one or more exterior cameras. For example, the vehicle may include one or more front-end exterior cameras arranged at a front end of the vehicle, one or more rear-and exterior camera arranged at a rear end of the vehicle, and / or one or more exterior cameras arranged at a side, undercarriage, or roof of the vehicle. The one or more exterior cameras outputting exterior camera data 252 may be used to aid the driver in backing up and / or adjusting a position of vehicle, for example, when parking the vehicle. The one or more exterior cameras outputting exterior camera data 252 may also be used to detect objects in a proximityto the vehicle. For example, the one or more exterior cameras outputting exterior camera data 252 may be used to detect one or more other vehicles operating next to or near the vehicle, a curb of a road the vehicle is operating on, one or more pedestrians positioned or moving near the vehicle, or other objects that may be encountered on or near the road. Exterior camera data 252 may also be used to detect or determine one or more conditions of a surrounding environment of the vehicle, such as a weather or climate condition, such as a presence of rain, snow, or wind (e.g.. trees blowing, etc.); a condition of the road, such as a type of the road (e.g., paved, unpaved, etc ), a presence of potholes or obstacles, and / or a presence of snow, ice, dust or dirt; and so on.

[0046] In some embodiments, vehicle bus data 250 may also include proximity sensor data 254, radar data 256, and / or lidar data 258, which may also be used to detect objects in proximity to the vehicle. For example, radar data 256, lidar data 258, and proximity sensor data 254 for a front-end proximity sensor may be used to determine a distance between the vehicle and a leading vehicle, or the radar data 256, lidar data 258, and proximity sensor data 254 for a rear- end proximity sensor may be used to determine a distance between the vehicle and a following vehicle. Because a presence of one or more objects (e.g., vehicles) in proximity to the vehicle may increase a cognitive load of the driver and / or a level of stress of the driver, in various embodiments, proximity sensor data 254, radar data 256, and / or lidar data 258 may be used by the driver model to predict an emotional or physiological state of the driver.

[0047] In some embodiments, vehicle bus data 250 may include in-cabin sensor data 260. In-cabin sensor data 260 may include sensors arranged around an interior of a cabin of the vehicle, which may be used to detect contextual information of the cabin that could be pertinent to a physiological state of the driver. For example, in-cabin sensor data 260 may include one or more passenger seat sensors, which may indicate whether one or more respective passenger seats of the vehicle are occupied or unoccupied. In some embodiments, the one or more passenger seat sensors may output additional information of occupants of the one or more respective passenger seats, such as, for example, a weight of the occupants, temperatures of the occupants, or vital signs of the occupants, or a level of restlessness of the occupants, or a different characteristic of the occupants.

[0048] In-cabin sensor data 260 may also include one or more in-cabin microphones, which may be arranged at any location within a cabin of the vehicle. For example, a microphone may be positioned on a ceiling of the cabin, or on a seat of the cabin, such as on the back of a front seat of the vehicle or on a headrest of one of the seats of the cabin, or in a console between two seats of the cabin, or in a door or side wall of the cabin, or at a different location of the cabin.The one or more in-cabin microphones may be used to determine a noise level in the cabin, or to differentiate between different types of noise present in the cabin. For example, the one or more in-cabin microphones may be used to detect the presence of a crying infant, or arguing children, cross conversation between occupants in different seats of the vehicle, and / or audio signals of devices of occupants or infotainment systems of the vehicle. As described in greater detail below, in some examples, advertisement delivery system 110 may analyze the cross conversation to determine whether an advertisement displayed to the driver or a passenger may have been viewed. For example, if the driver or passenger views the ad, the cross conversation may pause momentarily.

[0049] In-cabin sensor data 260 may include cabin temperature data (e.g., an output of an in-cabin temperature sensor), or interior lighting data. The cabin temperature data may include one or more settings of an air conditioning (AC) / heating system of the vehicle, such as a target AC temperature status and / or any changes made by the driver, a strength status of an AC system and / or any changes made by the driver, and an AC mode (e g., direct or diffuse).

[0050] In some embodiments, vehicle bus data 250 may include sensor data of one or more driving controls of the vehicle, such as vehicle speed data 262. a steering wheel angle 264. a brake pedal position 266, and / or an accelerator pedal position 267. For some embodiments, vehicle bus data 250 may include a light status 268, a windshield wiper status 270, and / or a roof status 274 (e.g.. whether a roof of the vehicle is open or closed) which may be used to determine a level of visibility in an exterior environment and whether rain or snow may be falling. For some embodiments, vehicle bus data 250 may include a turn indicator status 272, which may indicate whether the drivers intending to turn the vehicle to the left or to the right, or to change lanes on a road. In some embodiments, vehicle bus data 250 may include an engine stop / start status 276 (e.g., a detection of whether an engine of the vehicle is on or off, for example, during a stop-start routine to save fuel). For some embodiments, vehicle bus data 250 may include one or more instrument cluster (IC) warnings 278 visible on a dashboard of the vehicle. For example, a check engine light may be detected, or a low fuel level indicator, a parking brake indicator, or a different IC warning.

[0051] In some embodiments. V2X / Telematics data 229 may include sensor data transmitted from a V2X / Telematics module (e.g., V2X / Telematics module 129) located outside the vehicle. For example, V2X / Telematics data 229 may include data collected in an environment of the vehicle, including weather or meteorological data, traffic data, road condition data, and the like.

[0052] Turning to DMS data 204, in some embodiments, DMS data 204 may include a driver identification 280. For example, the DMS may identify the driver based on a presence of a key fob of the driver at a driver’s seat of the vehicle. In some embodiments, the identification of the driver may be used to retrieve a driving profile of the driver from a memory of the vehicle (e.g., driver profile 108 of controller 102 of FIG. 1) or on a cloud-based server (as disclosed further herein). For example, the driving profile of the driver may include driving style data of the driver (e.g.. a braking style, an acceleration style, a steering style, and / or one or more preferential cruising speeds of the driver), and / or other data of the driver relevant to operation of the vehicle, such as a detection of one or more typical and / or historical behavioral patterns of the driver when operating the vehicle.

[0053] In some embodiments. DMS data 204 may include a dashboard camera data 282 and / or vehicle occupancy data 284 of the vehicle. Dashboard camera data 282 may include images of a face and / or head of the driver. In some embodiments, dashboard camera data 282 may include data of one or more passengers of the vehicle. For example, dashboard camera data 282 may detect images of faces of occupants at one or more seats of the vehicle, which may be used to augment and / or generate vehicle occupancy data 284. Vehicle occupancy data 284 may also include data from one or more seat sensors, and / or seatbelt sensors.

[0054] For some embodiments, DMS data 204 may include interior illumination data 286 of the cabin of the vehicle and / or biometric data 288 of the driver. For example, one or more steering wheel sensors may output vital sign data of the driver, such as a heart rate of the driver. The one or more steering wheel sensors may include galvanic skin response data of the driver, or a different type of biometric sensor data.

[0055] Additionally, in some embodiments, DMS data 204 may include analysis and / or result data for a DMS. In other words, in addition to raw data collected by sensors of a DMS (e.g., dashboard cameras outputting dashboard camera data 282, biometric data 288), DMS data 204 may include intermediate and / or final determinations made by the DMS as a result of processing and analyzing the raw data. For example, the DMS may detect patterns in the raw data. The patterns of the raw data may include driver movement pattern data 290, which may be a pattern detected in how the driver moves his head and / or a different part of his body repeatedly or periodically in a specific manner. The patterns of the raw data may include driver eye gaze data 292, which may include what the driver is looking at and / or patterns where the driver shifts an eye gaze repeatedly or periodically at a specific manner.

[0056] The patterns of the raw data may be analyzed and codified by the DMS to aid in predicting a physiological state of the driver. For example, for some embodiments, the DMSmay generate a driver stress assessment 294, which may be an estimated level of stress of the driver based on the patterns of the raw data. In some embodiments, the DMS may generate a driver drowsiness assessment 296, which may be an estimated level of drowsiness of the driver based on the patterns of the raw data. For some embodiments, DMS may generate a driver distraction assessment 298, which may be an estimated level of distraction of the driver based on the patterns of the raw data. Driver stress assessment 294, driver drowsiness assessment 296, and / or driver distraction assessment 298 may be used by the driver model, in addition to environmental sensor data 202, to predict the physiological state of the driver.

[0057] Referring now to FIG. 3, a data flow diagram 300 shows how data of the driver, the vehicle, and / or passengers of the vehicle such as the environmental sensor data 202 and DMS data 204 of FIG. 2 may be received as inputs into the advertisement delivery system 110 of the vehicle, for generating an advertisement 350 to be displayed to the driver. The inputs may be used by local ad timing model 112 of vehicle control system 100 of FIG. 1, to determine a timing of displaying ads to the driver of the vehicle. Determining the timing of displaying ads to the driver may include determining whether to display an ad, when to display an ad, how long an ad may be displayed, and / or when a displayed ad should be stopped. Some of the inputs may also be used by local ad success model 116 to determine whether an ad was displayed to the driver at an appropriate time. The inputs may be used to generate training data that may be stored in training database 114, to be used to train local ad timing model 112, as described below in reference to FIG. 4.

[0058] The inputs into advertisement delivery system 110 may include one or more environmental inputs 302. Environmental inputs 302 may include the environmental sensor data 202 of FIG. 2, such as IVI system data regarding information currently displayed on an IVI (e.g.. dashboard) display, navigation system data (e.g., navigation system 126 of FIG. 1), and vehicle bus data (e.g., traffic and weather data, steering and braking data, engine data, etc.). The navigation system data may include route information including a location of the vehicle, whether or not the driver has selected or is navigating along an active route of the navigation system, one or more destinations of the driver, a type of driving environment (e.g., urban environment, rural environment), a type of road the driver is navigating on (e.g., a multilane road, single lane road, highway, unpaved road, and so on), or a different type of information outputted by the navigation system. Environmental inputs 302 may be reformatted to create a set of reformatted environmental inputs 303, which may have a reduced size and / or dimensionality than environmental inputs 302. Reformatted environmental inputs 303 may include scores generated for different categories of environmental inputs 302, such as a trafficscore, a weather score, a road condition score, etc. That is, reformatting environmental inputs 302 may include various steps in which environmental inputs 302 may be categorized, analyzed, ranked and / or weighted within the different categories, culminating in a set of numeric (or binary) scores into which the environmental inputs 302 may be converted to generate the reformatted environmental inputs 303.

[0059] In some embodiments, the inputs into advertisement delivery system 110 may include driving style inputs 304. For example, driving style inputs 304 may include a braking style of the driver, an acceleration style of the driver, and a steering style the driver. Driving style inputs 304 may be retrieved from a driver profile of a driver stored in a memory' of the vehicle.

[0060] For example, a first driver may have a first driving style, including a first braking style, a first acceleration style, and a first steering style, and a second driver may have a second driving style different from the first driving style, including a second braking style, a second acceleration sty le, and a second steering style, which are different from the first braking sty le, the first acceleration style, and the first steering style, respectively. The first braking style of the first driver may be an inpatient braking style, characterized by an abrupt manipulation of a brake pedal, and the second braking style of the second driver may be a cautious braking style, characterized by a less abrupt manipulation of the brake pedal. The first acceleration style of the first driver may be an impatient acceleration style, characterized by rapid positive and negative accelerations, and the second acceleration style of the second driver may be a cautious acceleration style, characterized by slow and steady positive and negative accelerations. The first steering style of the first driver may be an impatient steering style, characterized by fast rotational movements of a steering wheel, and the second steering style of the second driver may be a cautious steering style, characterized by slow rotational movements of the steering wheel. Driving style inputs may also include one or more preferred cruising speeds of the driver. For example, a first driver may prefer to drive at a first cruising speed when operating the vehicle on a highway, the first cruising speed at a speed limit of the highway, and a second driver may prefer to drive at a second cruising speed when operating the vehicle on the highway, the second cruising speed above the speed limit of the highway.

[0061] In various embodiments, driving style inputs 304 may be based on integrated, aggregated or average driving sty le input data collected from drivers over a period of time. Driving sty le inputs 304 may be retrieved from a driver model generated and updated by software running at a cloud-based server, as described below in reference to FIG. 4. Driving style inputs 304 may be reformatted into a set of reformatted driving style inputs 305, to reducea size and / or dimensionality of driving style inputs 304. For example, instead of inputting a plurality of driving style inputs 304 into local ad timing model 112, driving style inputs 304 may be processed to generate a driving style score of the driver, and the driving style score may be inputted into the local ad timing model.

[0062] The inputs into advertisement delivery' system 110 may include driver status inputs 306. Driver status inputs 306 may include DMS data (e.g., DMS data 204 of FIG. 2) of a DMS (e.g.. DMS 140 of FIG. 1) of the vehicle. The DMS data may include a pattern of head and / or eye movements in images captured by a dashboard camera of the vehicle, audio signals recorded by a microphone arranged in a cabin of the vehicle, biometric data collected via sensors arranged in the cabin, steering wheel sensors, seat sensors, and the like. In some embodiments, driver status inputs 306 includes inputs explicitly provided by the driver. For example, as described in greater detail below, the driver may indicate a willingness to view an advertisement, for example, to earn an incentive.

[0063] Driver status inputs 306 may be reformatted into a set of reformatted driver status inputs 308, which may be estimated or predicted cognitive states of the driver. In some embodiments, the estimated or predicted cognitive states of the driver may be generated using a rules-based model, or a statistical model, or a machine learning model, or a combination of a rules-based model, a statistical model, and / or a machine learning model. In various embodiments, the model may be provided by the DMS. Reformatted driver status inputs 308 may include, for example, one or more of an estimated level of stress of the driver, an estimated level of drowsiness of the driver, an estimated level of distraction of the driver, and / or an estimated cognitive load of the driver, or an estimation of a different cognitive state of the driver. In other words, rather than inputting raw eye gaze data into local ad timing model 112, a driver stress score, a driver drowsiness score, and / or a driver distraction score generated from the raw eye gaze data may be inputted into local ad timing model 1 12.

[0064] The inputs into advertisement delivery system 110 may include one or more ad timing feedback inputs 310. The one or more ad timing feedback inputs 310 may be generated explicitly by the driver, in response to viewing an ad displayed by advertising delivery system 110. For example, advertisement delivery system 110 may display an ad to the driver, and the driver may provide feedback (e.g., via a steering wheel control, or a button on a dashboard of the vehicle, by voice, etc.) regarding whether the ad was displayed at an appropriate time, based on driver status inputs 306 and environmental inputs 302. The feedback may be used as ground truth data for training the local ad timing model 112.

[0065] In various embodiments, the training data generated at advertising delivery system 110 may be transmitted from the vehicle to an ad delivery training system located at a cloudbased server, as described below in reference to FIG. 4.

[0066] Turning to FIG. 4, a wireless communication diagram 400 is shown, including a vehicle 401 in communication with a driver profile database 412 and an ad timing training system 430 via a cloud 406. In various embodiments, vehicle 401 may access cloud 406 and driver profile database 412 via a wireless network, such as a wireless cellular network 420. using a modem 416 of the vehicle. As described above, vehicle sensor data 402 and driver status data of driver 404 may be transmitted to ad timing training system 430 to train a master ad timing model 444.

[0067] Driver profile database 412 may include a plurality of driver profiles of a respectively corresponding plurality of drivers. Each driver profile of the plurality of driver profiles may include data of a corresponding driver, such as identifying information of the driver, demographic information of the driver, current and previous vehicle usage data of the driver, preferences of the driver with respect to settings of one or more vehicles of the driver, and the like. During operation of vehicle 401. vehicle 401 may receive driver profile data corresponding to the driver, and a controller of vehicle 401 may adjust settings of vehicle 401 based on the driver profile data. For example, based on the driver profile, the controller may adjust a position of a driving seat of vehicle 401, or a preferred radio station of the driver, or a preferred interior lighting of vehicle 401, or a different setting of vehicle 401. Each driver profile may additionally include a driving style model, which may be generated based on sensor data transmitted by vehicle 401. In some embodiments, a master driver profile may be stored in driver profile database 412, and a local copy of the master driver profile may be stored in a memory of vehicle 401 (e.g., driver profile 108). The local copy may be used, for example, in the absence of conductivity with wireless cellular network 420. The local copy may be updated periodically via cloud 406. Driver profile data may be inputted into the ad timing training system, and may be included as input into a master ad timing model 444 during training of the master ad timing model 444.

[0068] Ad timing training system 430 includes one or more processors 434 configured to execute machine readable instructions stored in a non-transitory memory 436. Processor(s) 434 may be any suitable processor, processing unit, or microprocessor, for example. The processor(s) may be a multi-processor system, and, thus, may include one or more additional processors that are identical or similar to each other and that are communicatively coupled via an interconnection bus. Processor 434 may be single core or multi-core, and the programsexecuted thereon may be configured for parallel or distributed processing. In some embodiments, processor 434 may optionally include individual components that are distributed throughout two or more devices, which may be remotely located and / or configured for coordinated processing. In some embodiments, one or more aspects of processor 434 may be virtualized and executed by remotely-accessible networked computing devices configured in a cloud computing configuration.

[0069] Non-transitory memory 436 may include one or more data storage structures, such as optical memory devices, magnetic memory devices, or solid-state memory devices, for storing programs and routines executed by the processor(s) of the local server to carry out various functionalities disclosed herein. The memory may include any desired type of volatile and / or non-volatile memory such as, for example, static random access memory (SRAM), dynamic random access memory (DRAM), flash memory, read-only memory (ROM), etc.

[0070] Non-transitory memory' 436 may include an Al module 442, a training module 446, an inference module 448, and a communication module 450. In some embodiments, the non- transitory memory 436 may include components disposed at two or more devices, which may be remotely located and / or configured for coordinated processing. In some embodiments, one or more aspects of the non-transitory memory 436 may include remotely-accessible networked storage devices configured in a cloud computing configuration.

[0071] Al module 442 may include one or more trained and / or untrained models, such as neural network models, and may further include various data, or metadata pertaining to the one or more neural network models stored therein. Al module 442 may include a plurality local neural network model parameters, batch size, weighting coefficients, global learning rate, and the like. In particular, Al module 442 may include the master ad timing model 444. Master ad timing model 444 may be trained using the ad timing training system 430 to predict a timing of displaying ads to a driver of vehicle 401, as described in greater detail in relation to FIGS. 5, 8, and 9. In some embodiments, the ads may be targeted to the driver based on driver data (DMS data 204 and driver profile data from driver profile 108) and / or environmental data(e.g., environmental sensor data 202) collected from a surrounding environment of the vehicle.

[0072] Al module 442 may include an ad success model 445, which may determine whether an ad displayed to the driver was displayed at an appropriate time. In one embodiment, determining whether an ad displayed to the driver was displayed at an appropriate time may include determining whether the driver looked at the ad, as detected via eye-tracking software in conjunction with dashboard camera data such as dashboard camera data 282. The eyetracking software may be installed in a DMS such as DMS 140. In other embodiments, adsuccess model 445 may be used in conjunction with driver feedback (e.g., the ad timing feedback inputs 310 of FIG. 3) regarding an ad to determine whether the ad was displayed at an appropriate time.

[0073] Training module 446 may comprise instructions for training one or more of the neural networks stored in Al module 442. In some embodiments, training module 446 includes instructions for implementing one or more gradient descent algorithms, applying one or more loss functions, and / or training routines, for use in adjusting parameters of the one or more neural networks. In particular, training module 446 may include instructions that, when executed by the processor 434, cause ad timing training system 430 to train master ad timing model 444, in accordance with method 900, discussed in more detail below in reference to FIG. 9.

[0074] Non-transitory memory 106 may also store an inference module 448 that comprises instructions for predicting a suitable time to display ads to a driver of a vehicle with a trained master ad timing model 444. That is. a copy of the trained master ad timing model 444 may be transmitted to vehicle 401 as a local ad timing model (e.g., local ad timing model 112), and the local ad timing model may be used to predict a suitable time to display ads to a driver 404 of vehicle 401. In some embodiments, the training module 446 and the inference module 448 may be stored and / or executed on separate devices.

[0075] The ad timing training system 430 may include a communication module 450. The communication module 450 may facilitate transmission of the trained master ad timing model 444 to vehicles such as vehicle 401. Communication via the communication module 450 may be implemented using one or more protocols. The communication module can be a wired interface (e.g., a data bus, a Universal Serial Bus (USB) connection, etc.) and / or a wireless interface (e.g., radio frequency, infrared, near field communication (NFC), etc.). For example, the communication module may communicate via wired local area network (LAN), wireless LAN, wide area network (WAN), etc. using any past, present, or future communication protocol (e.g., BLUETOOTH™, USB 2.0, USB 3.0, etc.).

[0076] Ad timing training system 430 may be operably / communicatively coupled to a user input device 422 and display device 424. User input device 422 may comprise one or more of a touchscreen, a keyboard, a mouse, a trackpad, a motion sensing camera, or other device configured to enable a user to interact with and manipulate data within ad timing training system 430. Display device 424 may include one or more display devices utilizing virtually any type of technology. In some embodiments, display device 424 may comprise a computer monitor. Display device 424 may be combined with processor 434, non-transitory memory436, and / or user input device 422 in a shared enclosure, or may be peripheral display devices and may comprise a monitor, touchscreen, projector, or other display device known in the art, which may enable a user to interact with various data stored in non-transitory memory 436, such as to adjust batch size and a weighting coefficient of a set of accumulated gradients.

[0077] It may be understood that ad timing training system 430 shown in FIG. 4 is for illustration, not for limitation. Another appropriate ad timing training system may include more, fewer, or different components.

[0078] Referring to FIG. 5, an exemplary data flow diagram 500 is shown depicting a flow of data during training of a master ad timing model 502, using an ad timing training system such as ad timing training system 430 of FIG. 4. The ad timing training system may be used to train master ad timing model 502 to predict a suitability’ of conditions for displaying an advertisement to a driver while the driver is operating a vehicle, such as vehicle 401. In various embodiments, master ad timing model 502 may be trained by following one or more steps of method 800 of FIG. 8 and / or method 900 of FIG. 9, described below.

[0079] Data flow diagram 500 shows a flow of data between a vehicle 594 (e.g., vehicle 401) and a cloud-based server 592 (e.g., located in cloud 406 of FIG. 4). Training master ad training model 502 may include collecting training data at vehicle 594 in a first stage, and training master ad training model 502 at server 592 in a second stage. Further, the training data collected at vehicle 594 may be processed to generate a set of training pairs 522, which may be transmitted to server 592. The training pairs 522 may be received at server 592, and may be inputted into master ad timing model 502 during a training procedure. In some embodiments, the training pairs 522 may be transmitted to server 592 at a time of creation. In other embodiments, the training pairs 522 may be stored in a training database 528 (e.g., training database 114 of FIG. 1) when created, and transmitted to server 592 at a later time. In other words, the training pairs 522 may be sent periodically to server 592 for retraining master ad timing model 502. For example, the training pairs 522 may be sent at regular intervals, or when requested by server 592, or in response to a threshold number of training pairs being collected, or in accordance with a different strategy.

[0080] The training pairs 522 may include input data 524, and ground truth data 526. The input data 524 may include environmental sensor data 514 and driver state data 516 collected at a time that an advertisement 510 is displayed to the driver. The environmental sensor data 514 and driver state data 516 may be the same as or similar to the environmental sensor data 202 and DMS data 204 of FIG. 2, respectively. Advertisement 510 may be displayed to the driver by an ad delivery system 504 of the vehicle (e.g., advertisement delivery system 110).A timing of the display of advertisement 510 may be determined by a local ad timing model 506 of ad delivery system 504. After advertisement 510 is displayed, an ad success score 518 may be estimated by a local ad success model 508 (e.g., local ad success model 116 of FIGS. 1 and 3) of ad delivery system 504. The ad success score may be totally or partially determined by driver state data 516 upon delivery of the ad, and / or totally or partially determined by driver feedback 517 collected from the driver, which may include explicit feedback regarding an appropriateness of the timing of the display of advertisement 510 (e.g., ad timing feedback inputs 310 of FIG. 3). Environmental sensor data 514, driver state data 516, and ad success score 518 collected during a data collection stage 512 may be received by a dataset generator 520, which may reformat the environmental sensor data 514 and driver state data 516 and generate the training pairs 522 using reformatted environmental sensor data 514 and driver state data 516 as input data and ad success score 518 as ground truth data.

[0081] Once the training pairs 522 have been generated, the training pairs 522 may be assigned by dataset generator 520 to either a training dataset, validation dataset, or a test dataset. The training dataset may be used for the optimization of master ad timing model 502. The validation dataset may be used to prevent overfitting, and avoid master ad timing model 502 learning to map features specific to samples of the training set that are not present in the validation set. The test dataset may be used to estimate the model’s performance in deployment. The number of training pairs 522 in the test and validation datasets may be less than the number of training pairs 522 in the training dataset.

[0082] In some embodiments, the training pairs 522 may be randomly assigned to either the training dataset, the validation dataset, or the test dataset in a pre-established proportion. For example, 80% of the training pairs 522 generated may be assigned to the training dataset, and 10% of the training pairs 522 generated may be assigned to the validation and test dataset each. In other embodiments, different proportions of training pairs 522 may be assigned to the training dataset, validation dataset, and the test dataset. It should be appreciated that the examples provided herein are for illustrative purposes, and the training pairs 522 may be assigned to the training dataset, validation dataset, or the test dataset via a different procedure and / or in a different proportion without departing from the scope of this disclosure.

[0083] During training, master ad timing model 502 may be configured to receive the training pairs 522, and one or more parameters of master ad timing model 502 may be iteratively adjusted in order to minimize a loss function based on a difference between a prediction of an appropriateness of displaying an ad outputted by master ad timing model 502 based on the input data 524, and the ground truth 526, until an error rate decreases below a firstthreshold error rate. Training of master ad timing model 502 is described in greater detail below, in reference to FIG. 8.

[0084] The ad timing training system may include a validator 540 that validates a performance of master ad timing model 502. Validator 540 may take as input a trained or partially trained master ad timing model 502 and a validation dataset of training pairs 522. If the error rate of the trained or partially trained master ad timing model 502 on the validation dataset of training pairs 522 decreases below a second threshold error rate, the performance of the trained or partially trained master ad timing model 502 may be validated, whereby a training stage of the trained or partially trained master ad timing model 502 may end.

[0085] After training ends, a validated master ad timing model 550 may be used to generate an ad success prediction 552 based on a new set of input data, at a plurality of vehicles. For such purpose, a copy of validated master ad timing model 550 may be transmitted back to vehicle 594 (and other participating vehicles) to replace local ad timing model 506. In this way, local ad timing model 506 may be periodically updated.

[0086] FIG. 6 shows an interior of a cabin 600 of a vehicle 602, in which a driver and / or one or more passengers may be seated. Vehicle 602 of FIG. 6 may be a motor vehicle including drive wheels (not shown) and an internal combustion engine 604. Internal combustion engine 604 may include one or more combustion chambers which may receive intake air via an intake passage and exhaust combustion gases via an exhaust passage. Vehicle 602 may be a road automobile, among other types of vehicles. In some examples, vehicle 602 may include a hybrid propulsion system including an energy conversion device operable to absorb energy from vehicle motion and / or the engine and convert the absorbed energy to an energy form suitable for storage by an energy storage device. Vehicle 602 may include a fully electric vehicle, incorporating fuel cells, solar energy capturing elements, and / or other energy storage systems for powering the vehicle.

[0087] Vehicle 602 may include a plurality of vehicle systems, including a braking system for providing braking, an engine system for providing motive power to wheels of the vehicle, a steering system for adjusting a direction of the vehicle, a transmission system for controlling a gear selection for the engine, an exhaust system for processing exhaust gases, and the like. Further, the vehicle 602 includes an in-vehicle computing system 609. The in-vehicle computing system 609 may include an autonomous vehicle control system for at least partially controlling vehicle systems during autonomous driving. As an example, while operating in an autonomous mode, the autonomous vehicle control system may monitor vehicle surroundings via a plurality of sensors (e.g., such as cameras, radars, ultrasonic sensors, a GPS signal, andthe like). The in-vehicle computing system 609 is described in greater detail below in reference to FIG. 7.

[0088] As shown, an instrument panel 606 may include various displays and controls accessible to a human user (e.g., a driver or a passenger) of vehicle 602. For example, instrument panel 606 may include a touch screen 608 of an in-vehicle computing system or infotainment system 609 (e.g., an infotainment system), an audio system control panel, and an instrument cluster 610. Touch screen 608 may receive user input to the in-vehicle computing system or infotainment system 609 for controlling audio output, visual display output, user preferences, control parameter selection, and so on. In some examples, instrument panel 606 may include one or more controls for driver assistance programs, such as a cruise control system, a collision avoidance system, and the like. Further, additional user interfaces, not shown, may be present in other portions of the vehicle, such as proximate to at least one passenger seat. For example, the vehicle may include a row of back seats with at least one touch screen controlling the in-vehicle computing system 609.

[0089] Vehicle 602 may also include various steering wheel controls 640 accessible to a human user via a steering wheel 641. The steering wheel controls 640 may duplicate functionalities offered by the various displays and controls of instrument panel 606, and as such, may offer the driver more convenient interaction with vehicle 602 (e.g., without having to take their hands off of steering wheel 641). The steering wheel controls 640 may also include functionalities not available at instrument panel 606 (e.g., cruise control, etc.). In particular, the steering wheel controls 640 may be advantageously used by the driver to provide feedback to an ad delivery’ system of vehicle 602 that may display ads to the driver, for example, via a display screen 611. For example, an ad may be displayed on display screen 611, and the driver may select a first control to indicate that a timing of the display of the ad was appropriate (e.g., given driver, driving, and environmental conditions), and the driver may select a second control to indicate that a timing of the display of the ad was not appropriate. Alternatively, the driver may adjust a control in a first direction or a second direction to rate the timing of the display of the ad. or a different control scheme may be used.

[0090] While the example system shown in FIG. 6 includes audio system controls that may be performed via a user interface of in-vehicle computing system or infotainment system 609, such as touch screen 608 without a separate audio system control panel, in other embodiments, the vehicle may include an audio system control panel, which may include controls for a conventional vehicle audio system such as a radio, compact disc player. MP3 player, and so on. The audio system controls may include features for controlling one or more aspects of audiooutput via one or more speakers 612 of a vehicle speaker system. For example, the in-vehicle computing system or the audio system controls may control a volume of audio output, a distribution of sound among the individual speakers of the vehicle speaker system, an equalization of audio signals, and / or any other aspect of the audio output. In further examples, in-vehicle computing system or infotainment system 609 may adjust a radio station selection, a playlist selection, a source of audio input (e.g., from radio or CD or MP3), and so on, based on user input received directly via touch screen 608, or based on data regarding the user (such as a physical state and / or environment of the user) received via one or more external devices 650 and / or a mobile device 628. The audio system of the vehicle may include an amplifier (not shown) coupled to plurality of loudspeakers (not shown). In some embodiments, one or more hardware elements of in-vehicle computing system or infotainment system 609, such as touch screen 608, display screen 611, various control dials, knobs and buttons, memory, processor(s), and any interface elements (e.g., connectors or ports) may form an integrated head unit that is installed in instrument panel 606 of the vehicle. The head unit may be fixedly or removably attached in instrument panel 606. In additional or alternative embodiments, one or more hardware elements of the in-vehicle computing system or infotainment system 609 may be modular and may be installed in multiple locations of the vehicle.

[0091] The cabin 600 may include one or more sensors for monitoring the vehicle, the user, and / or the environment. For example, the cabin 600 may include one or more seat-mounted pressure sensors configured to measure the pressure applied to the seat to determine the presence of a user, door sensors configured to monitor door activity, humidity sensors to measure the humidity content of the cabin, microphones to receive user input in the form of voice commands, to enable a user to conduct telephone calls, and / or to measure ambient noise in the cabin 600, and so on. It is to be understood that the above-described sensors and / or one or more additional or alternative sensors may be positioned in any suitable location of the vehicle. For example, sensors may be positioned in an engine compartment, on an external surface of the vehicle, and / or in other suitable locations for providing information regarding the operation of the vehicle, ambient conditions of the vehicle, a user of the vehicle, and so on. Information regarding ambient conditions of the vehicle, vehicle status, or vehicle driver may also be received from sensors external to / separate from the vehicle (that is, not part of the vehicle system), such as sensors coupled to external devices 650 and / or mobile device 628. Sensor data of various sensors of the vehicle may be transmitted to and / or accessed by the in- vehicle computing system 609 via a bus of the vehicle, such as a controller area network (CAN) bus.

[0092] Cabin 600 may also include one or more user objects, such as mobile device 628, that are stored in the vehicle before, during, and / or after travelling. The mobile device 628 may include a smart phone, a tablet, a laptop computer, a portable media player, and / or any suitable mobile computing device. The mobile device 628 may be connected to the in-vehicle computing system via a communication link 630. The communication link 630 may be wired (e.g., via Universal Serial Bus (USB). Mobile High-Defmition Link (MHL). High-Definition Multimedia Interface (HDMI). Ethernet, and so on) or wireless (e.g., via Bluetooth®. Wi-Fi®. Wi-Fi Direct®, Near-Field Communication (NFC), cellular connectivity, and so on) and configured to provide two-way communication between the mobile device and the in-vehicle computing system. (Bluetooth® is a registered trademark of Bluetooth SIG, Inc., Kirkland, WA. Wi-Fi® and Wi-Fi Direct® are registered trademarks of Wi-Fi Alliance, Austin, Texas.) The mobile device 628 may include one or more wireless communication interfaces for connecting to one or more communication links (e.g., one or more of the example communication links described above). The wireless communication interface may include one or more physical devices, such as antenna(s) or port(s) coupled to data lines for carrying transmitted or received data, as well as one or more modules / drivers for operating the physical devices in accordance with other devices in the mobile device. For example, the communication link 630 may provide sensor and / or control signals from various vehicle systems (such as vehicle audio system, climate control system, and so on) and the touch screen 608 to the mobile device 628 and may provide control and / or display signals from the mobile device 628 to the in-vehicle systems and the touch screen 608. The communication link 630 may also provide power to the mobile device 628 from an in-vehicle power source in order to charge an internal batten' of the mobile device.

[0093] In-vehicle computing system or infotainment system 609 may also be communicatively coupled to additional devices operated and / or accessed by the user but located external to vehicle 602, such as one or more external devices 650. In the depicted embodiment, external devices are located outside of vehicle 602 though it will be appreciated that in alternate embodiments, external devices may be located inside cabin 600. The external devices may include a server computing system, personal computing system, portable electronic device, electronic wrist band, electronic head band, portable music player, electronic activity tracking device, pedometer, smart-watch, GPS system, and so on. External devices 650 may be connected to the in-vehicle computing system via a communication link 636 which may be wired or wireless, as discussed with reference to communication link 630, and configured to provide two-way communication betw een the external devices and the in-vehiclecomputing system. For example, external devices 650 may include one or more sensors and communication link 636 may transmit sensor output from external devices 650 to in-vehicle computing system or infotainment system 609 and touch screen 608. External devices 650 may also store and / or receive information regarding contextual data, user behavior / preferences, operating rules, and so on. and may transmit such information from the external devices 650 to in-vehicle computing system or infotainment system 609 and touch screen 608.

[0094] In-vehicle computing system or infotainment system 609 may analyze the input received from external devices 650, mobile device 628, and / or other input sources and select settings for various in-vehicle systems (such as climate control system or audio system), provide output via touch screen 608 and / or speakers 612, communicate with mobile device 628 and / or external devices 650. and / or perform other actions based on the assessment. In some embodiments, all or a portion of the assessment may be performed by the mobile device 628 and / or the external devices 650.

[0095] In some embodiments, one or more of the external devices 650 may be communicatively coupled to in-vehicle computing system or infotainment system 609 indirectly, via mobile device 628 and / or another of the external devices 650. For example, communication link 636 may communicatively couple external devices 650 to mobile device 628 such that output from external devices 650 is relayed to mobile device 628. Data received from external devices 650 may then be aggregated at mobile device 628 with data collected by mobile device 628, the aggregated data then transmitted to in-vehicle computing system or infotainment system 609 and touch screen 608 via communication link 630. Similar data aggregation may occur at a server system and then transmitted to in-vehicle computing system or infotainment system 609 and touch screen 608 via communication link 636 and / or communication link 630.

[0096] FIG. 7 shows a block diagram of an in-vehicle computing system or infotainment system 609 configured and / or integrated inside vehicle 602. In-vehicle computing system or infotainment system 609 may perform one or more of the methods described herein in some embodiments. In some examples, the in-vehicle computing system or infotainment system 609 may be a vehicle infotainment system configured to provide information-based media content (audio and / or visual media content, including entertainment content, navigational services, and so on) to a vehicle user to enhance the operator’s in-vehicle experience. The in-vehicle computing system or infotainment system 609 may include, or be coupled to, various vehicle systems, sub-systems, hardware components, as well as software applications and systems that are integrated in, or integratable into, vehicle 602 in order to enhance an in-vehicle experiencefor a driver and / or a passenger. Further, the in-vehicle computing system may be coupled to systems for providing autonomous vehicle control.

[0097] In-vehicle computing system or infotainment system 609 may include one or more processors including an operating system processor 714 and an interface processor 720. Operating system processor 714 may execute an operating system on the in-vehicle computing system, and control input / output, display, playback, and other operations of the in-vehicle computing system. Interface processor 720 may interface with a vehicle control system 730 via an inter-vehicle system communication module 722.

[0098] Inter-vehicle system communication module 722 may output data to one or more other vehicle systems 731 and / or one or more other vehicle control elements 761, while also receiving data input from other vehicle systems 731 and other vehicle control elements 761, e.g. by way of vehicle control system 730. When outputting data, inter-vehicle system communication module 722 may provide a signal via a bus corresponding to any status of the vehicle, the vehicle surroundings, or the output of any other information source connected to the vehicle. Vehicle data outputs may include, for example, analog signals (such as cunent velocity), digital signals provided by individual information sources (such as clocks, thermometers, location sensors such as GPS sensors, and so on), digital signals propagated through vehicle data networks (such as an engine CAN bus through which engine related information may be communicated, a climate control CAN bus through which climate control related information may be communicated, and a multimedia data network through which multimedia data is communicated between multimedia components in the vehicle). For example, vehicle data outputs may be output to vehicle control system 730, and vehicle control system 730 may adjust vehicle control elements 761 based on the vehicle data outputs. As another example, the in-vehicle computing system or infotainment system 609 may retrieve from the engine CAN bus the current speed of the vehicle estimated by the wheel sensors, a power state of the vehicle via a batten and / or power distribution system of the vehicle, an ignition state of the vehicle, and so on. In addition, other interfacing means such as Ethernet may be used as well without departing from the scope of this disclosure.

[0099] A storage device 708 may be included in in-vehicle computing system or infotainment system 609 to store data such as instructions executable by operating system processor 714 and / or interface processor 720 in non-volatile form. The storage device 708 may store application data, including prerecorded sounds, to enable the in-vehicle computing system or infotainment system 609 to run an application for connecting to a cloud-based server and / or collecting information for transmission to the cloud-based server. The application may retrieveinformation gathered by vehicle systems / sensors, input devices (e.g., a user interface 718), data stored in one or more storage devices, such as a volatile memory 719A or a non-volatile memory 719B, devices in communication with the in-vehicle computing system (e.g., a mobile device connected via a Bluetooth® link), and so on. In-vehicle computing system or infotainment system 609 may further include a volatile memory' 719A. Volatile memory' 719A may be RAM. Non-transitory storage devices, such as and / or non-volatile memory 719B, may store instructions and / or code that, when executed by a processor (e.g., operating system processor 714 and / or interface processor 720), controls the in-vehicle computing system or infotainment system 609 to perform one or more of the actions described in the disclosure.

[0100] A microphone 702 may be included in the in-vehicle computing system or infotainment system 609 to receive voice commands from a user, to measure ambient noise in the vehicle, to determine whether audio from speakers of the vehicle is tuned in accordance with an acoustic environment of the vehicle, and so on. A speech processing unit 704 may process voice commands, such as the voice commands received from the microphone 702. In some embodiments, in-vehicle computing system or infotainment system 609 may also be able to receive voice commands and sample ambient vehicle noise using a microphone included in an audio system 732 of the vehicle.

[0101] One or more additional sensors may be included in a sensor subsystem 710 of the in-vehicle computing system or infotainment system 609. For example, the sensor subsystem 710 may include a plurality of cameras 725. such as a rear view camera for assisting a user in parking the vehicle and / or other external cameras, radars, lidars, ultrasonic sensors, and the like. The sensor subsystem 710 may include an in-cabin camera (e.g., a dashboard cam) for identifying a user (e.g., using facial recognition and / or user gestures). For example, an in-cabin camera may be used to identify one or more users of the vehicle via facial recognition software, and / or to detect a status or state of the one or more users (e.g., drowsy, distracted, stressed, high cognitive load, and so on) Sensor subsystem 710 of in-vehicle computing system or infotainment system 609 may communicate with and receive inputs from various vehicle sensors and may further receive user inputs. For example, the inputs received by sensor subsystem 710 may include transmission gear position, transmission clutch position, gas pedal input, brake input, transmission selector position, vehicle speed, engine speed, mass airflow through the engine, ambient temperature, intake air temperature, and so on, as w ell as inputs from climate control system sensors (such as heat transfer fluid temperature, antifreeze temperature, fan speed, passenger compartment temperature, desired passenger compartment temperature, ambient humidify, and so on), an audio sensor detecting voice commands issuedby a user, a fob sensor receiving commands from and optionally tracking the geographic location / proximity of a fob of the vehicle, and so on.

[0102] One or more additional sensors may be included in and / or communicatively coupled to a sensor subsystem 710 of the in-vehicle computing system 609. For example, the sensor subsystem 710 may include and / or be communicatively coupled to a camera, such as a rear view camera for assisting a user in parking the vehicle, a cabin camera for identifying a user, and / or a front view camera to assess quality of the route segment ahead. The abovedescribed cameras may also be used to provide images to a computer vision-based facial recognition and / or facial analysis module. For example, the facial analysis module may be used to determine an emotional or psychological state of users of the vehicle. Sensor subsystem 710 of in-vehicle computing system 609 may communicate with and receive inputs from various vehicle sensors and may further receive user inputs.

[0103] While certain vehicle system sensors may communicate with sensor subsystem 710 alone, other sensors may communicate with both sensor subsystem 710 and vehicle control system 730, or may communicate with sensor subsystem 710 indirectly via vehicle control system 730. Sensor subsystem 710 may serve as an interface (e.g.. ahardware interface) and / or processing unit for receiving and / or processing received signals from one or more of the sensors described in the disclosure.

[0104] A navigation subsystem 711 of in-vehicle computing system or infotainment system 609 may generate and / or receive navigation information such as location information (e.g.. via a GPS sensor and / or other sensors from sensor subsystem 710), route guidance, traffic information, point-of-interest (POI) identification, and / or provide other navigational services for the user. Navigation sub-system 711 may include inputs / outputs including analog to digital converters, digital inputs, digital outputs, network outputs, radio frequency transmitting devices, and so on. In some examples, navigation sub-system 711 may interface with vehicle control system 730.

[0105] An external device interface 712 of in-vehicle computing system or infotainment system 609 may be coupleable to and / or communicate with one or more external devices 650 located external to vehicle 602. While the external devices are illustrated as being located external to vehicle 602, it is to be understood that they may be temporarily housed in vehicle 602, such as when the user is operating the external devices while operating vehicle 602. In other words, the external devices 650 are not integral to vehicle 602. The external devices 650 may include a mobile device 628 (e.g., connected via a Bluetooth®, NFC, WI-FI Direct®, or other wireless connection) or an alternate Bluetooth®-enabled device 752.

[0106] Mobile device 628 may be a mobile phone, smart phone, wearable devices / sensors that may communicate with the in-vehicle computing system via wired and / or wireless communication, or other portable electronic device(s). Other external devices include one or more external services 746. For example, the external devices may include extra-vehicular devices that are separate from and located externally to the vehicle. Still other external devices include one or more external storage devices 754, such as solid-state drives, pen drives, USB drives, and so on. External devices 650 may communicate with in-vehicle computing system or infotainment system 609 either wirelessly or via connectors without departing from the scope of this disclosure. For example, external devices 650 may communicate with in-vehicle computing system or infotainment system 609 through the external device interface 712 over a network 760, a USB connection, a direct wired connection, a direct wireless connection, and / or other communication link.

[0107] The external device interface 712 may provide a communication interface to enable the in-vehicle computing system to communicate with mobile devices associated with contacts of the driver. For example, the external device interface 712 may enable phone calls to be established and / or text messages (e.g., Short Message Service (SMS), Multimedia Message Service (MMS), and so on) to be sent (e g., via a cellular communications network) to a mobile device associated with a contact of the driver. The external device interface 712 may additionally or alternatively provide a wireless communication interface to enable the in- vehicle computing system to synchronize data with one or more devices in the vehicle (e.g., the driver’s mobile device) via Wi-Fi Direct®.

[0108] One or more applications 744 may be operable on mobile device 628. As an example, a mobile device application 744 may be operated to aggregate user data regarding interactions of the user with the mobile device. For example, mobile device application 744 may aggregate data regarding music playlists listened to by the user on the mobile device, telephone call logs (including a frequency and duration of telephone calls accepted by the user), positional information including locations frequented by the user and an amount of time spent at each location, and so on. The collected data may be transferred by application 744 to external device interface 712 over network 760. In addition, specific user data requests may be received at mobile device 628 from in-vehicle computing system or infotainment system 609 via the external device interface 712. The specific data requests may include requests for determining where the user is geographically located, an ambient noise level and / or music genre at the user’s location, an ambient weather condition (temperature, humidity, and so on) at the user’s location, and so on. Mobile device application 744 may send control instructions to components(e.g., microphone, amplifier, and so on) or other applications (e.g., navigational applications) of mobile device 628 to enable the requested data to be collected on the mobile device or requested adjustment made to the components. Mobile device application 744 may then relay the collected information back to in-vehicle computing system or infotainment system 609.

[0109] Likewise, one or more applications 748 may be operable on external services 746. As an example, external services applications 748 may be operated to aggregate and / or analyze data from multiple data sources. For example, external services applications 748 may aggregate data from one or more social media accounts of the user, data from the in-vehicle computing system (e.g., sensor data, log files, user input, and so on), data from an internet query (e.g., weather data, POI data), and so on. The collected data may be transmitted to another device and / or analyzed by the application to determine a context of the driver, vehicle, and environment and perform an action based on the context (e.g., requesting / sending data to other devices).

[0110] The one or more applications 748 operable on external services 746 may include a cloud-based driver model generation service, which may receive data of a driver of the vehicle from the vehicle 602. The data of the driver may include, for example, driving data (e.g.. acceleration style, braking style, steering style, and so on). The data of the driver may also include in-cabin environmental data, such as preferred settings for lighting, temperature, preferred audio content, typical cabin context data (e.g., how often the driver drives with passengers, whether the passengers are children, head movement and / or eye gaze patterns detected via a dashboard cam, and the like. The data of the driver may be used to generate a model or profile of the driver, which may be used, for example, to personalize an intervention by an ADAS system of the vehicle 602, or to personalize an adjustment to in-cabin environmental controls based on driver behavior.

[0111] Vehicle control system 730 may include controls for controlling aspects of various vehicle systems 731 involved in different in-vehicle functions. These may include, for example, controlling aspects of vehicle audio system 732 for providing audio entertainment to the vehicle occupants, aspects of a climate control system 734 for meeting the cabin cooling or heating needs of the vehicle occupants, as well as aspects of a telecommunication system 736 for enabling vehicle occupants to establish telecommunication linkage with others.

[0112] Audio system 732 may include one or more acoustic reproduction devices including electromagnetic transducers such as one or more speakers 735. Vehicle audio system 732 may be passive or active such as by including a power amplifier. In some examples, in-vehicle computing system or infotainment system 609 may be the only audio source for the acousticreproduction device, or there may be other audio sources that are connected to the audio reproduction system (e.g., external devices such as a mobile phone). The connection of any such external devices to the audio reproduction device may be analog, digital, or any combination of analog and digital technologies.

[0113] Climate control system 734 may be configured to provide a comfortable environment within the cabin or passenger compartment of vehicle 602. Climate control system 734 includes components enabling controlled ventilation such as air vents, a heater, an air conditioner, an integrated heater and air-conditioner system, and so on. Other components linked to the heating and air-conditioning setup may include a windshield defrosting and defogging system capable of clearing the windshield and a ventilation-air filter for cleaning outside air that enters the passenger compartment through a fresh-air inlet.

[0114] Vehicle control system 730 may also include controls for adjusting the settings of various vehicle control elements 761 (or vehicle controls, or vehicle system control elements) related to the engine and / or auxiliary elements within a cabin of the vehicle, such as one or more steering wheel controls 762 (e.g., steering wheel-mounted audio system controls, cruise controls, windshield wiper controls, headlight controls, turn signal controls, and so on), instrument panel controls, microphone(s), accelerator / brake / clutch pedals, a gear shift, door / window controls positioned in a driver or passenger door, seat controls, cabin light controls, audio system controls, cabin temperature controls, and so on. Vehicle control elements 761 may also include internal engine and vehicle operation controls (e.g., engine controller module, actuator controls, valves, and so on) that are configured to receive instructions via the CAN bus of the vehicle to change operation of one or more of the engine, exhaust system, transmission, and / or other vehicle system. The control signals may also control audio output at one or more speakers 735 of the vehicle’s audio system 732. For example, the control signals may adjust audio output characteristics such as volume, equalization, audio image (e.g., the configuration of the audio signals to produce audio output that appears to a user to originate from one or more defined locations), audio distribution among a plurality of speakers, and so on. Likewise, the control signals may control vents, air conditioner, and / or heater of climate control system 734. For example, the control signals may increase delivery of cooled air to a specific section of the cabin. For example, the control signals may increase delivery of cooled air to a specific section of the cabin. Additionally, while operating in an autonomous mode, the autonomous vehicle control system may control some or all of the above vehicle controls.

[0115] Vehicle controls 761 may include a steering control system 762, a braking control system 763, and an acceleration control system 764. Vehicle controls 761 may include additional control systems. In some example, vehicle controls 761 may be operated autonomously, such as during autonomous vehicle operation. In other examples, vehicle controls 761 may be controlled by a user. Further, in some examples, a user may primarily control vehicle controls 761, while one or more ADAS 765 may intermittently adjust vehicle controls 761 in order to increase vehicle performance. For example, the one or more ADAS 765 may include a cruise control system, a lane departure warning system, a collision avoidance system, an adaptive braking system, and the like.

[0116] Steering control system 762 may be configured to control a direction of the vehicle. For example, during a non-autonomous mode of operation, steering control system 762 may be controlled by a steering wheel. For example, the user may turn the steering wheel in order to adjust a vehicle direction.

[0117] Braking control system 763 may be configured to control an amount of braking force applied to the vehicle. For example, during a non-autonomous mode of operation, braking control system 763 may be controlled by a brake pedal. For example, the user may depress the brake pedal in order to increase an amount of braking applied to the vehicle.

[0118] Acceleration control system 764 may be configured to control an amount of acceleration applied to the vehicle. For example, during anon-autonomous mode of operation, acceleration control system 764 may be controlled by an acceleration pedal. For example, the user may depress the acceleration pedal in order to increase an amount of torque applied to wheels of the vehicle, causing the vehicle to accelerate in speed.

[0119] Control elements positioned on an outside of a vehicle (e.g.. controls for a security system) may also be connected to in-vehicle computing system or infotainment system 609, such as via inter-vehicle system communication module 722. The control elements of vehicle control system may be physically and permanently positioned on and / or in the vehicle for receiving user input. In addition to receiving control instructions from in-vehicle computing system or infotainment system 609, vehicle control system 730 may also receive input from one or more external devices 650 operated by the user, such as from mobile device 628. This allows aspects of vehicle systems 731 and vehicle control elements 761 to be controlled based on user input received from the external devices 650.

[0120] In-vehicle computing system or infotainment system 609 may further include one or more antennas 706. The in-vehicle computing system may obtain broadband wireless internet access via antennas 706, and may further receive broadcast signals such as radio,television, weather, traffic, and the like. The in-vehicle computing system or infotainment system 609 may receive positioning signals such as GPS signals via antennas 706. The in- vehicle computing system may also receive wireless commands via radio frequency (RF such as via antennas 706 or via infrared or other means through appropriate receiving devices. In some embodiments, antenna 706 may be included as part of audio system 732 or telecommunication system 736. Additionally, antenna 706 may provide AM / FM radio signals to external devices 650 (such as to mobile device 628) via external device interface 712.

[0121] One or more elements of the in-vehicle computing system or infotainment system 609 may be controlled by a user via user interface 718. User interface 718 may include a graphical user interface presented on a touch screen, such as touch screen 608 and / or display screen 611 of FIG. 6, and / or user-actuated buttons, switches, knobs, dials, sliders, etc. For example, user-actuated elements may include steering wheel controls, door and / or window controls, instrument panel controls, audio system settings, climate control system settings, and the like. A user may also interact with one or more applications of the in-vehicle computing system or infotainment system 609 and mobile device 628 via user interface 718. In addition to receiving a user’s vehicle setting preferences on user interface 718. vehicle settings selected by in-vehicle control system may be displayed to a user on user interface 718. Notifications and other messages (e.g., received messages), navigational assistance, advertisements, and / or other information may be displayed to the user on a display of the user interface. User preferences / information and / or responses to presented messages may be performed via user input to the user interface.

[0122] The in-vehicle computing system or infotainment system 609 may include a DMS 721. The DMS 721 may receive data from various sensors and / or systems of the vehicle (e.g., sensor subsystem 710, cameras 725, microphone 702) and may monitor aspects of driver behavior to increase a performance of the vehicle and / or a driving experience of the driver. In some examples, one or more outputs of the DMS 721 may be inputs into a driver model 723. In various embodiments, the driver model 723 may be used to estimate a cognitive state of the driver, to determine whether to display an ad to the driver based on the estimated cognitive state of the driver.

[0123] Referring now to FIG. 8, an example machine-implemented method 800 is shown for determining whether to display an ad to a driver of a vehicle during operation of the vehicle, based on one or more of a driving sty le model of a driver of a vehicle, driver status data of the driver, and environmental sensor information of the vehicle. The driving style model, driver status data, and environmental sensor information may be non-limiting examples of drivermodel 304, driver status 308, and environmental inputs 302 of FIG. 3, respectively. Instructions for carrying out method 800 may be stored in a memory of the vehicle and executed by a controller of the vehicle, such as memory 106 and controller 102, respectively, of FIG. 1.

[0124] At 802, method 800 includes estimating and / or measuring vehicle operating conditions. For example, the vehicle operating conditions may include, but are not limited to, a status of an engine of the vehicle (e.g., whether the engine is switched on), and an engagement of one or more gears of a transmission of the vehicle (e.g., whether the vehicle is moving). Vehicle operating conditions may include engine speed and load, vehicle speed, transmission oil temperature, exhaust gas flow rate, mass air flow rate, coolant temperature, coolant flow rate, engine oil pressures (e.g., oil gallery pressures), operating modes of one or more intake valves and / or exhaust valves, electric motor speed, battery’ charge, engine torque output, vehicle wheel torque, and so on. In one example, the vehicle is a hybrid electric vehicle, and estimating and / or measuring vehicle operating conditions includes determining whether the vehicle is being powered by an engine or an electric motor.

[0125] Estimating and / or measuring vehicle operating conditions may also include determining a driver of the vehicle. For example, the driver may be identified by a key fob of the driver. When the driver is identified, a driver profile of the driver stored in a memory of the vehicle may be accessed, and settings of the vehicle may be adjusted based on the driver profile. Additionally, various driving sty le inputs (e.g., driving style inputs 304) of the driver may be identified from the driver profile.

[0126] At 804, method 800 includes collecting driver status data of the driver. In various embodiments, the driver status data may include data from a DMS system of the vehicle. The DMS system may be substantially similar to DMS 140 of FIG. 1, and the data from the DMS system may include a portion or all of the DMS data 204 of FIG. 2. The driver status data and / or the data from the DMS system may include data from one or more interior (e.g., dashboard) cameras of the vehicle; biometric data received from one or more sensors arranged at the steering wheel, seat, or at another location in the vehicle; a level of interior illumination inside the vehicle, and whether the interior illumination is automatic, manually selected, and / or a result of the driver opening a door of the vehicle; and / or seat occupancy data received from seat sensors of the vehicle and / or in-cabin cameras of the vehicle.

[0127] At 806, method 800 includes collecting environmental sensor data of the vehicle. Collecting the environmental sensor data includes collecting data from an IVI system of the vehicle. The IVI system may be substantially similar to IVI system 124 of FIG. 1, and the data from the IVI system may include a portion or all of the IVI system data 210 of FIG. 2. Forexample, the data from the I VI system may include a detection of whether the driver has received an incoming phone call or is making an outgoing phone call; selection data of audio content of an audio system of the vehicle, such as a radio station selection or selection of a different media source, and / or a selection of an individual track of the different media source (e.g., a song from a collection of songs, or podcast); and / or one or more settings of the IVI system, such as a language, or a personalization of the IVI system. It should be appreciated that the above list is provided for illustrative purposes, and more, less, or different data may be included in the IVI system data without departing from the scope of this disclosure.

[0128] Collecting the environmental sensor data includes collecting data from a navigational system of the vehicle. The navigational system may be substantially similar to navigational guidance system 126 of FIG. 1. and the data from the navigation system may include a portion or all of the navigation system data 224 of FIG. 2. For example, the data from the navigation system may include a detection that a route to a destination manually selected by the driver has been initiated, completed, or is currently underway. The data from the navigation system may also include a detection of any changes made to the route by the driver, such as switching from a first route proposed by the navigation system to a second route proposed by the navigation system, and / or any manual changes made to the route by the driver, such as any additional destinations added. The data from the navigation system may include data regarding an interaction between the driver and the navigation system, such as whether the driver is currently changing settings of the navigation system via a voice interface or a touchscreen interface. For example, the driver may be manually specifying a destination, visually examining a portion of the route, zooming in or zooming out of a display of the route, or using an interactive mode selection of the navigation software. It should be appreciated that the above list is provided for illustrative purposes, and more, less, or different data may be included in the navigation system data without departing from the scope of this disclosure.

[0129] Collecting the environmental sensor data includes collecting data from one or more bus systems of the vehicle. The one or more bus systems of the vehicle may include, for example, a CAN bus, a LIN, bus and / or an automotive Ethernet network of the vehicle. The one or more bus systems may be substantially similar to vehicle bus systems 128 of FIG. 1. and the data from the one or more bus systems may include a portion or all of the vehicle bus data 250 of FIG. 2. For example, the data from the bus systems may include object proximity and / or detection data received from one or more exterior cameras, proximity sensors, radar and / or lidar systems of the vehicle. The data from the bus systems may include data from one or more in-cabin sensors of the vehicle, such as a temperature of the cabin, a level of lightingof the cabin, or level of noise in the cabin. The data from the bus systems may include vehicle control data during operation of the vehicle, such as a speed of the vehicle (e.g., from a wheel speed sensor), a position of a brake pedal of the vehicle, a position of an accelerator of the vehicle, and a position of a steering wheel the vehicle (e.g., an angle of the steering wheel from a default angle of zero when the vehicle is operating in a straight line and not turning). The data from the bus systems may also include status data of one or more windshield wipers of the vehicle, one or more turn signals of the vehicle, and / or a position of a movable roof (e.g.. open or closed, open to a certain degree) of the vehicle. The data from the bus systems may also include, for example, an engine status of the vehicle, one or more system warnings indicated to a user via a dashboard of the vehicle, engine temperature data, fuel level data, or additional and / or alternative data from other systems of the vehicle.

[0130] Collecting the environmental sensor data may include collecting data from one or more V2X / Telematics modules in an environment of the vehicle. As described above in reference to FIGS. 1-3, the one or more V2X / Telematics modules may transmit sensor data including meteorological / weather data, road condition data, traffic data, and / or other types of data.

[0131] It should be appreciated that the above lists are provided for illustrative purposes, and more, less, or different data may be included in the IVI system data, the navigation system data, and the bus system data without departing from the scope of this disclosure.

[0132] At 808, method 800 includes reformatting the collected driver status data and environmental sensor data to reduce a dimensionality of the collected data so that the collected data may be inputted into an ad timing model, such as ad timing model 112 of FIG. 1 and / or local ad timing model 506 of FIG. 5.

[0133] Reformatting the collected driver status data may include pre-processing the collected driver status data to estimate a psychological or emotional state of the driver from raw DMS data such as postural data, eye-gaze data, etc., and generate an encoding the psychological or emotional state. For example, the dashboard camera data and biometric data may track head movements and / or an eye gaze of the driver, and the DMS system may detect patterns in the head movements and / or eye gaze, and may further codify the detected patterns. The codifications of the detected patterns may be used as input into various processes of the DMS system. The various processes of the DMS system may analyze the codifications of detected patterns along with other data of the DMS system to generate an assessment of a physiological state of the user.

[0134] In various embodiments, a level of stress, a level of drowsiness, and / or a level of distraction of the driver may be determined by analyzing movements of the driver captured by a dashboard camera of the DMS or of the vehicle. If the movements match a pattern of movements associated with a distracted driver (e.g., if the driver is not frequently looking at a road the vehicle is on), then it may be inferred that the driver is distracted. Alternatively, if movements associated with the distracted driver are detected a threshold number of times, and / or for a threshold duration, it may be inferred that a threshold level of distraction is achieved. The level of distraction may also be determined by atone or quality of a voice of the driver when the driver is on a phone call, or by a combination of different detected patterns. Additionally or alternatively, the level of stress may be determined by biometric data of the driver (e.g., the biometric data 288 of FIG. 2) such as a pulse rate, where if the pulse rate of the driver exceeds a typical driving pulse rate of the driver by a percent (e.g., 30%), then it may be inferred that the threshold level of stress is achieved. For example, the pulse rate of the driver may be detected by a sensor of a steering wheel of the vehicle.

[0135] The encoding of the psychological or emotional state of the driver may be a single binary / numeric or multi-dimensional score indicating a degree to which the driver is anxious, stressed, distracted, etc. In some examples, a cognitive model (e.g., of the DMS) may be relied on to determine the psychological or emotional state.

[0136] Reformatting the collected environmental sensor data may include grouping elements of the collected environmental sensor data into different groups, and analyzing the collected environmental sensor data of each different group to determine a relative contribution of each element of the collected environmental sensor data of that group. Determining the relative contribution may include, for example, applying a rules-based system to assign rankings and / or weights to each element, or applying a different kind of model (e.g., an ML model). The assigned rankings and / or weights may then be converted into a numeric score or encoding for each group of the different groups. The numeric scores or encodings may be inputted into input nodes of the ad timing model. In this way, an amount of data relied on by the ad timing model to determine whether or not to display an ad to the driver may be reduced, increasing an efficiency of the training and deployment of the ad timing model.

[0137] After an ad is displayed, ground truth data may be collected from the driver to determine whether the ad timing model displayed the ad at a suitable time (e.g., when conditions were suitable). The ground truth data and the data used by the ad timing model may be transmitted to an ad timing training system (e.g.. ad timing training system 430 of FIG. 4), for further training the ad timing model. Thus, by reformatting the collected environmentalsensor data, the amount of transmitted data may be greatly reduced. That is, in an alternative ad timing training system where the collected environmental sensor data is transmitted from the vehicle to the alternative ad timing training system, and training pairs for training the ad timing model are generated at the alternative ad timing training system based on the collected environmental sensor data, significantly more bandwidth may be consumed during the transmission, which may limit an amount of collected environmental sensor data that can be feasibly sent to the alternative ad timing training system, which may result in a poorer posttraining performance of the ad timing model.

[0138] For example, environmental sensor data relating to external weather conditions of the vehicle may be grouped into a first grouping. The environmental sensor data relating to external weather conditions may include external sensor / camera data of the vehicle; windshield wiper status data; roof status data; light status data; and navigation system data. Based on the environmental sensor data relating to external weather conditions, a weather status score may be generated, where a higher weather status score may indicate easier and / or safer driving conditions, and a lower weather status score may indicate harder and / or more risky driving conditions.

[0139] Environmental sensor data relating to traffic conditions of the vehicle may be grouped into a second grouping. The environmental sensor data relating to traffic conditions may include external sensor / camera data of the vehicle; brake pedal and acceleration pedal data; navigation system data; proximity sensor data; radar data; engine stop / start status data; and V2X data received from connected vehicles in a vicinity of the vehicle. Based on the environmental sensor data relating to traffic conditions, a traffic status score may be generated, where a higher traffic status score may indicate more stressful driving conditions, and a lower traffic status score may indicate less stressful driving conditions.

[0140] The environmental sensor data may also be grouped into other groupings or categories, to generate other types of interpretive scores. The weather status score, the traffic score, and the other ty pes of interpretive scores may be inputs into the ad timing model, both at the vehicle to determine whether to display an ad to the driver, and during further training of the ad timing model at the ad timing training system.

[0141] At 810, method 800 includes predicting a suitability of conditions for displaying an ad to the driver, based on the driver status data, the environmental sensor data, and the driver model, using the ad timing model. In various embodiments, the ad timing model may take as input reformatted elements of the driver status data, the environmental sensor data, and thedriver model, and may output a prediction of a suitability of conditions represented by the input elements for displaying the ad at a current time.

[0142] In some embodiments, the predicted suitability of the conditions may be a binary value, where a zero may indicate that the conditions are not predicted to be suitable for displaying the ad, and a one may indicate that the conditions are predicted to be suitable for displaying the ad. In other embodiments, the predicted suitability of the conditions may be a value indicating a degree of suitability of the conditions for displaying the ad. If the degree of suitability of the conditions exceeds a threshold suitability, the ad may be displayed. If the degree of suitability of the conditions does not exceed the threshold suitability, the ad may not be displayed.

[0143] In some embodiments, the predicted suitability of conditions for displaying the ad may be explicitly indicated by the driver. That is, the ad delivery system may postpone display of the ad until the driver requests to display the ad. The ad delivery system may notify the driver that an ad is available to display, and the driver may select to view the ad at a suitable time, via one or more controls or virtual controls (e.g., on a touchscreen display) of the vehicle. Additionally or alternatively, the ad delivery system may deliver ads to a queue, and the driver may view the ads sequentially when convenient. When the driver selects to view an ad, the ad delivery7system may infer that the degree of suitability7of the conditions exceeds the threshold suitability.

[0144] The ad delivery system may notify the driver that an ad may be displayed on demand in various ways. In some embodiments, an audio notification may be made via a speaker of the vehicle (e.g., speaker 612), where a volume and timing of the notification may be determined by the ad delivery7system to avoid startling the driver. An indication of the ad may be shown via a visual element displayed in a display of the vehicle, or via an indicator light. Additionally or alternatively, haptic feedback may be used, for example, where gentle vibrations or pulses may be generated at a seat or seat belt of the driver, or at the steering yvheel. A pattern of the vibrations or pulses may vary depending on a type of ad or notification available to display. It should be appreciated that the examples provided herein are for illustrative purposes, and the driver may be notified of an available ad in other ways without departing from the scope of this disclosure.

[0145] In various embodiments, the driver may be incentivized to vieyv the ads. For example, the driver may earn points towards features of the vehicle that are unlocked upon a target number of points being earned, or may be awarded discounts on vehicle features or subscription services (e.g., music, news, or entertainment services, etc.). Additionally oralternatively, the ads may include special offers or discounted prices that may be attractive to the driver. As one example, the vehicle may be operating along a route during a meal time, and a restaurant located on the route may be participating in a vehicle-oriented ad campaign that offers discounts to drivers passing by. When the vehicle enters a threshold proximity of the restaurant, the ad delivery system may notify the driver that the restaurant is offering discounted prices to the driver. In response to the notification, the driver may select a control on a steering wheel of the vehicle. In response to the driver selecting the control, the ad delivery system may display an ad for the restaurant including a discount code to the driver. The driver may view the ad, and may decide to visit the restaurant based on the content of the ad. The driver may request, for example, by voice, that the discount code be transferred to a connected smart phone of the driver (e.g.. mobile device 628 of FIG. 6). The ad delivery system may send a text message or notification to the connected smart phone with the discount code. The driver may then arrive at the restaurant, and may use the discount code on the smart phone when ordering food at the restaurant.

[0146] As another example, a manufacturer of the vehicle may offer a premium on-board subscription music system that the driver may purchase. The ad delivery system may offer the driver a month of the subscription service for free in response to agreeing to view an ad for the sendee.

[0147] At 812, method 800 includes determining whether the conditions are met for displaying the ad, based on the output of the ad timing model. If at 810 it is determined that the conditions are not met, method 800 proceeds back to 804, where driver status data and environmental sensor data continue to be collected. Alternatively, if at 810 it is determined that the conditions are met, method 800 proceeds to 812.

[0148] At 814, method 800 includes displaying the ad to the driver. In various embodiments, the ad may be displayed to the driver on a dashboard display of the vehicle (e.g., display screen 611 of FIG. 6). In some embodiments, the ad may be projected on a portion of a windshield of the vehicle.

[0149] At 816, method 800 includes determining a success score of the ad (e.g., a success of the timing of the ad), where the success score indicates a degree to which the conditions for displaying the ad were met based on a reaction of the driver to the ad. In one embodiment, the success score may be a percentage value. For example, a first success score may be 90, indicating that the conditions for displaying the ad were 90% met (e.g., where it may have been appropriate to display the ad based on the input data). A second success score may be 50, indicating that the conditions for displaying the ad were 50% met (e.g., where it may not havebeen appropriate to display the ad based on the input data). In a case where the driver is notified of an ad, and the driver selects to view the ad, a success score of 100% may be assigned to the ad, indicating that the conditions for displaying the ad were explicitly met. The success score may be used as ground truth data for further training of the ad timing model.

[0150] The success score may be generated by a local ad success model (e.g., local ad success model 116 of FIGS. 1 and 3) of the ad delivery system. The local ad success model may generate the success score at least partially based on driver status data collected from the driver by the DMS system concurrently with the display of the ad and immediately after the display of the ad. In other words, a first set of DMS data may be used by the ad delivery7system to determine whether to display the ad, and a second set of DMS data that captures the driver’s reaction to the ad may be used by the ad delivery system to determine the success score of the ad. The local ad success model may be a copy of a master ad success model (e.g., ad success model 445 of FIG. 4) stored in an ad timing training system, and the master ad success model may be trained in a manner similar to the master ad timing model used to generate the local ad timing model. That is, the master ad success model may be trained using ground truth feedback data collected from the driver, as described below.

[0151] The success score may also be determined at least partially based on feedback regarding the ad provided by the driver. For example, after the ad is displayed, the driver may be prompted to indicate whether the timing of the ad was suitable, or to rate the suitability of the timing of the ad. The driver may provide the feedback by selecting or adjusting a control of the vehicle, such as a button on a dashboard of the vehicle or steering wheel of the vehicle, or a virtual control element displayed on a display screen of the vehicle. Alternatively, in some embodiments, the driver may provide the feedback by voice, which may be captured by the ad delivery system via an in-cabin microphone of the vehicle (e.g., microphone 702), or via a gesture, such as a hand gesture, which may be captured by an in-cabin camera or sensor of the vehicle (e.g., cameras 725).

[0152] At 818, method 800 includes creating a training pair to be inputted into the ad timing model during training, based on the ad. The training pair may include a set of input data (e.g., input data 524 of FIG. 5), which may include a set of scores (e.g., weather status score, traffic status score, etc.) based on groupings of elements of the driver status data, the environmental sensor data, and the driver model, as described above. The training pair may include the success score as ground truth data (e.g., ground truth data 526). At 820, method 800 includes storing the training pair in a memory7of the vehicle.

[0153] At 822, method 800 includes determining whether the conditions are met for sending the training pairs to the ad timing training system. In some examples, the training pairs may be sent to the ad timing training system upon generation of the training pairs. In other examples, the conditions for sending the training pairs to the ad timing training system may be met when a threshold amount of training pairs has been generated. In other examples, the conditions may be met when a request is received at the vehicle for the training pairs from the ad timing training system, or when a threshold amount of time has passed. If at 822 it is determined that the conditions are not met, method 800 proceeds back to 804, where driver status data and environmental sensor data continue to be collected. Alternatively, if at 822 it is determined that the conditions are met, method 800 proceeds to 824.

[0154] At 824. method 800 includes sending the training pairs to the ad timing training system. In various embodiments, the ad timing training system may be a cloud-based ad timing training system, and the training pairs may be sent via a wireless network, such as wireless network 420 of FIG. 4. At the ad timing training system, a master ad timing model may be further trained on the training pairs, as described below in reference to FIG. 9.

[0155] It should be appreciated that method 800 may be performed in an iterative or cyclical manner, where after the training pairs are sent to the ad timing training system, the driver status data and environmental sensor data may continue to be collected and used for displaying ads to drivers as described above. Additionally, the training pairs may continue to be generated and stored until conditions are met for sending to the ad timing training system. That is, in various embodiments, the ad timing model may be periodically retrained and refined over time as additional data regarding the display of ads is collected.

[0156] Referring now to FIG. 9, an exemplary' method 900 is shown for training a master ad timing model (e.g., master ad timing model 502 of FIG. 5). based on training pair data collected at one or more vehicles. In various embodiments, the master ad timing model may be trained at an ad timing training system, such as ad timing training system 430 of FIG. 4. Instructions for carrying out method 900 may be stored in a memory and executed by a processor of the ad timing training system, such as non-transitory memory 436 and processor 434 of ad timing training system 430. While method 900 describes a supervised learning approach to training the master ad timing model, using the training pairs generated as described above in reference to method 800 of FIG. 8, it should be appreciated that in other embodiments, the master ad timing model may be trained using a different approach, such as, for example, a semi-supervised or unsupervised learning framework.

[0157] Method 900 begins at 902, where method 900 includes receiving training pairs from a plurality of vehicles. The training pairs may be generated and transmitted as described above in reference to FIG. 8.

[0158] At 904, method 900 includes storing the training pairs in the memory until a sufficient amount of training pairs has been received. At 906, method 900 includes determining whether the sufficient amount of training pairs has been received. If at 906 it is determined that the sufficient amount of training pairs has not been received, method 900 proceeds back to 902. and training pairs continue to be received and stored. If at 906 it is determined that the sufficient amount of training pairs has been received, method 900 proceeds to 908.

[0159] At 908, method 900 includes training the master ad timing model on the received training pairs. In various embodiments, the master ad timing model may be a deep learning (DL) neural network, such as a convolutional neural network (CNN). The CNN may include one or more convolutional layers, which in turn comprise one or more convolutional filters. The convolutional filters may comprise a plurality of weights, wherein the values of the weights are learned during a training procedure. The convolutional filters may correspond to one or more features / pattems, thereby enabling the master ad timing model to identify and extract features from the input data.

[0160] Training the master ad timing model may include iteratively the input data of corresponding training pairs into an input layer of the master ad timing model. The master ad timing model maps the input data to an output layer, which may output an ad timing score. The ad timing score may be a prediction of a suitability of conditions for displaying an ad to a driver of the vehicle at a point in time, given the input data collected at the point in time. The master ad timing model may propagate the input data from the input layer, through one or more hidden layers, until reaching the output layer.

[0161] The master ad timing model may be configured to iteratively adjust one or more of a plurality of parameters (e.g., weights) of the master ad timing model in order to minimize a loss function, based on an assessment of a difference between the ad timing score outputted by the master ad timing model and the ground truth data of the relevant training pair. The weights of the master ad timing model may then be adjusted based on the difference. The difference (or loss), as determined by the loss function, may be back-propagated through the master ad timing model to update the weights of the hidden (convolutional) layers. In some embodiments, back propagation of the loss may occur according to a gradient descent algorithm, wherein a gradient of the loss function (a first derivative, or approximation of the first derivative) is determined for each weight of the deep neural network. Each weight of the master ad timingmodel is then updated by adding the negative of the product of the gradient determined (or approximated) for the weight with a predetermined step size. Updating of the weights may be repeated until the weights of the master ad timing model converge, or the rate of change of the weights of the deep neural network for each iteration of weight adjustment are under a threshold. It should also be noted that back-propagation is used as an example, and that other optimization schemes are valid for fitting the master ad timing model’s parameters.

[0162] At 910. method 900 includes validating the master ad timing model. In order to avoid overfitting, training of the master ad timing model may be periodically interrupted to validate a performance of the master ad timing model on the validation data set, as described above in reference to FIG. 5. Training of the master ad timing model may end when a performance of the master ad timing model on the validation data set converges (e.g., when an error rate on the validation data set converges on or to within a threshold of a minimum value). In this way, the master ad timing model may be trained to leam to predict the ad timing score, which may indicate a suitability7of conditions for displaying an ad to the driver given a new set of input data (e.g., new environmental sensor data and new driver status data).

[0163] At 912. if the master ad timing model is not validated, method 900 proceeds back to 902, and training pairs continue to be received and stored for further training. Alternatively, if the master ad timing model is validated at 912, method 900 proceeds to 914.

[0164] At 914, method 900 includes sending a copy of the master ad timing model to one or more vehicles, where the master ad timing model may replace a local ad timing model (e.g., local ad timing model 112) used to determine when to display ads to drivers of the vehicles. For example, the copy of the master ad timing model may be sent to the one or more vehicles via an over-the-air (OTA) update. After the copy of the master ad timing model is sent to the one or more vehicles, method 900 may be started again, where new training pairs may be received from the one or more vehicles and the master ad timing model may be further trained on the new training pairs. Method 900 ends.

[0165] Thus, an advertisement delivery7system for a vehicle is disclosed herein that may determine, based on current sensor data collected at the vehicle, whether an ad could be displayed to a driver of the vehicle at a current time. The current sensor data is first reformatted to reduce a size and number of dimensions of the current sensor data, by grouping the current sensor data into different categories of data that may inform the determination (e.g., weather, traffic, driver state, etc). Numerical scores may be generated for each of the categories. The reformatted sensor data is inputted into an ad timing model, such as a ML model, that outputs an assessment of the current time for displaying the ad.

[0166] Additionally, the ad is displayed to the driver in a manner such that explicit feedback regarding the ads may be provided by the driver. After the ad has been displayed, the driver may provide feedback in a variety of ways, such as via a control element in a dashboard display, a button on a steering wheel of the vehicle, by voice, etc. The feedback may be processed along with driver reaction data captured by an in-cabin camera and / or microphone of the vehicle, to generate a success score that indicates whether the timing of the ad was successful (e.g., was viewed by the driver without increasing an amount of stress of the driver or increasing an amount of driving risk). A training pair for training the ad timing model is then generated, where input data of the training pair includes the reformatted sensor data of the vehicle, and the success score as ground truth. The training pair may be sent to a remote ad timing training system and used to train the ML model to leam to predict when to display ads to drivers during operation of the vehicle.

[0167] In this way, in a first stage, the driver may be initially incentivized to aid the training of the ad timing model both by indicating to the system when a suitable time is for displaying an ad, and supplying explicit feedback to the display of ads after they are displayed. Over time, the ad timing model may leam to predict when to display an ad and / or to terminate the display of an ad based on sensor data, and not to rely on driver feedback, whereby the driver incentives may be discontinued. As a result, the ad timing model may be trained using supervised learning without increasing a risk of the driver.

[0168] Additionally, by reformatting the input data to reduce a dimensionality and a size of the input data with respect to the raw environmental sensor data, DMS data and driver profile data, the input data may be more efficiently stored and transmitted to the ad timing training system, and the ad timing model may be trained on the reformatted input data in a faster and more efficient manner than on the raw data. An additional advantage of the disclosed system is that the ad timing model may be leveraged to determine a timing for displaying other types of data to the driver. That is, an output of the ad timing model that indicates suitable conditions for displaying an ad, may also indicate suitable conditions for displaying a notification of a different type to the driver. Thus, the ad delivery system may be used in conjunction with other vehicle systems to increase an efficiency of driver-vehicle communication overall.

[0169] The technical effect of reformatting the environmental sensor data and DMS data to reduce its dimensionality and size and collecting explicit feedback from the driver regarding a timing of a display of ads by an ad delivery' system of a vehicle, is that an amount of data used to train an ad timing model of the ad delivery system may be reduced, increasing anefficiency of training the ad timing model and efficiency of the use of the trained ad timing model at the vehicle.

[0170] The disclosure also provides support for a machine-implemented method, comprising: collecting environmental sensor data of a vehicle during operation of the vehicle by a driver, collecting driver status data from a driver monitoring system (DMS) of the vehicle, reformatting the collected environmental sensor data and the driver status data to reduce a size and / or dimensionality of the collected environmental sensor data and the driver status data, predicting a suitability of conditions for displaying an advertisement to the driver, based on the reformatted environmental sensor data and driver status data, using a machine learning (ML) model, and in response to the predicted suitability exceeding a threshold suitability, displaying the advertisement to the driver. In a first example of the method, the driver status data includes at least one of: identification information of the driver, output of a dashboard camera of the vehicle, seat occupancy data of the vehicle, interior illumination data of the vehicle, biometric data of the driver, eye gaze data of the driver, and a movement of a head or body of the driver. In a second example of the method, optionally including the first example, reformatting the collected driver status data to reduce the size and / or dimensionality of the driver status data further comprises at least one of: generating a first cognitive score indicating a degree of drowsiness of the driver, based on the driver status data, generating a second cognitive score indicating a degree of stress or anxiety of the driver, based on the driver status data, and generating a third cognitive score indicating a degree of distraction of the driver based on the driver status data. In a third example of the method, optionally including one or both of the first and second examples, the environmental sensor data includes: data from a navigational system of the vehicle, data from an in-vehicle infotainment (IVI) system of the vehicle, and sensor data from one or more bus systems of the vehicle. In a fourth example of the method, optionally including one or more or each of the first through third examples, reformatting the collected environmental sensor data to reduce the size and / or dimensionality of the environmental sensor data further comprises generating a traffic status score indicating an amount of traffic in which the vehicle is being operated, based on collected environmental sensor data of the vehicle including: external sensor / camera data, brake pedal and acceleration pedal data, data of an onboard navigation system of the vehicle, proximity sensor data, engine stop / start status data, and V2X data received from connected vehicles in a vicinity of the vehicle. In a fifth example of the method, optionally including one or more or each of the first through fourth examples, reformatting the collected environmental sensor data to reduce the size and / or dimensionality of the environmental sensor data further comprises generating a weather status score, based oncollected environmental sensor data of the vehicle including: external sensor / camera data, windshield wiper status data, roof status data, light status data, and data of an on-board navigation system of the vehicle. In a sixth example of the method, optionally including one or more or each of the first through fifth examples, displaying the advertisement to the driver further comprises at least one of: displaying the advertisement to the driver on a dashboard display of the vehicle, and projecting the advertisement on a portion of a windshield of the vehicle. In a seventh example of the method, optionally including one or more or each of the first through sixth examples, the reformatted environmental sensor data and the reformatted driver status data are inputs into the ML model. In a eighth example of the method, optionally including one or more or each of the first through seventh examples, the ML model is a local copy of a master advertisement timing model trained at a cloud-based ad timing training system and transmitted to the vehicle. In a ninth example of the method, optionally including one or more or each of the first through eighth examples, the master advertisement timing model is trained on training data collected from a plurality of drivers at a plurality of vehicles, the training data including a plurality of training pairs, each training pair including reformatted environmental sensor data and reformatted driver status data collected from a vehicle of the plurality of vehicles at a moment in time as input data, and an ad success score indicating an estimated suitability of the moment in time for displaying an ad to the driver as ground truth data. In a tenth example of the method, optionally including one or more or each of the first through ninth examples, the estimated suitability of the moment in time for displaying the ad to the driver is estimated by an ad success model based at least partially on at least one of: eye gaze data of the driver acquired by an in-cabin camera of the vehicle, and audio data acquired by an in-cabin microphone of the vehicle. In a eleventh example of the method, optionally including one or more or each of the first through tenth examples, the estimated suitability’ of the moment in time for displaying the ad to the driver is estimated by the ad success model based at least partially on feedback regarding the timing of the ad generated by the driver by one of: selecting a first control on a dashboard of the vehicle, selecting a second control on a steering wheel of the vehicle, selecting a virtual control in a touchscreen display of the vehicle, issuing a voice command, and making a gesture that is recorded by a camera of the vehicle. In a twelfth example of the method, optionally including one or more or each of the first through eleventh examples, the method further comprises: at a first time, in response to the predicted suitability exceeding the threshold suitability, displaying the advertisement to the driver, and at a second time after the first time, in response to the predicted suitability not exceeding the threshold suitability, terminating the displaying of the advertisement to the driver. In athirteenth example of the method, optionally including one or more or each of the first through twelfth examples: in a first condition, the predicted suitability exceeds the threshold suitability, and the advertisement is displayed to the driver, and in a second condition, the predicted suitability does not exceed the threshold suitability, and the advertisement is not displayed to the driver.

[0171] The disclosure also provides support for a system of a vehicle, comprising: one or more processors having executable instructions stored in a non-transitory memory that, when executed, cause the one or more processors to: collect environmental sensor data of the vehicle during operation of the vehicle by a driver, collect driver status data from a driver monitoring system (DMS) of the vehicle, reformat the collected environmental sensor data and the driver status data to reduce a size and / or dimensionality of the collected environmental sensor data and the driver status data, predict a suitability of conditions for displaying an advertisement to the driver, based on the reformatted environmental sensor data and the reformatted driver status data, using a machine learning (ML) model, in response to the predicted suitability exceeding a threshold suitability, display the advertisement to the driver on a dashboard display of the vehicle, and in response to the predicted suitability not exceeding the threshold suitability, not display the advertisement to the driver. In a first example of the system, the reformatted environmental sensor data and driver status data includes a plurality of numerical scores generated from different groupings of the collected environmental sensor data and collected driver status data, the numerical scores including at least: a traffic status score generated from collected environmental sensor data including external sensor / camera data, brake pedal and acceleration pedal data, data of an on-board navigation system of the vehicle, proximity’ sensor data, engine stop / start status data, and V2X data received from connected vehicles in a vicinity of the vehicle, and a cognitive score estimating a psychological or emotional state of the driver generated from collected driver status data including: identification information of the driver, output of a dashboard camera of the vehicle, seat occupancy data of the vehicle, interior illumination data of the vehicle, biometric data of the driver, ey e gaze data of the driver, and a movement of a head or body of the driver. In a second example of the system, optionally including the first example, further instructions are stored in the non-transitory memory that when executed, cause the one or more processors to: collect feedback from the driver regarding a timing of an advertisement, the feedback generated by the driver by one of: selecting a control on a dashboard of the vehicle, selecting a control on a steering wheel of the vehicle, selecting a virtual control in a touchscreen display of the vehicle, issuing a voice command, and making a gesture that is recorded by a camera of the vehicle, generate a training pair for training theML model, the training pair including the reformatted environmental sensor data and driver status data as input data, and an ad success score generated from the collected feedback as ground truth data, and transmit the training pair to an ad timing training system for training a master copy of the ML model. In a third example of the system, optionally including one or both of the first and second examples, further instructions are stored in the non-transitory memory that when executed, cause the one or more processors to: notify the driver that the advertisement is available for displaying on the dashboard display, via an audio notification, a visual element displayed on the dashboard display, an indicator light, or via haptic feedback generated at a seat or seat belt of the driver or at a steering wheel of the vehicle, display the advertisement on the dashboard display in response to the driver selecting to view the advertisement, and generate a training pair for training the ML model, the training pair including the reformatted environmental sensor data and driver status data as input data, and an ad success score indicating the suitability of the conditions for displaying the advertisement as ground truth data.

[0172] The disclosure also provides support for a system for training a machine learning (ML) model to predict a suitability of conditions for displaying an advertisement to a driver of a vehicle on a dashboard display of the vehicle during operation of the vehicle, the system comprising: a cloud-based server including a processor having executable instructions stored in a non-transitory memory that, when executed, cause the processor to: receive training data comprising a plurality of training pairs from the driver, each training pair data including, as input data, environmental sensor information of the vehicle and driver status data of the driver collected at a point in time, and as ground truth data, a score indicating the suitability7of conditions for displaying the advertisement to the driver at the point in time, train the ML model on the training pairs, and transmit a copy of the trained ML model to the vehicle, wherein the environmental sensor information of the vehicle and the driver status data of the driver collected at the point in time is reformatted to reduce a size and / or dimensionality’ of the environmental sensor information of the vehicle and the driver status data of the driver. In a first example of the system, the score indicating the suitability of conditions for displaying the advertisement to the driver at the point in time is based on one of: explicit feedback collected from the driver regarding the timing of the display of the advertisement, and the driver selecting the point in time for displaying the advertisement.

[0173] The description of embodiments has been presented for purposes of illustration and description. Suitable modifications and variations to the embodiments may be performed in light of the above description or may be acquired from practicing the methods. For example,unless otherwise noted, one or more of the described methods may be performed by a suitable device and / or combination of devices, such as the embodiments described above with respect to FIGS. 1-7. The methods may be performed by executing stored instructions with one or more logic devices (e.g., processors) in combination with one or more hardware elements, such as storage devices, memory', hardware network interfaces / antennas, switches, clock circuits, and so on. The described methods and associated actions may also be performed in various orders in addition to the order described in this application, in parallel, and / or simultaneously. The described systems are exemplary in nature, and may include additional elements and / or omit elements. The subject matter of the present disclosure includes all novel and non-obvious combinations and sub-combinations of the various systems and configurations, and other features, functions, and / or properties disclosed.

[0174] As used in this application, an element or step recited in the singular and proceeded with the word “a” or ‘"an” should be understood as not excluding plural of said elements or steps, unless such exclusion is stated. Furthermore, references to "one embodiment” or “one example” of the present disclosure are not intended to be interpreted as excluding the existence of additional embodiments that also incorporate the recited features. The terms “first,” “second,” “third,” and so on are used merely as labels, and are not intended to impose numerical requirements or a particular positional order on their objects. The following claims particularly point out subject matter from the above disclosure that is regarded as novel and non-obvious.

Claims

CLAIMS:

1. A machine-implemented method, comprising: collecting environmental sensor data of a vehicle during operation of the vehicle by a driver; collecting driver status data from a driver monitoring system (DMS) of the vehicle; reformatting the collected environmental sensor data and the driver status data to reduce a size and / or dimensionality of the collected environmental sensor data and the driver status data; predicting a suitability of conditions for displaying an advertisement to the driver, based on the reformatted environmental sensor data and driver status data, using a machine learning (ML) model; and in response to the predicted suitability exceeding a threshold suitability, displaying the advertisement to the driver.

2. The machine-implemented method of claim 1, wherein the driver status data includes at least one of: identification information of the driver; output of a dashboard camera of the vehicle; seat occupancy data of the vehicle; interior illumination data of the vehicle; biometric data of the driver; eye gaze data of the driver: and a movement of a head or body of the driver.

3. The machine-implemented method of claim 1, wherein reformatting the collected driver status data to reduce the size and / or dimensionality of the driver status data further comprises at least one of: generating a first cognitive score indicating a degree of drowsiness of the driver, based on the driver status data; generating a second cognitive score indicating a degree of stress or anxiety of the driver, based on the driver status data; and generating a third cognitive score indicating a degree of distraction of the driver based on the driver status data.

4. The machine-implemented method of claim 1, wherein the environmental sensor data includes: data from a navigational system of the vehicle; data from an in-vehicle infotainment (IVI) system of the vehicle; and sensor data from one or more bus systems of the vehicle.

5. The machine-implemented method of claim 1, wherein part of reformatting the collected environmental sensor data to reduce the size and / or dimensionality7of the environmental sensor data further comprises generating a traffic status score indicating an amount of traffic in which the vehicle is being operated, based on collected environmental sensor data of the vehicle including: external sensor / camera data; brake pedal and acceleration pedal data; data of an on-board navigation system of the vehicle; proximity- sensor data; engine stop / start status data; andV2X data received from connected vehicles in a vicinity of the vehicle.

6. The machine-implemented method of claim 1. wherein part of reformatting the collected environmental sensor data to reduce the size and / or dimensionality of the environmental sensor data further comprises generating a weather status score, based on collected environmental sensor data of the vehicle including: external sensor / camera data; windshield wiper status data; roof status data; light status data; and data of an on-board navigation system of the vehicle.

7. The machine-implemented method of claim 1, wherein displaying the advertisement to the driver further comprises at least one of: displaying the advertisement to the driver on a dashboard display of the vehicle; and projecting the advertisement on a portion of a windshield of the vehicle.

8. The machine-implemented method of claim 1, wherein the reformatted environmental sensor data and the reformatted driver status data are inputs into the ML model.

9. The machine-implemented method of claim 1 , wherein the ML model is a local copy of a master advertisement timing model trained at a cloud-based ad timing training system and transmitted to the vehicle.

10. The machine-implemented method of claim 9, wherein the master advertisement timing model is trained on training data collected from a plurality' of drivers at a plurality' of vehicles, the training data including a plurality of training pairs, each training pair including reformatted environmental sensor data and reformatted driver status data collected from a vehicle of the plurality of vehicles at a moment in time as input data, and an ad success score indicating an estimated suitability of the moment in time for displaying an ad to the driver as ground truth data.

11. The machine-implemented method of claim 10, wherein the estimated suitability of the moment in time for displaying the ad to the driver is estimated by an ad success model based at least partially on at least one of: eye gaze data of the driver acquired by an in-cabin camera of the vehicle; and audio data acquired by an in-cabin microphone of the vehicle.

12. The machine-implemented method of claim 11 , wherein the estimated suitability of the moment in time for displaying the ad to the driver is estimated by the ad success model based at least partially on feedback regarding the timing of the ad generated by the driver by one of: selecting a first control on a dashboard of the vehicle; selecting a second control on a steering wheel of the vehicle; selecting a virtual control in a touchscreen display of the vehicle; issuing a voice command; and making a gesture that is recorded by a camera or sensor of the vehicle.

13. The machine-implemented method of claim 1, further comprising: at a first time, in response to the predicted suitability exceeding the threshold suitability', displaying the advertisement to the driver; andat a second time after the first time, in response to the predicted suitability' not exceeding the threshold suitability, terminating the displaying of the advertisement to the driver.

14. The machine-implemented method of claim 1, wherein: in a first condition, the predicted suitability' exceeds the threshold suitability', and the advertisement is displayed to the driver; and in a second condition, the predicted suitability does not exceed the threshold suitability, and the advertisement is not displayed to the driver.

15. A system of a vehicle, comprising: one or more processors having executable instructions stored in a non-transitory memory that, when executed, cause the one or more processors to: collect environmental sensor data of the vehicle during operation of the vehicle by a driver; collect driver status data from a driver monitoring system (DMS) of the vehicle; reformat the collected environmental sensor data and the driver status data to reduce a size and / or dimensionality of the collected environmental sensor data and the driver status data; predict a suitability of conditions for displaying an advertisement to the driver, based on the reformatted environmental sensor data and the reformatted driver status data, using a machine learning (ML) model; in response to the predicted suitability exceeding a threshold suitability’, display the advertisement to the driver on a dashboard display of the vehicle; and in response to the predicted suitability not exceeding the threshold suitability', not display the advertisement to the driver.

16. The system of claim 15, wherein the reformatted environmental sensor data and driver status data includes a plurality' of numerical scores generated from different groupings of the collected environmental sensor data and collected driver status data, the numerical scores including at least: a traffic status score generated from collected environmental sensor data including external sensor / camera data, brake pedal and acceleration pedal data, data of an on-board navigation sy stem of the vehicle, proximity sensor data, engine stop / start status data, and V2X data received from connected vehicles in a vicinity’ of the vehicle; anda cognitive score estimating a psychological or emotional state of the driver generated from collected driver status data including: identification information of the driver; output of a dashboard camera of the vehicle; seat occupancy data of the vehicle; interior illumination data of the vehicle; biometric data of the driver; eye gaze data of the driver; and a movement of a head or body of the driver.

17. The system of claim 15. wherein further instructions are stored in the non-transitory memory that when executed, cause the one or more processors to: collect feedback from the driver regarding a timing of an advertisement, the feedback generated by the driver by one of: selecting a control on a dashboard of the vehicle; selecting a control on a steering wheel of the vehicle; selecting a virtual control in a touchscreen display of the vehicle; issuing a voice command; and making a gesture that is recorded by a camera of the vehicle; generate a training pair for training the ML model, the training pair including the reformatted environmental sensor data and driver status data as input data, and an ad success score generated from the collected feedback as ground truth data; and transmit the training pair to an ad timing training system for training a master copy of the ML model.

18. The system of claim 17, wherein further instructions are stored in the non-transitory memory that when executed, cause the one or more processors to: notify the driver that the advertisement is available for displaying on the dashboard display, via an audio notification, a visual element displayed on the dashboard display, an indicator light, or via haptic feedback generated at a seat or seat belt of the driver or at a steering wheel of the vehicle; display the advertisement on the dashboard display in response to the driver selecting to view the advertisement; andgenerate a training pair for training the ML model, the training pair including the reformatted environmental sensor data and driver status data as input data, and an ad success score indicating the suitability of the conditions for displaying the advertisement as ground truth data.

19. A system for training a machine learning (ML) model to predict a suitability of conditions for displaying an advertisement to a driver of a vehicle on a dashboard display of the vehicle during operation of the vehicle, the system comprising: a cloud-based server including a processor having executable instructions stored in a non-transitory memory that, when executed, cause the processor to: receive training data comprising a plurality of training pairs from the driver, each training pair data including, as input data, environmental sensor information of the vehicle and driver status data of the driver collected at a point in time, and as ground truth data, a score indicating the suitability of conditions for displaying the advertisement to the driver at the point in time; tram the ML model on the training pairs; and transmit a copy of the trained ML model to the vehicle; wherein the environmental sensor information of the vehicle and the driver status data of the driver collected at the point in time is reformatted to reduce a size and / or dimensionality of the environmental sensor information of the vehicle and the driver status data of the driver.

20. The system of claim 19. where the score indicating the suitability' of conditions for displaying the advertisement to the driver at the point in time is based on one of: explicit feedback collected from the driver regarding a timing of the display of the advertisement; and the driver selecting the point in time for displaying the advertisement.

Images