Over-the-air renewal system for vehicle

Abstract

The invention relates to an over-the-air update system for a vehicle. An example vehicle air (OTA) update system includes a vehicle control module configured to: obtain vehicle sensor data from a plurality of vehicle sensors and voice data from a vehicle microphone; obtaining driving history data indicating a driving behavior associated with the vehicle over a specified historical time period; synchronizing the vehicle sensor data, the voice data, and the driving history data using the plurality of time frames to produce a synchronized data set; tokenizing the synchronous data set to formalize the embedded vector; generating a set of OTA feature priorities by feeding the embedded vector to a trained multi-modal machine learning model; determining an OTA update execution schedule according to the OTA feature driver preference output, wherein the OTA execution schedule includes a first OTA update feature having a higher priority than a second OTA update feature; and control the reception of OTA update data via the vehicle antenna.

Description

Background Technology

[0001] The information provided in this section is for the purpose of generally presenting the context of this disclosure. The work of the currently named inventor (to the extent described in this section) and aspects of the description that might not have qualified as prior art at the time of filing are neither expressly nor implicitly acknowledged as prior art to this disclosure.

[0002] This disclosure generally relates to an over-the-air (OTA) update system for vehicles, including prioritizing OTA updates based on sensed vehicle data and driving history.

[0003] Vehicles include numerous control features for various vehicle systems, such as autonomous driving and voice commands. Vehicles can receive over-the-air updates to upgrade different control systems. Summary of the Invention

[0004] An exemplary vehicle over-the-air (OTA) update system includes: a plurality of vehicle sensors, each configured to detect vehicle sensor data indicating one or more vehicle operating parameters of the vehicle; a vehicle microphone configured to capture audio from the interior of the vehicle; an antenna configured to wirelessly receive vehicle OTA update data; and a vehicle control module configured to: acquire vehicle sensor data from the plurality of vehicle sensors and voice data from the vehicle microphone; acquire driving history data indicating driving behavior associated with the vehicle over a specified historical time period; synchronize the vehicle sensor data, the voice data, and the driving history data using multiple time frames to generate a synchronized dataset; tokenize the synchronized dataset to formalize an embedding vector for input to a trained multimodal machine learning model configured to generate an OTA feature driver preference output; generate an OTA feature priority set by feeding the embedding vector to the trained multimodal machine learning model; determine an OTA update execution schedule based on the OTA feature driver preference output, wherein the OTA execution schedule includes a first OTA update feature having a higher priority than a second OTA update feature; and control the reception of OTA update data via the antenna based on the priority of the OTA update features in the OTA update execution schedule.

[0005] In some examples, the trained multimodal machine learning model is trained to receive input data from sources with multiple modalities and to produce outputs indicating the likelihood of driver preferences for multiple OTA update features.

[0006] In some examples, the trained multimodal machine learning model includes at least one of a transformer model, a neural network, a large language model (LLM), or a memory-aware regression model.

[0007] In some examples, the first OTA update feature includes at least one of voice command control features, adaptive cruise control features, advanced driver assistance system features, autopilot features, or software-defined vehicle features.

[0008] In some examples, controlling the reception of OTA update data includes receiving updates to advanced driver assistance system features, and the vehicle control module is configured to automatically control the acceleration, braking, and steering of the vehicle based on the updates to the advanced driver assistance system features.

[0009] In some examples, obtaining vehicle sensor data includes: formatting the vehicle sensor data in the format of {I, L, D} to include a sensor dataset (I) with semantic annotation labels (L) and depth information (D); and formatting the speech data in the format of {S, W} using a large language model, where S is the original sound signal wave and W includes the recognized language words.

[0010] In some examples, synchronizing the vehicle sensor data, the voice data, and the driving history data includes: obtaining Global Positioning System (GPS) time frame data associated with the vehicle sensor data, the voice data, and the driving history data; and performing synchronization of the vehicle sensor data, the voice data, and the driving history data based on the GPS time frame data.

[0011] In some examples, synchronizing the vehicle sensor data, the voice data, and the driving history data includes interpolating time frames associated with the vehicle sensor data to create matching time frames associated with each element of the vehicle sensor data.

[0012] In some examples, controlling the reception of OTA update data includes: receiving multiple data packets wirelessly from a wireless transmitter via the vehicle's antenna; combining the multiple data packets to define an OTA update block; and storing the OTA update block in the memory of the vehicle control module.

[0013] In some examples, controlling the reception of OTA update data includes: receiving a message indicating that a first OTA upgrade corresponding to the first OTA update feature is available and a second OTA upgrade corresponding to the second OTA update feature is available; receiving the first OTA upgrade via wireless transmission to the vehicle's antenna; and specifying a future time period for delaying the reception of the second OTA upgrade feature. In some examples, the specified future time period is at least one week.

[0014] An exemplary method for controlling over-the-air (OTA) updates for a vehicle includes: receiving vehicle sensor data from multiple vehicle sensors and voice data from a vehicle microphone configured to capture audio from inside the vehicle, wherein the vehicle sensor data indicates one or more vehicle operating parameters of the vehicle; obtaining driving history data indicating driving behavior associated with the vehicle over a specified historical time period; synchronizing the vehicle sensor data, the voice data, and the driving history data using multiple time frames to generate a synchronized dataset; tokenizing the synchronized dataset to formalize embedding vectors for input to a trained multimodal machine learning model configured to generate OTA feature driver preference outputs; generating an OTA feature priority set by feeding the embedding vectors to the trained multimodal machine learning model; determining an OTA update execution schedule based on the OTA feature driver preference outputs, wherein the OTA execution schedule includes a first OTA update feature having a higher priority than a second OTA update feature; and controlling the reception of OTA update data via the vehicle's wireless antenna based on the priority of the OTA update features in the OTA update execution schedule.

[0015] In some examples, the trained multimodal machine learning model is trained to receive input data from sources with multiple modalities and to produce outputs indicating the likelihood of driver preferences for multiple OTA update features.

[0016] In some examples, the trained multimodal machine learning model includes at least one of a transformer model, a neural network, a large language model (LLM), or a memory-aware regression model.

[0017] In some examples, the first OTA update feature includes at least one of voice command control features, adaptive cruise control features, advanced driver assistance system features, autopilot features, or software-defined vehicle features.

[0018] In some examples, controlling the reception of OTA update data includes receiving updates to advanced driver assistance system features, and the method includes automatically controlling the acceleration, braking, and steering of the vehicle based on the updates to the advanced driver assistance system features.

[0019] In some examples, obtaining vehicle sensor data includes: formatting the vehicle sensor data in the format of {I, L, D} to include a sensor dataset (I) with semantic annotation labels (L) and depth information (D); and formatting the speech data in the format of {S, W} using a large language model, where S is the original sound signal wave and W includes the recognized language words.

[0020] In some examples, synchronizing the vehicle sensor data, the voice data, and the driving history data includes: obtaining Global Positioning System (GPS) time frame data associated with the vehicle sensor data, the voice data, and the driving history data; and performing synchronization of the vehicle sensor data, the voice data, and the driving history data based on the GPS time frame data.

[0021] In some examples, synchronizing the vehicle sensor data, the voice data, and the driving history data includes interpolating time frames associated with the vehicle sensor data to create matching time frames associated with each element of the vehicle sensor data.

[0022] In some examples, controlling the reception of OTA update data includes: receiving multiple data packets wirelessly from a wireless transmitter via the vehicle's wireless antenna; combining the multiple data packets to define an OTA update block; and storing the OTA update block in the memory of the vehicle control module.

[0023] Further areas of applicability of this disclosure will become clear from the detailed description, claims, and drawings. The detailed description and specific examples are intended for illustrative purposes only and are not intended to limit the scope of this disclosure. Attached Figure Description

[0024] This disclosure will become more fully understood through detailed description and accompanying drawings.

[0025] Figure 1 This is an illustration of an exemplary vehicle including an antenna configured to receive over-the-air wireless updates.

[0026] Figure 2 This is a block diagram of an exemplary system for prioritizing vehicle OTA updates based on sensed vehicle data and driving history.

[0027] Figure 3 This is a flowchart depicting an exemplary process for prioritizing vehicle OTA updates based on sensed vehicle data and driving history.

[0028] Figure 4 It is a description used in Figure 3 A flowchart illustrating an exemplary process of synchronizing sensed vehicle data and driving history during an exemplary process.

[0029] Figure 5 It is a description used in Figure 3 A flowchart of an exemplary process for extracting driver preferences associated with OTA vehicle updates during an exemplary process.

[0030] Figure 6 It is a description used in Figure 3 A flowchart illustrating an exemplary process of performing OTA vehicle updates based on determined driver priorities during an exemplary process.

[0031] Figure 7A and 7B This is a graphical representation of an exemplary recurrent neural network used to predict driver OTA preferences based on vehicle sensor data and driving history.

[0032] Figure 8 This is a graphical representation of a layer in an exemplary Long Short-Term Memory (LSTM) machine learning model.

[0033] Figure 9 This is a flowchart illustrating an exemplary process for training a machine learning model.

[0034] In the accompanying drawings, reference numerals may be reused to identify similar and / or identical elements. Detailed Implementation

[0035] In some exemplary embodiments, a custom over-the-air (OTA) system for vehicle updates may utilize one or more multimodal machine learning models to improve or optimize OTA vehicle update strategies, such as OTA feature update frequency, priority of emergency features, user preferences, critical timing, and feature context. For example, the system may be configured to tokenize multimodal data, including synchronized driving data and vehicle sensor data, and formalize a type consistency matrix data source.

[0036] This system can be configured to extract learned features and driving preferences from data such as driving history data and vehicle sensor data. The mapping between learned driving preferences and OTA features can guide improvements or optimizations to OTA vehicle update strategies, which can help reduce bandwidth and costs (e.g., the bandwidth and cost required to transmit OTA data to the vehicle's antennas for delivery and download), and increase or maximize the efficiency of OTA-related software and firmware upgrades based on the actual needs of vehicle users (e.g., drivers and passengers).

[0037] Some OTA strategies are based on mandatory upgrade requirements to address firmware and software issues. OTA procedures can be inflexible and time-consuming, requiring high data exchange, which may not fully meet the actual needs of users or the specific OTA feature updates they require. In some examples presented in this article, OTA vehicle update procedures may be based on various factors, including driver preferences, urgency level, frequency, and relevance to OTA feature content.

[0038] For example, driving behavior and OTA feature usage factors, along with corresponding metrics, can be automatically extracted and learned from multimodal data from daily driving (such as vehicle sensor data, driving data, and habit and event estimations). The consistency of data representations from different data sources can be maintained by the system to learn a unified model to optimize latent variables and mappings to various OTA strategy objectives. This facilitates the extraction and prioritization of OTA update features, and OTA vehicle update strategies can be rated without introducing redundancy.

[0039] This improves efficiency in performing OTA vehicle updates and reduces or avoids the need for a full OTA vehicle update that runs all available update features simultaneously (which could require more bandwidth and time compared to performing only a subset of OTA updates prioritized based on user preferences and usage history). For example, some systems can reduce bandwidth and optimize OTA features based on the actual requirements or expectations of vehicle users (e.g., drivers and passengers), where customized OTA vehicle update features fully meet user requirements regarding OTA update strategies and content. Utilizing customized features can significantly reduce the cost of OTA updates and maximize the efficiency of OTA-related software and firmware.

[0040] Now refer to Figure 1 The vehicle 10 includes front wheels 12 and rear wheels 13. Figure 1 In this configuration, drive unit 14 selectively outputs torque to the front wheels 12 and / or the rear wheels 13 via drive lines 16 and 18, respectively. Vehicle 10 may include different types of drive units. For example, the vehicle may be an electric vehicle (such as a battery electric vehicle (BEV)), a hybrid vehicle, a fuel cell vehicle, a vehicle including an internal combustion engine (ICE), or other types of vehicles.

[0041] Some examples of drive unit 14 may include any suitable electric motor, power inverter, and motor controller, the motor controller being configured to control power switches within the power inverter to adjust motor speed and torque during propulsion and / or regeneration. During propulsion or regeneration, the battery system supplies power to the motor of drive unit 14 via the power inverter or receives power from the motor of drive unit 14.

[0042] Despite Figure 1 The vehicle 10 includes a drive unit 14, but the vehicle 10 may have other configurations. For example, two separate drive units may drive the front wheels 12 and the rear wheels 13, or one or more individual drive units may drive individual wheels, etc. As will be understood, other vehicle configurations and / or drive units may be used.

[0043] The vehicle control module 20 can be configured to control the operation of one or more vehicle components, such as drive unit 14 (e.g., by commanding the torque setting of the electric motor of drive unit 14). The vehicle control module 20 can receive inputs for controlling vehicle components, such as signals received from the steering wheel, accelerator pedal, brake pedal, etc. The vehicle control module 20 can monitor vehicle telematics for safety purposes, such as vehicle speed, vehicle location, vehicle braking, and acceleration.

[0044] The vehicle control module 20 can receive signals from any suitable components for monitoring one or more aspects of the vehicle, including one or more vehicle sensors (such as cameras, microphones, pressure sensors, steering wheel position sensors, brake sensors, location sensors (such as GPS antennas), wheel height and / or position sensors, accelerometers, etc.). Some sensors may be configured to monitor the vehicle's current motion, vehicle acceleration, vehicle braking, vehicle's current steering direction, the current height and / or position of one or more wheels, etc. In some examples, the vehicle microphone 22 is configured to capture audio from inside the vehicle 10, such as voice commands from the driver or passengers (e.g., for controlling autonomous driving features, hands-free navigation, or entertainment features, etc.).

[0045] The vehicle control module 20 can communicate with another device via a wireless communication interface, which may include one or more wireless antennas for transmitting and / or receiving wireless communication signals. For example, the wireless communication interface can communicate via any suitable wireless communication protocol, including but not limited to vehicle-to-anything (V2X) communication, Wi-Fi communication, wireless local area network (WAN) communication, cellular communication, personal area network (PAN) communication, short-range wireless communication (e.g., Bluetooth), etc. The wireless communication interface can communicate with a remote computing device via one or more wireless and / or wired networks. Regarding vehicle-to-vehicle (V2X) communication, vehicle 10 may include one or more V2X transceivers (e.g., V2X signal transmitting and / or receiving antennas).

[0046] like Figure 2 As shown, the vehicle control module 20 may include an over-the-air (OTA) update antenna 28, which can be configured to receive OTA vehicle update data from an OTA server, OTA update provider, etc. For example, radio towers, cellular towers, satellites, WiFi routers, etc., can be configured to wirelessly transmit data to the OTA update antenna 28 to enable OTA vehicle feature upgrades. The OTA update antenna 28 may include any suitable receiver, transceiver, antenna, antenna array, etc., configured to receive OTA vehicle update data.

[0047] The received OTA vehicle update data can be stored in the memory of vehicle 10 (such as the memory of vehicle control module 20). The received OTA vehicle update data can be used to modify vehicle software, firmware, etc. For example, OTA vehicle update data may include upgrades to the autonomous driving control system, wherein vehicle control module 20 is configured to implement OTA vehicle updates to automatically control the acceleration of vehicle 10 (e.g., via the accelerator or by controlling the motor of drive unit 14 to provide more power to the front wheels 12 and rear wheels 13), control the braking of vehicle 10 (e.g., via brakes applied to the front wheels 12 and rear wheels 13 or via engine braking in the motor of drive unit 14), control the automatic steering of vehicle 10 (e.g., by rotating the steering mechanism or directly changing the orientation of the front wheels 12), etc.

[0048] The vehicle control module 20 may store driving history data 26, such as in its memory or in a database. Driving history data 26 may include data obtained via the vehicle's CAN bus, the use of autonomous driving features, etc.

[0049] Figure 2 This is a block diagram of an exemplary system 100 for prioritizing vehicle OTA updates based on sensed vehicle data and driving history. (See diagram for example.) Figure 1As shown, sensor data 124 is acquired (e.g., via...). Figure 2 The vehicle sensor 24), and can be used with the acquired voice data 122 (e.g., via... Figure 2 The vehicle microphone 22) combination.

[0050] For example, through multimodal fusion 130, sensor data 124 and voice data 122 from the original data source can be combined. Multimodal fusion 130 can generate a token 134, which can be supplied to the feature extraction module 138, which maps vehicle sensor features to OTA vehicle update features.

[0051] For example, by classifying the actions in driving history data 126 based on the date of the action and the corresponding Advanced Driver Assistance System (ADAS) features, driving history data 126 can be parsed into event data 132. Event data 132 may include defined action relationships 136 corresponding to OTA vehicle update features and can be fed to feature extraction module 138.

[0052] Figure 3 This is a flowchart depicting an exemplary process for prioritizing vehicle OTA updates based on sensed vehicle data and driving history. This process can be, for example... Figure 1 The vehicle control module 20 executes this. At 304, the process begins by obtaining input data from the vehicle sensors and the vehicle microphone.

[0053] For example, the system can be configured to collect data on the vehicle's internal and external environment by utilizing built-in sensors without introducing any additional device costs. The output of the data collection component can be a sensor dataset (I) with semantically annotated labels (L) and depth information (D), in the format {I, L, D}.

[0054] The system can also collect data on the conversations between the driver or passenger and the voice assistant system. Based on a large language model (LLM) speech model, each user's audio track can be tagged with a data format {S, W}, where S is the raw sound signal wave (e.g., frame-by-frame), and W includes the recognized language words.

[0055] In 308, the vehicle control module is configured to acquire driving history data based on past vehicle movements. For example, the system can be configured to collect driving history data {H}, which covers the driver's driving behavior. Some driving history data inputs can be retrieved via CAN data, which continuously monitors the status of actuators in the vehicle (e.g., acceleration, emergency braking, steering wheel, etc.).

[0056] In addition, the system can collect ADAS / AV related operations (e.g., activating ACC mode, autopilot mode, HMI attribute settings, control operation transmission, configuration, etc.). This information may include the frequency of events, and the corresponding OTA feature status can also be recorded in the dataset (e.g., {H|ADAS / AV, frequency, OTA feature}), which may be important for mapping the driver's intentions to autonomous features.

[0057] In section 312, the vehicle control module is configured to synchronize data from vehicle sensors and driving history. For example, the system can be configured to synchronize vehicle sensor data, microphone data, and driving history data by utilizing GPS time synchronization to align the time frames of each element of the data. For vehicle sensor data {I, L, D}, because the frame rates from different sensors are different, a predictive model can be used to interpolate between frames to retrieve all sensor data within the same time frame.

[0058] The system can be configured to fuse a list of objects (e.g., bird's-eye view (BEV) extracted from each sensor based on the estimated relative location for the master vehicle). This results in semantically annotated labels (L') of a fully fused system aligned on each frame from all sensors, and the system can be configured to map depth information (D') to formalize the dataset in the format {I', L', D' | I', L', D' ∈ fusion results of camera / LiDAR / radar with accurate temporal alignment}.

[0059] In some examples, multimodal data sources can be fused during the preprocessing step. Datasets {S, W} can be aligned with temporal frames of {I', L', D'}. The frame rate for sound and recognized words may be much lower than the frame rate of the sensed vehicle data. This allows for the determination of corresponding datasets {Vt|Vt∈(S,W)} that continuously span different frames of sensor data, paired with certain frames of sensor data {Pt|Pt∈(I', L', D')}.

[0060] After applying a transformer pattern to Vt and a vision transformer (ViT) to Pt, the system encodes the data into an embedding vector and finds similarities between Vt and Pt using an interior-point operation Mt = (Vt * Pt). The system can also be configured to align driving history with sensor data. For example, it can be configured to use Ht to represent the driver's behavior in each frame t of Mt from multimodal data. The system can output an embedding vector Mt = {Vt * Pt | Ht} fused from different data sources (e.g., sensor data and voice dialogue).

[0061] In section 316, the vehicle control module is configured to perform feature extraction and action mapping based on synchronized data. For example, OTA feature extraction can be used to determine the frequency level of OTA feature usage across different maneuvers. Based at least in part on the dataset {H}, the system can be configured to map driver habits to OTA features for each time step. Examples of OTA feature usage are provided in Table 1:

[0062]

[0063] Table 1

[0064] In the sampled data in Table 1, each row illustrates the actions driven by the driver at each time step (e.g., the sampling rate may be the same as the rate of dataset H, such as 100 Hz or higher). The values ​​of the "Date / Time" and "OTA Characteristics" columns can be retrieved directly from dataset H.

[0065] Based on dataset H, the column "Action" can be mapped using rule-based metrics. For example, an emergency braking action can be labeled when the vehicle's acceleration is greater than -3G to -6G. The column "Event" can be learned from data Mt using supervised learning procedures, such as training a neural network with hidden variables to estimate events based on surrounding information and driver behavior (e.g., traffic jam events and traffic light events can be learned using surrounding information from sensor data). If a lower confidence level exists in the "Event" probability output, the system can default to identifying the "Event" as "No Special Event".

[0066] In 320, the vehicle control module is configured to tokenize data for input into a trained machine learning model. For example, a tokenizer can be used to formalize the embedding vector Mt based on different frames into tokens recognizable by the multimodal training model. The multimodal training model can be configured to identify different OTA update strategies based on different driving behaviors. Because Mt may already be time-aligned and the embedding vectors can have consistent representations from different data sources, the unified multimodal LLM training model can be configured to learn the latent variable Z via a neural network Z = LLM(Mt) and generalize it into OTA strategies (e.g., frequency of OTA feature usage, urgency level, execution time based on certain OTA features, etc.).

[0067] In some examples, the policy can be represented at least partially as Policy ~ df(Z). Because LLM models are available, the policy terms of OTA features can be self-estimated based on driver behavior. For example, OTA policy features related to ACC may only have preferences regarding update frequency, while OTA features related to ADAS may have preferences regarding safety and enhancement updates.

[0068] In 324, the vehicle control module is configured to extract driving preferences based on processed data. For example, driving preferences can be estimated based on driving behavior history and related operations. In some examples, a memory-aware regression model can be used to predict driving preferences based on the frequency of use, startup time, and duration of use of OTA features. The output of this component can be a mapping between operational preferences for each OTA feature (e.g., ADAS – always on, ACC – off, voice assistant – on demand, etc.).

[0069] In 328, the vehicle control module is configured to use the model's output to prioritize over-the-air (OTA) features. For example, OTA features can be ranked based on the probability values ​​of mapped driver preferences and OTA policies, such as by using the equation Rank = Σweight*softmax(p(Policy)) mapped to driver preferences. OTA feature extraction and prioritization can be customized without introducing redundancy to guide the efficient execution of OTA features (avoiding the need to run all available OTA procedures at once).

[0070] In section 332, the vehicle control module is configured to set an OTA delivery schedule based on determined OTA feature priorities. For example, the OTA strategy can be improved or optimized based on ratings and attributes learned from previous steps. An exemplary OTA delivery schedule is illustrated in Table 2:

[0071]

[0072] Table 2

[0073] In 336, the vehicle control module is configured to perform OTA deliveries for the vehicle according to an OTA delivery schedule. Some exemplary embodiments can help reduce the bandwidth used for OTA feature updates and optimize OTA features based on the needs of the vehicle users. Customized OTA features can fully meet the needs of user update strategies and content. Utilizing customized features can significantly reduce costs and increase or maximize the efficiency of OTA-related software and firmware.

[0074] Figure 4 It is a description used in Figure 3 A flowchart illustrating an exemplary process for synchronizing sensed vehicle data and driving history during an exemplary process. This process can be, for example, [details of the process]. Figure 1 The vehicle control module 20 executes this process. At 404, the process begins by obtaining a sensor dataset (I) with semantic labels (L) and depth information (D). For example, the sensor dataset can be obtained from signals detected by vehicle sensors.

[0075] At 408, the vehicle control module is configured to use a large language model to tag dialogue data. The tagged data may include raw audio signal waves (S) and recognized language words (W). At 412, the vehicle control module is configured to retrieve CAN data (e.g., via the vehicle's CAN bus) and perform ADAS-related operations.

[0076] At 416, the control records driving history data in a dataset, which may include driving history events or frequency of use, OTA features associated with driving history events, etc. At 420, the vehicle control module is configured to use Global Positioning System (GPS) time synchronization to align the time frames of the data. At 424, the vehicle control module is configured to fuse the embedding vectors of each time frame from different data sources.

[0077] Figure 5 It is a description used in Figure 3 A flowchart illustrating an exemplary process for extracting driver preferences associated with OTA vehicle updates during an exemplary process. This process may be performed by, for example... Figure 1 The vehicle control module 20 executes this. At 504, the process begins by accessing a list of OTA features. At 508, the control then selects the first OTA feature from the list.

[0078] In step 512, the vehicle control module is configured to acquire driving behavior data related to the selected OTA feature. In step 516, the control module then determines the usage frequency of the selected OTA feature, the start time of events related to the selected OTA feature, the usage duration of the selected OTA feature, etc.

[0079] In step 520, the vehicle control module is configured to use, for example, a memory-perceptual regression module to predict driving preferences. In step 524, the control then generates a mapping between the driver's operating preferences and associated OTA features based on the model output.

[0080] At 528, the vehicle control module is configured to determine whether any additional OTA features remain on the list. If any additional OTA features remain on the list, control proceeds to 532 to select the next OTA feature from the list, and then returns to 512 to obtain driving behavior data associated with the next selected OTA feature.

[0081] Figure 6 It is a description used in Figure 3 A flowchart illustrating an exemplary process for performing OTA vehicle updates based on a determined driver priority during an exemplary process. This process may be performed by, for example... Figure 1 The vehicle control module 20 executes this. At 604, the process begins by obtaining the driver's OTA feature preference list.

[0082] In 608, the vehicle control module is configured to access a list of available OTA updates (e.g., a current list of OTA features stored on an update server that has a more recent version than the version currently installed on the vehicle and can be wirelessly transmitted to the vehicle for download and update).

[0083] At 612, the control selects the first available OTA update from the list. At 616, the control then obtains the last vehicle OTA update time of the selected available OTA feature update (e.g., the timestamp of the last received update of the selected OTA update feature delivered to the vehicle).

[0084] In section 620, the vehicle control module is configured to compare the last update time with the driver's OTA preference time period. For example, the driver's OTA preference time period may indicate that ADAS features should only be updated once a month. If at least one month has passed since the vehicle last received an ADAS feature OTA update, the vehicle control module may be configured to receive only the most recent OTA update related to the ADAS feature.

[0085] If an OTA update is indicated at 624 (e.g., because the last OTA update received for the selected feature was at least a specified interval prior to the determined OTA feature priority of the driver), control proceeds to 632 to receive the OTA update via wireless transmission to the vehicle's antenna.

[0086] In 636, the vehicle control module is configured to update the vehicle control system, which may include storing received OTA update data in the vehicle's memory, installing OTA update features, or upgrading the vehicle's control software or firmware based on OTA update data.

[0087] At 640, the control determines whether the OTA update features include any ADAS features. If the OTA update features include any ADAS features, the control proceeds to 644 to automatically control the vehicle's acceleration, braking, and steering based on the received OTA update features.

[0088] After storing or implementing an updated OTA feature, or after determining at 624 that an OTA update should not be executed due to the lower priority of the OTA feature, control proceeds to 628 to determine if any remaining OTA updates exist. If any remaining OTA updates exist, control at 648 selects the next available OTA update and returns to 616 to obtain the last vehicle OTA update time for the next available OTA update feature.

[0089] Figure 7A and 7BAn example of a recurrent neural network used to generate models (such as the model described above) using machine learning techniques is shown. Machine learning is a method for designing complex models and algorithms suitable for prediction (e.g., patient and provider matching prediction). Models generated using machine learning (such as the model described above) are capable of producing reliable and repeatable decisions and outcomes, and uncover hidden insights by learning from historical relationships and trends in the data.

[0090] As described above, the purpose of using a recurrent neural network-based model and training it with machine learning is to directly predict the dependent variable without converting the relationships between variables into mathematical form. The neural network model comprises a large number of virtual neurons operating in parallel and arranged in layers. The first layer is the input layer 703, which receives the raw input data 701. Each subsequent layer modifies the output from the previous layer and sends it to the next layer. The final layer is the output layer 707, which produces the system's output 709.

[0091] Figure 7A This illustrates a fully connected neural network where each neuron in a given layer is connected to every neuron in the next layer. In the input layer, each input node is associated with a numerical value, which can be any real number. In each layer, each connection leaving an input node has a weight associated with it, which can also be any real number (see [link to diagram]). Figure 7B In the input layer, the number of neurons equals the number of features (columns) in the dataset. The output layer can have multiple consecutive outputs.

[0092] The layer between input layer 703 and output layer 707 is hidden layer 705. The number of hidden layers can be one or more (for most applications, one hidden layer is likely sufficient). A neural network without hidden layers can represent linearly separable functions or decisions. A neural network with one hidden layer can perform continuous mappings from one finite space to another. A neural network with two hidden layers can approximate any smooth mapping to any accuracy.

[0093] The number of neurons can be optimized. At the start of training, the network configuration is more likely to have too many nodes. During training, some nodes that will not significantly affect network performance can be removed from the network. For example, nodes with near-zero weights after training can be removed (this process is called pruning). The number of neurons can cause underfitting (failing to adequately capture the signals in the dataset) or overfitting (insufficient information to train all neurons; the network performs well on the training dataset but poorly on the test dataset).

[0094] Various methods and criteria can be used to measure the performance of neural network models. For example, the root mean square error (RMSE) measures the average distance between observations and model predictions. The coefficient of determination (R²) measures the correlation (not accuracy) between observations and predictions. This method may be unreliable if the data has large variance. Other performance metrics include irreducible noise, model bias, and model variance. High model bias indicates that the model fails to capture the true relationship between predictions and outcomes. Model variance indicates whether the model is stable (small perturbations in the data can significantly alter the model fit). Neural networks can receive inputs, such as vectors, which can be used to generate models that can be used to predict a driver's vehicle OTA update priority based on vehicle sensor data and driving history.

[0095] Figure 8 The illustration shows an example of a Long Short-Term Memory (LSTM) neural network 802. This LSTM neural network 802 is used to generate models (such as the model described above) using machine learning techniques, but other exemplary embodiments may include other types of machine learning models (including transformer layers, other model topologies, etc.). The general exemplary LSTM neural network 802 can be used to implement machine learning models, and various implementations may use other types of machine learning networks (such as transformer layers, other model topologies or architectures, etc.). The LSTM neural network 802 includes an input layer 804, hidden layers 808, and an output layer 812. The input layer 804 includes inputs 804a, 804b…804n, which may correspond to input data 801a, 801a…801n. The hidden layer 808 includes neurons 808a, 808b…808n. The output layer 812 includes outputs 812a, 812b…812n.

[0096] Each neuron in hidden layer 808 receives input from input layer 804 and outputs a value to the corresponding output in output layer 812. For example, neuron 808a receives input from input 804a and outputs a value to output 812a. In addition to neuron 808a, each neuron also receives the output of the previous neuron as input. For example, neuron 808b receives input from input 804b and output 812a. In this way, the output of each neuron is fed forward to the next neuron in hidden layer 808. The last output 812n in output layer 812 outputs the probability 816 associated with input 804a–804n. Although input layer 804, hidden layer 808, and output layer 812 are depicted as each consisting of three elements, each layer may contain any number of elements.

[0097] In various implementations, each layer of the LSTM neural network 802 must include the same number of elements as each of the other layers in the LSTM neural network 802. In some exemplary embodiments, a convolutional neural network can be implemented. Similar to an LSTM neural network, a convolutional neural network includes an input layer, hidden layers, and an output layer. However, in a convolutional neural network, the output layer includes one less output than the number of neurons in the hidden layers, and each neuron is connected to each output. Additionally, each input in the input layer is connected to each neuron in the hidden layers. In other words, input 804a is connected to each neuron in neurons 808a, 808b…808n.

[0098] In various implementations, each input node in the input layer can be associated with a numerical value, which can be any real number. Within each layer, each connection leaving an input node has a weight associated with it, which can also be any real number. In the input layer, the number of neurons equals the number of features (columns) in the dataset. The output layer can have multiple consecutive outputs.

[0099] As mentioned above, the layer between the input layer and the output layer is the hidden layer. The number of hidden layers can be one or more (for many applications, one hidden layer may be sufficient). A neural network without hidden layers can represent linearly separable functions or decisions. A neural network with one hidden layer can perform continuous mappings from one finite space to another. A neural network with two hidden layers can approximate any smooth mapping to any accuracy. Figure 8 The neural network can receive inputs, such as vectors, which can be used to generate a model that can, for example, be used to predict a driver's OTA vehicle update preferences based on vehicle sensor data and driving history.

[0100] Figure 9 The diagram illustrates an exemplary process for generating a machine learning model. At 907, control retrieves data from a database 902 (e.g., a data warehouse). The data may include any suitable data used to develop the machine learning model.

[0101] At 911, control separates the data obtained from database 902 into training data 915 and test data 919. Training data 915 is used to train the model at 923, and test data 919 is used to test the model at 927. Typically, the set of training data 915 is chosen to be larger than the set of test data 919, depending on the desired model development parameters. For example, training data 915 may include approximately 70%, approximately 80%, approximately 90%, etc., of the data obtained from database 902. The remaining 30%, 20%, or 10% is then used as test data 919.

[0102] Separating a portion of the acquired data as test data (919) allows the trained model to be tested against actual output data, facilitating more accurate training and model development in (923) and (927). The model can be trained in (923) using any suitable machine learning modeling technique, including those described herein such as random forests, generalized linear models, decision trees, and neural networks.

[0103] In 931, the test results of the evaluation model are controlled. For example, using test data 919, the trained model can be tested in 927, and the results from the output data of the model under test can be compared with the actual output of test data 919 to determine the level of accuracy. The model results can be evaluated using any suitable machine learning model analysis (such as exemplary techniques described further below).

[0104] After evaluating the model's test results in September 31, if the model's test results are satisfactory, the model can be deployed in September 35. Deploying the model may include using the model to make predictions on a large-scale input dataset with unknown outputs. If the evaluation results of the model's test results in September 31 are unsatisfactory, the model may be further developed using different parameters, different modeling techniques, or other model types. Figure 9 The machine learning model method can receive inputs, such as vectors, which can be used to generate a model that can, for example, be used to predict a driver's OTA vehicle update preferences based on vehicle sensor data and driving history.

[0105] The foregoing description is illustrative in nature and is in no way intended to limit this disclosure, its application, or use. The broad teachings of this disclosure can be implemented in various forms. Therefore, although this disclosure includes specific examples, its true scope should not be limited thereto, as other modifications will become apparent upon studying the drawings, specification, and the following claims. It should be understood that one or more steps within the method may be performed in a different order (or simultaneously) without altering the principles of this disclosure. Furthermore, while each embodiment in the examples is described above as having certain features, any one or more of those features described for any embodiment of this disclosure can be implemented in any embodiment of other embodiments, and / or combined with features of any embodiment of other embodiments, even if such combination is not explicitly described. In other words, the described embodiments are not mutually exclusive, and substitutions of one or more embodiments for each other remain within the scope of this disclosure.

[0106] Various terms are used to describe spatial and functional relationships between elements (e.g., between modules, circuit elements, semiconductor layers, etc.), including “connection,” “engagement,” “coupling,” “adjacent,” “next to,” “on top of,” “above,” “below,” and “placed.” Unless explicitly described as “direct,” when describing the relationship between the first and second elements in the above disclosure, the relationship can be a direct relationship where no other intermediary element is present between the first and second elements, but it can also be an indirect relationship where one or more intermediary elements are present (spatially or functionally) between the first and second elements. As used herein, the phrase at least one of A, B, and C should be interpreted as referring to the logic (A OR B OR C) using non-exclusive logic OR, and should not be interpreted as referring to “at least one of A, at least one of B, and at least one of C.”

[0107] In the accompanying drawings, as indicated by the arrowheads, the direction of the arrows typically shows the flow of information of interest to the illustration (such as data or instructions). For example, when components A and B exchange various information, but the information transmitted from component A to component B is relevant to the illustration, the arrow may point from component A to component B. This unidirectional arrow does not imply that no other information is transmitted from component B to component A. Furthermore, for information sent from component A to component B, component B may send a request for said information or an acknowledgment of receipt of said information to component A.

[0108] In this application, which includes the following definitions, the term "module" or "controller" may be replaced by the term "circuit". The term "module" may refer to, be part of, or include the following: application-specific integrated circuit (ASIC); digital, analog, or mixed-signal analog / digital discrete circuit; digital, analog, or mixed-signal analog / digital integrated circuit; combinational logic circuit; field-programmable gate array (FPGA); processor circuit (shared, dedicated, or group) that executes code; memory circuit (shared, dedicated, or group) that stores code executed by the processor circuit; other suitable hardware components that provide the described functionality; or combinations of some or all of the above, such as in a system-on-a-chip.

[0109] A module may include one or more interface circuits. In some examples, the interface circuits may include wired or wireless interfaces connected to a local area network (LAN), the Internet, a wide area network (WAN), or a combination thereof. The functionality of any given module of this disclosure may be distributed among multiple modules connected via interface circuits. For example, multiple modules may allow for load balancing. In another example, a server (also referred to as a remote or cloud) module may perform some functions on behalf of a client module.

[0110] As used above, the term "code" can include software, firmware, and / or microcode, and can refer to programs, routines, functions, classes, data structures, and / or objects. The term "shared processor circuit" covers a single processor circuit that executes some or all of the code from multiple modules. The term "group processor circuit" covers a processor circuit that, in conjunction with additional processor circuitry, executes some or all of the code from one or more modules. References to multiple processor circuits cover multiple processor circuits on a discrete die, multiple processor circuits on a single die, multiple cores of a single processor circuit, multiple threads of a single processor circuit, or a combination of the above. The term "shared memory circuit" covers a single memory circuit that stores some or all of the code from multiple modules. The term "group memory circuit" covers a memory circuit that, in conjunction with additional memory, stores some or all of the code from one or more modules.

[0111] The term memory circuit is a subset of the term computer-readable medium. As used herein, the term computer-readable medium does not cover transient electrical or electromagnetic signals propagating through a medium (such as on a carrier wave); the term computer-readable medium can therefore be considered tangible and non-transient. Non-limiting examples of non-transient tangible computer-readable media are non-volatile memory circuits (such as flash memory circuits, erasable programmable read-only memory circuits, or mask read-only memory circuits), volatile memory circuits (such as static random access memory circuits or dynamic random access memory circuits), magnetic storage media (such as analog or digital magnetic tape or hard disk drives), and optical storage media (such as CDs, DVDs, or Blu-ray discs).

[0112] The apparatus and methods described in this application can be implemented, in part or in whole, by a special-purpose computer created by configuring a general-purpose computer to perform one or more specific functions embodied in a computer program. The aforementioned functional blocks, flowchart components, and other elements serve as software specifications that can be routinely converted into computer programs by those skilled in the art or by programmers.

[0113] A computer program includes processor-executable instructions stored on at least one non-transitory tangible computer-readable medium. A computer program may also include or depend on stored data. A computer program may encompass a basic input / output system (BIOS) for interacting with the hardware of a special-purpose computer, device drivers for interacting with specific devices of the special-purpose computer, one or more operating systems, user applications, background services, background applications, etc.

[0114] Computer programs may include: (i) descriptive text to be parsed, such as HTML (Hypertext Markup Language), XML (Extensible Markup Language), or JSON (JavaScript Object Notation); (ii) assembly code; (iii) object code generated from source code by a compiler; (iv) source code for execution by an interpreter; and (v) source code for compilation and execution by a just-in-time (JIT) compiler, etc. As an example only, source code may be written using syntax from languages ​​including C, C++, C#, Objective-C, Swift, Haskell, Go, SQL, R, Lisp, etc. Fortran, Perl, Pascal, Curl, OCaml, HTML5 (Hypertext Markup Language, Fifth Revision), Ada, ASP (Dynamic Server Pages), PHP (PHP: Hypertext Preprocessor), Scala, Eiffel, Smalltalk, Erlang, Ruby, Visual Lua, MATLAB, SIMULINK and

Claims

1. A vehicle over-the-air (OTA) update system, comprising: Multiple vehicle sensors, each configured to detect vehicle sensor data indicating one or more vehicle operating parameters of the vehicle; A vehicle microphone is configured to capture audio from inside the vehicle; The antenna is configured to wirelessly receive vehicle OTA update data; and The vehicle control module is configured as follows: Obtain vehicle sensor data from the plurality of vehicle sensors and voice data from the vehicle microphone; Obtain driving history data of driving behavior associated with the vehicle during a specified historical time period; Multiple time frames are used to synchronize the vehicle sensor data, the voice data, and the driving history data to generate a synchronized dataset; The synchronized dataset is tokenized to formalize the embedding vectors and then fed into a trained multimodal machine learning model, which is configured to produce OTA-featured driver preference outputs. An OTA feature priority set is generated by feeding the embedding vector into a trained multimodal machine learning model; The OTA update execution schedule is determined based on the OTA feature driver preference output, wherein the OTA execution schedule includes a first OTA update feature with higher priority compared to a second OTA update feature; and The reception of OTA update data via the antenna is controlled according to the priority of the OTA update features in the OTA update execution schedule.

2. The vehicle OTA update system of claim 1, wherein the trained multimodal machine learning model is trained to receive input data from a source having multiple modalities and to produce an output indicating the likelihood of driver preferences for multiple OTA update features.

3. The vehicle OTA update system as claimed in claim 1, wherein the trained multimodal machine learning model includes at least one of a transformer model, a neural network, a large language model (LLM), or a memory-aware regression model.

4. The vehicle OTA update system as described in claim 1, wherein the first OTA update feature includes at least one of voice command control feature, adaptive cruise control feature, advanced driver assistance system feature, autopilot feature, or software-defined vehicle feature.

5. The vehicle OTA update system as described in claim 4, wherein: Controlling the reception of OTA update data includes receiving updates to advanced driver assistance system features; and The vehicle control module is configured to automatically control the acceleration, braking, and steering of the vehicle based on the updates to the features of the advanced driver assistance system.

6. The vehicle OTA update system as described in claim 1, wherein obtaining vehicle sensor data includes: The vehicle sensor data is formatted according to the {I, L, D} format to include a sensor dataset (I) with semantic annotation labels (L) and depth information (D); and The speech data is formatted using a large language model in the format {S, W}, where S is the original sound signal wave and W includes the recognized language words.

7. The vehicle OTA update system as described in claim 1, wherein synchronizing the vehicle sensor data, the voice data, and the driving history data includes: Obtain Global Positioning System (GPS) time frame data associated with the vehicle sensor data, the voice data, and the driving history data; and The vehicle sensor data, voice data, and driving history data are synchronized based on the GPS time frame data.

8. The vehicle OTA update system of claim 7, wherein synchronizing the vehicle sensor data, the voice data, and the driving history data includes interpolating time frames associated with the vehicle sensor data to create a matching time frame associated with each element of the vehicle sensor data.

9. The vehicle OTA update system as described in claim 1, wherein controlling the reception of OTA update data includes: The antenna in the vehicle receives multiple data packets wirelessly from a wireless transmitter. The multiple data packets are combined to define an OTA update block; and The OTA update block is stored in the memory of the vehicle control module.

10. A method for controlling over-the-air (OTA) updates for vehicles, the method comprising: The vehicle control module receives vehicle sensor data from multiple vehicle sensors and voice data from a vehicle microphone configured to capture audio from inside the vehicle, wherein the vehicle sensor data indicates one or more vehicle operating parameters of the vehicle. Obtain driving history data of driving behavior associated with the vehicle during a specified historical time period; Multiple time frames are used to synchronize the vehicle sensor data, the voice data, and the driving history data to generate a synchronized dataset; The synchronized dataset is tokenized to formalize the embedding vectors and then fed into a trained multimodal machine learning model, which is configured to produce OTA-featured driver preference outputs. An OTA feature priority set is generated by feeding the embedding vector into a trained multimodal machine learning model; The OTA update execution schedule is determined based on the OTA feature driver preference output, wherein the OTA execution schedule includes a first OTA update feature with higher priority compared to a second OTA update feature; and The reception of OTA update data via the vehicle's wireless antenna is controlled according to the priority of OTA update features in the OTA update execution schedule.

Images