AUTOMATIC LABELING OF DATA BY A DATA COLLECTING UNIT

Abstract

A system receives an identification of data to be collected via a request configured based on a configuration file associated with a machine learning model stored by a vehicle. The system receives an identification specifying how the data should be labeled, as defined by the configuration file, and creates one or more topics to publish the data on board the vehicle. The publication includes both the collected data and any metadata usable to label the data according to the definitions in the configuration file. The system also subscribes to the topics to receive the published data and metadata, and appends labels to the data using the metadata to label the data according to the labeling definitions in the configuration file.The system stores the labeled data in a vehicle memory as data assigned to the model.

Description

CROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application is a continuation of U.S. application Serial No. 17 / 746,746, filed May 17, 2022, the disclosure(s) of which are hereby incorporated by reference in their entirety. FIELD OF TECHNOLOGY

[0002] The illustrative embodiments generally relate to automatic labeling of data by an entity that collects the data. GENERAL STATE OF THE ART

[0003] Vehicles are becoming increasingly interconnected and more powerful. Complex computing systems and onboard microprocessors create a mobile computing environment capable of delivering a suite of services, intelligent sensing, reactive and dynamic control, and advanced driver feedback. Connectivity capabilities provide opportunities for vehicles to communicate with each other, with infrastructure, and with cloud-based backend servers. New software updates can be pushed directly to the vehicle, updating various software and firmware installed by original equipment manufacturers (OEMs), as well as optional applications included by a vehicle owner.

[0004] Continuous software development brings with it continuous opportunity for improvement. Each vehicle represents a mobile data-collecting entity, possessing countless data observations and a unique experience. Connected vehicles can report observations, but marshaling and distilling such a massive amount of data becomes a virtually impossible task, even for a group of people. Intelligent systems can become smarter through machine learning, but modifying and improving a model for such a disparate set of vehicles or circumstances can be difficult, as it involves handling a significant amount of data and requires the model to be tuned to observations that may only be relevant to a single vehicle or group of vehicles. The data collected may include habits, environments, local conditions, etc.whose applicability can be highly situation-specific. This presents an incredible challenge when deciding how to tune models and how to allow software to learn both for global applicability and individual improvement. Continuously tuning such models for even tens of thousands of vehicles could impose a massive load on backend servers, let alone tens of millions of active vehicles. SUMMARY

[0005] In a first illustrative embodiment, a system includes one or more processors of a vehicle configured to receive an identification of data to be collected via a request configured based on a configuration file associated with a machine learning model stored by the vehicle. The one or more processors are further configured to receive an identification of how to label the data, which is defined by the configuration file, and to generate one or more topics for publishing the data onboard the vehicle, wherein the publication includes both the collected data and any metadata usable to label the data according to the definitions in the configuration file.The one or more processors are further configured to subscribe to the topics to receive the published data and metadata, and to append labels to the data using the metadata to label the data according to the label definitions in the configuration file. The one or more processors are further configured to store the labeled data in a vehicle memory as data associated with the model.

[0006] In one illustrative embodiment, a method includes receiving, via a request, an identification of data to be collected configured based on a configuration file associated with a machine learning model stored by the vehicle. The system also includes receiving an identification of how to label the data, which is defined by the configuration file, and creating one or more topics for publishing the data onboard the vehicle, wherein the publication includes both the collected data and any metadata usable to label the data according to the definitions in the configuration file.The method further includes subscribing to the topics to receive the published data and the metadata, appending labels to the data using the metadata to label the data according to the labeling definitions in the configuration file, and storing the labeled data in a vehicle storage as data associated with the model.

[0007] In a third illustrative embodiment, a non-transitory, computer-readable storage medium stores instructions that, when executed by one or more processors, cause the one or more processors to perform a method that includes receiving, via a request configured based on a configuration file associated with a machine learning model stored by the vehicle, an identification of data to be collected. The system also includes receiving an identification of how to label the data, which is defined by the configuration file, and generating one or more topics for publishing the data onboard the vehicle, wherein the publication includes both the collected data and any metadata usable to label the data according to the definitions in the configuration file.The method further includes subscribing to the topics to receive the published data and the metadata, appending labels to the data using the metadata to label the data according to the labeling definitions in the configuration file, and storing the labeled data in a vehicle storage as data associated with the model. BRIEF DESCRIPTION OF THE DRAWINGS Fig. Figure 1 shows an illustrative example of an end-to-end AI / ML operations pipeline; Fig. Figure 2 shows an illustrative example of a learning process for a deployed model that may occur on board a vehicle; Fig. Figure 3 shows an illustrative example of a continuous monitoring process and flow; Fig. Figure 4 shows an illustrative example of an interface analysis of model monitoring; Fig. Figure 5 shows an illustrative example of inference as a service (IaaS); Fig. Figure 6 shows an illustrative example of model monitoring; Fig. Figure 7 shows an illustrative example of an online learning flow; Fig. Figure 8 shows an illustrative example of an online learning sequence; and Fig. Figure 9 shows an illustrative example of an automatic validation sequence. DETAILED DESCRIPTION

[0008] This specification describes embodiments of the present disclosure. It should be understood, however, that the disclosed embodiments are merely examples, and other embodiments may take various and alternative forms. The figures are not necessarily to scale; some features may be exaggerated or reduced to show details of specific components. Therefore, specific structural and functional details disclosed in this specification are not to be interpreted as limiting, but merely as a representative basis for teaching one skilled in the art to variously employ the present invention.It will be understood by those of ordinary skill in the art that various features illustrated and described with reference to any of the figures may be combined with features illustrated in one or more other figures to produce embodiments not explicitly illustrated or described. The illustrated combinations of features provide representative embodiments for typical applications. However, various combinations and modifications of the features consistent with the teachings of this disclosure may be desirable for particular applications or implementations.

[0009] In addition to example processes being performed by a vehicle computing system located in a vehicle, in certain embodiments, the example processes may be performed by a computing system in communication with a vehicle computing system. Such a system may include, but is not limited to, a wireless device (e.g., and without limitation, a cellular phone) or a remote computing system (e.g., and without limitation, a server) connected by the wireless device. Collectively, such systems may be referred to as vehicle-associated computing systems (VACS). In certain embodiments, particular components of the VACS may perform particular portions of a process depending on the particular implementation of the system.For example, and without limitation, if a process includes a step of sending or receiving information with a paired wireless device, then the wireless device is likely not performing that portion of the process, as the wireless device would not "send and receive" information to or from itself. Those of ordinary skill in the art will understand when it is inappropriate to apply a specific computational framework to a given solution.

[0010] Process execution may be facilitated by the use of one or more processors, operating alone or in conjunction with each other, executing instructions stored on various non-transitory storage media, such as, but not limited to, flash memory, programmable memory, hard disk drives, etc. Communication between systems and processes may include, for example, the use of Bluetooth, Wi-Fi, cellular communications, and other suitable wireless and wired communications.

[0011] In each of the illustrative embodiments discussed herein, an exemplary, non-limiting example of a process performable by a computing system is shown. With respect to each process, it is possible for the computing system executing the process to be configured as a special-purpose processor for performing the process for the limited purpose of executing the process. Not all processes need to be performed in their entirety, but are to be understood as examples of types of processes that can be performed to achieve elements of the invention. Additional steps can be added to or removed from the exemplary processes as desired.

[0012] With respect to the illustrative embodiments described in the figures and depicting illustrative process flows, it should be noted that a general-purpose processor may be temporarily enabled as a special-purpose processor for the purpose of performing some or all of the exemplary methods depicted by these figures. When executing code that provides instructions for performing some or all of the steps of the method, the processor may temporarily be converted to a special-purpose processor until the method is completed. In another example, to an appropriate extent, firmware operating in accordance with a preconfigured processor may cause the processor to act as a special-purpose processor provided for the purpose of performing the method or a reasonable variation thereof.

[0013] Vehicles can include fully connected vehicles, such as vehicles with both internal and external communications. Internal communications can be achieved through short-range and long-range wireless communications, as well as wired communications through a vehicle bus, such as a control area network (CAN) bus, and / or other data connections. External connections can include wireless connections, for example, with other vehicles (V2V), infrastructure (V2I), edge processors (V2E), mobile and other devices (V2D), and the cloud (V2C) through cellular or other wireless connectivity. Collectively, these connections can be referred to as V2X communications, where X is any entity with which a vehicle is capable of communicating.These vehicles may include distributed onboard processing, the ability to leverage edge and cloud processing, and a specialized architecture that can be repurposed to provide advanced computing services under at least certain circumstances.

[0014] Vehicles may also include software and firmware modules, and vehicle electronic control units (ECUs) may further include onboard processing. Vehicle features may include artificial intelligence and machine learning models that can provide advanced occupant services leveraging vehicle sensors and shared cloud data. The AI / ML models may be capable of self-evolving (in-vehicle online learning) to tailor the models to a vehicle context and user-preferred experience. Sensors may include, but are not limited to, cameras, LIDAR, RADAR, RFID, NFC, suspension sensing, occupant sensors, occupant identification, device detection, etc.

[0015] The illustrative embodiments and the like propose an artificial intelligence (AI) / machine learning (ML) operational pipeline (collectively, the "AI / ML operational pipeline") that provides strategically timed and placed machine learning opportunities that improve both the local and global model. This can leverage onboard computing, edge computing, and cloud computing, strategically exploiting the massive computational resources represented by each connected vehicle in a fleet. Vehicles can work to improve their own models for their own observations, situations, and owner / occupant preferences.

[0016] Improvements to global models can be implemented and deployed to improve the experiences of all vehicles in a fleet, locality, subset (similar demographics, environments, etc.).

[0017] In at least one embodiment, model owners are able to define learning parameters for when a model should be trained, as well as provide training data and expected values ​​for model performance, so that regressive models can be identified and new models can be tested, evaluated, validated, and deployed. By giving model owners control over these variables, model owners can be responsible for setting parameters for when a model should learn, without requiring direct control over the learning or even access to the learning process.At the same time, the AI / ML operational pipeline controller can be model-agnostic; if it receives properly formatted and defined information, it can act within its own boundaries to improve a model according to developer parameters, without direct knowledge of why a model was developed or what an improvement strategy should achieve.

[0018] This paradigm allows disparate and dissatisfied teams of engineers and coders to develop a suite of self-improving tools and platforms without having to continually communicate about goals and intentions. If a model is suitable for deployment and contains the correct information, each team can proceed with its own goals, drawing on the cohesive, interconnected system to provide learning for improvement, without directly attempting to monitor and train millions of versions of a deployed model, and without concern that their specific model will suffer from a lack of oversight.

[0019] Artificial intelligence (AI) can play a role in enabling an OEM's digital transformation toward a fully connected smart vehicle. AI can impact the entire vehicle lifecycle, from design to operation and vehicle disposal. AI-enabled experiences such as personalization, virtual assistants, driver assistance, predictive maintenance, and self-driving cars may become increasingly important, and the business operations team may want to leverage AI / ML to solve real-world problems. An AI / ML framework for fully networked vehicles (FNVs) can provide tools, capabilities, and best practices to accelerate data collection / processing, development / training, evaluation, deployment, and operationalization of ML models at scale.To provide more computing, the framework can take companies further by leveraging vehicles, wallboxes, mobile apps, 5G multi-access edge computing (MEC), and cloud computing to optimize performance and compute utilization.

[0020] Most processes for generating, building, validating, deploying, retraining, and redeploying AI / ML models on vehicles are fragmented. They are rarely automated and involve significant manual steps involving multiple teams, resulting in longer lead times for launching and repairing adaptive vehicle features. These teams can suffer from a fragmented AI / ML deployment strategy and a lack of a single framework for monitoring and managing the lifecycle of AI / ML models. Furthermore, many teams, especially third-party application developers, are unable to perform rapid prototyping / validation of AI / ML models on vehicles or test benches. This leads to conflicting goals and strategies and significantly slows down model improvement and deployment, and in some cases can virtually eliminate model improvement.

[0021] The illustrative embodiments provide a methodology for operationalizing and managing the lifecycle of AI / ML models at scale. They represent examples of a holistic software platform for increasing the productivity and scalability of dispatching, monitoring, and managing the lifecycle of online AI / ML models for a wide variety of ML-based solutions across the cloud, vehicles, mobile applications, and wallboxes.

[0022] The illustrative embodiments and the like can provide a cloud online model FNV AI / MI framework that allows the integration of ML models developed independently, yet allows these models to improve in an efficient and targeted manner. Through various aspects of standardization, guiding principles can allow a coherent approach to providing all the necessary information for the framework to function, even if an application developer plays no role in the framework, and vice versa.

[0023] Self-service tools and dashboards can allow testing and monitoring of generated insights in real time. The framework can support structured and unstructured data collection. The framework supports federated learning (learning from deployed models) and online learning (a vehicle learning from its own observations).

[0024] In the automotive context, the platform can be compliant with the Automotive Open System Architecture (AUTOSAR), allowing standardized integration of features and applications from non-OEM devices in a manner that allows suppliers and vendors to remain relatively ignorant of the platform beyond the requirements to drive data for the model (e.g., without limitation, expected inputs and outputs, training data, validation data, training triggers, etc.). If such data is required, it can be requested in a preformatted version, allowing integration into the platform without requiring the supplier to monitor the model once deployed.

[0025] Vehicle services can be abstracted through standard application programming interfaces (APIs) and a service-oriented architecture (SOA), which allow request and response models that are application-independent, as long as the application can handle the standardized data. A common software development kit (SDK) can be abstracted from the platform, providing centralized, high-level functionality.

[0026] High-performance compute clusters (HPCC), an architecture with shared memory, hardware abstraction, and virtualization (hypervisors / containers), can provide (in-vehicle) online learning environments suitable for model training in a dedicated scenario, where a model can be trained with strategic use of onboard resources and which does not necessarily require cloud access to train the model.

[0027] Fig. Figure 1 shows an illustrative example of an end-to-end AI / ML operational pipeline. This example includes a portion of the pipeline that can be executed "online" (onboard vehicle 100) and a portion that executes in the cloud / backend 130. The proposed pipeline allows vehicle 100 to collect its own data and make its own observations, as well as train its own models. Data is also sent back to cloud 130 as desired, which can allow for opportunities for federated learning, long-term data storage and management, and use of the data to improve other models that may not be deployed on vehicle 100.The cloud 130 can also track deployed versions of models and store other models for rollback, as well as provide general management for the entire software system of a given vehicle, which can be useful in determining updates and global model deployment strategies.

[0028] An AI / ML feature 121 may be located in a vehicle 100 and communicate with inference and compute as a service (IaaS and CaaS, respectively) 117. Shadow software 119 may run a newly trained version of a model against live data sets to determine whether the expected output from the new model is met, as well as to determine whether the produced output reflects a better version of the output than a currently deployed model 121 achieves.

[0029] The vehicle 100 may collect data 103 from a variety of onboard systems and sensors, as specified by data collection strategies 105. Each feature 121 may request its own data, and the data may be collected and stored specifically for a given feature according to the collection requirements, labeling, and specific parameters associated with that feature (e.g., collection cycles, data types, etc.). The vehicle 100 may automatically sanitize the data 107, removing all personal information and irrelevant information (irrelevant to the feature 121), and may perform automatic labeling 109, as discussed in more detail herein.

[0030] Once collected and labeled, the data can be used as training data for online learning 109, allowing learning onboard the vehicle 100 based on data observed by the vehicle 100, resulting in a personalized training dataset directly applicable to the relevant vehicle 100. The data can be stored locally until no longer needed and, if necessary, uploaded to the cloud 130 via an upload manager 127. Model versioning and repository 115 can track deployed versions and older, functional versions in case a model reversion or rollback is required if a newly deployed model begins to regress, which can be determined based on, among other things, information provided by the model owner (developer).

[0031] Newly trained models can be validated on board 113, which may also include running the models in shadow software using live data streams, where model performance can be tracked in real time and considered against expected values ​​(and the currently deployed model performance) to avoid deploying a model that, although newly trained, does not represent a significant improvement, as defined, for example, by the model owner.

[0032] Model monitoring 123 may provide continuous supervision of model execution against parameters defined by the model owner to detect outliers 125 that may signal regression or further learning opportunities.

[0033] The cloud 130 receives the vehicle data 131 and stores the data in a data repository 133, which may be a large-scale data store capable of storing data from all vehicles to facilitate federal learning opportunities and model improvement. Data lifecycle management 135 may dictate when data is retained or removed, as well as ensure that data is retained, edited, and deleted according to any overarching policies.

[0034] The data repository 133 may also receive data from federated learning 137, third parties 139 (e.g., services that provide general information, such as, but not limited to, traffic, environmental data, etc.), and simulated data 141. Simulated data may be data usable for training and represent data that would be obtained or is expected to be obtained under certain conditions, and may be used instead of or in conjunction with real data when a dataset requires additional information. Simulation models may also be trained from real data to represent improved simulations of data as observed in real time as reported by vehicles 100 to the cloud 130.

[0035] Data in the repository 133 may be cleaned 143 and anonymized 145. The data may be preprocessed 147 for sorting and standardization. Preprocessing may include, but is not limited to, noise removal, other filtering, downsampling, upsampling, etc. It may be used for feature engineering 149 for new features and may be further annotated 151 and / or initially labeled (if not already labeled). The annotation onboard the vehicle 100 may take the form of a label related to a given AI / ML feature (e.g., how that feature requests that the data be labeled), but a given piece of data may be usable for more than one reason and may therefore benefit from a more comprehensive set of labels. Once labeled, the data may be used to train any appropriate models 153 to which the labeled data may apply.This can provide a massive data input stream for cloud-based model training and be useful for global training of models (deployed or new), all of which can benefit from the observations of the group as a whole.

[0036] Any newly trained model can be subjected to verification and validation 155 in the cloud 130, which can again be performed according to parameters defined by the feature owner. This can indicate whether a newly trained model is functional, for example, according to expected outputs. The model's performance against each deployed version (which may eventually be vehicle-specific due to online learning) may not be easily tracked, but performance against the expected values ​​and globally deployed versions should be something that can be considered.

[0037] The model may further undergo hardware optimization 157 and be executed in a simulated manner on an optimal set of virtual hardware 159 to verify the functionality and outputs of the model. In at least one example, this optimization may occur prior to distribution of the model. A working version of the model, if validated and verified through successful simulation, may be stored in the model repository 161. If desired, model governance may also be provided, which attempts to ensure that the model is not biased using ethical AI practices, and this may also occur prior to deployment.This model repository may also include shadow software 163, which allows the model to be executed using real-time and continuous data inputs to provide further validation prior to deployment and to ensure that the model represents an improvement over a previously deployed version. Feature and model owners can define the parameters for improvement and / or redeployment, as well as regression and reversion parameters, allowing precise control based on a configuration file without requiring live monitoring of the inputs and outputs of a given model and newly trained versions of that model when the model undergoes continuous integration and continuous development (CI / CD).

[0038] Model management processes 169 in the cloud can determine when and whether to deploy a new version of a model that has been improved, for example, through general data collection and retraining. Cloud-based monitoring 167 and interpretability can further utilize live data stored in the repository 133 to monitor the expected performance of deployed models, allowing the identification of regression and unexpected outputs before a model regresses too far and / or performs well outside its operating parameters. Interpretability can provide explanations for why a certain prediction was obtained, what pattern led to the result, an association of factors that led to a result, etc.

[0039] Policies and addressing 171 can determine which vehicles (or devices, e.g., phones, wall boxes, etc.) should receive a newly updated model and / or require a rollback / reversion to an old model, for example, if monitoring reveals that a previously deployed version is operating abnormally. Deployment / rollback 173 delivers the appropriate version of the model to a given vehicle 100 (or other entity), and online models 175 can track and mirror all deployed versions of models, allowing monitoring 167 to accurately track the performance of models under a much larger dataset than the model is likely to receive from any single entity in which it is deployed.The monitoring that occurs under the larger data set can often reveal defects or regression probabilities much earlier than a vehicle 100 can, given the fundamentally different volume of data provided to each entity relative to the cloud 130.

[0040] Through this interconnected pipeline, models are continuously improved both online (in the vehicle / entity) and in the cloud 130. Personalized improvements can be realized online in the vehicle 100, making the deployed version of the AI / ML feature 121 specific to that vehicle 100. Cross-platform improvements can be realized through federated learning and cloud-based learning and monitoring, allowing for global deployment of improved versions of models.Feature owners can monitor performance and improvements and redefine configuration files to represent more aggressive parameters for improvement, and both the vehicle 100 and the cloud 130 can monitor deployed models for outlier detection (again, if desired, using feature owner parameters) as well as run shadow versions of newly trained models against live datasets (local and crowd-based) to determine if a new model performs better than a deployed version and when the new model is ready for live deployment.

[0041] Fig. Figure 2 shows an illustrative example of a learning process for a deployed model that may occur onboard a vehicle. An AI / ML feature 201 may represent the currently executing version of a given ML model, which may have a configuration file defining its own requirements and parameters for data collection. The vehicle 100 may provide personalized data according to such requirements, with any permissible requested data 203 being collected at any interval over which the data is requested.Policy management may dictate how much data is stored for a given model or service, since storage is not unlimited, but each model 201 may have a personalized version of data collected and annotated according to its own parameters, so that highly specific versions of the same general data can be stored for each model's use.

[0042] An automatic annotation / labeling process 205 can be used to label the data in real time, allowing immediate use of the data without having to wait for a manual labeling process to occur. This adds relevance to the data from the perspective of the given feature 201, producing a cleaned dataset 207 that can be stored 209 with respect to the given feature 201. This data can be replaced with newer data or retained for as long as needed and / or policy permits. Automatic labeling is discussed in more detail later in this document.

[0043] Learning as a Service (LaaS) 211 can provide online learning without reliance on cloud or edge computing, if desired. This allows the vehicle 100 to improve its own models on its own without having to wait for global improvement or deployment. This also allows leveraging HPCC and a targeted architecture of the vehicle 100 to efficiently train a model using the architecture of the vehicle 100 and distributes the computation that would be used to train a model across the extensive fleet of deployed vehicles 100, thereby avoiding the need to keep all computing power in a central repository and allowing the use of existing computing resources residing in a given vehicle 100 or other entity.

[0044] A newly trained model can be validated 213 against a set of feature owner parameters to ensure overall functionality and to determine whether the model represents an improved version of the current model 201. Once validated, the model can be pushed to the model repository, which can track deployed versions and run the new model as a shadow against live data to ensure performance before deployment. All the while, model monitoring 203 can track the performance of the executing live model 201 to prevent regression and identify training opportunities.

[0045] This online process allows vehicles to self-improve their own models using data relevant to both the model and the given vehicle. Because vehicles observe significantly varying information based on location, overall utilization, specific utilization types, owner / operator tendencies, environmental variances, and so on, the experience of a given vehicle is much less homogeneous than, for example, that of an application residing on a desktop computer at a fixed location in an office environment. While the utilization of any deployed model may vary based on operator tendencies and goals, vehicles themselves represent continuously variable operating environments and may require many additional variances over a typical model to ensure continued improvement and performance under a much broader set of variables.This is not to say that models deployed in vehicles do not benefit from global updates, but rather that they also benefit from a variety of personalized improvements in a way that can reflect a much more customized environment relative to a given vehicle than a simple improvement of a model trained relative to one type of user (common goals and workload) over another.

[0046] The model repository can exist in both the cloud and the vehicle. The cloud version of the repository can store generic models and master models, which can be ML models widely applicable to a fleet, region, environment, etc. Both the cloud and the vehicle can store specialized models, which are ML models unique to a given user or vehicle / VIN. These are models trained specifically for that user or vehicle. This could represent literally tens of millions of models in the cloud, and it may require appropriate data management.

[0047] Model sources can be both online learning (e.g., for specialized models) and federated learning (e.g., for global models). Data stored related to a given model can include, but is not limited to, model descriptions, algorithm classes (e.g., linear regression, decision tree, neural network, etc.), ML frameworks, contact information for model / feature owners, versions of training datasets used to achieve the model, vehicle dependencies, deployment modes, dependencies, activation triggers, etc.

[0048] Cloud 130 may be configured to continuously deliver updated models, such as when a new ML model reaches a specific branch in the model repository. For a given model, updates may also be scheduled for vehicles that may benefit from an identified version change between a deployed model and a newly trained model. Other delivery triggers may include one-click (instantaneous) deployment or be based on vehicle and / or lifecycle events. For example, a valid vehicle state may be required prior to deployment, which may include, but is not limited to, subscription / registration, vehicle authorization, user authorization, the presence or absence of certain software or hardware features, etc.

[0049] Model versioning is a process that can be associated with any AI / ML infrastructure. In the illustrative FNV AI / ML architecture, the concepts of online learning and federated learning can benefit from modified model versioning in the vehicle to track entities that are trained, fine-tuned, validated, and updated in the vehicle. This can include an AUTOSAR-compliant storage structure.

[0050] The model repository component 115 may, for example, leverage the Autosar persistence structure to implement an in-vehicle model storage structure. The model repository may be configured to store model metadata / configuration (information describing model inputs and outputs and versioning information) and models / coefficients (data required to initialize a kernel) in standardized persistence interfaces based on ara::persist.

[0051] Models can be divided into two categories: general models and specialized models. General models represent a model that is applicable to a large fleet of vehicles, for example, all vehicles of a specific model or all vehicles in a specific region. These models can be deployed through the cloud and trained in the cloud (even if the training is based on data collected from vehicles or federated learning) and deployed to a vehicle as part of a normal software / configuration load. Specialized models are models trained for a specific user or vehicle. They may not be part of the software load and can be trained either in the vehicle or in the cloud. Specialized models are typically derived from non-specialized original models.

[0052] One can leverage ara::persist services to access the model in general memory, and then provide coefficient services in addition to the ara::persist services to retrieve and update models in specialized memory. Models and any additional data or information can be updated independently for the generalized and specialized versions.

[0053] According to this paradigm, model retrieval can occur in a non-limiting illustrative example as follows. An AI / ML application can request compute using a generic model identifier, which can be a call to CaaS. CaaS can forward a call to the ara::persist services for the general model, which retrieves the general model from the general model store and passes the model back to CaaS. If the compute call involves a specialized version of a model (e.g., an online improvement learned for this vehicle), this can result in a call to the coefficient services, which use the ara::persist services to retrieve the specialized model from the specialized model store and pass the model back to CaaS.

[0054] Similarly, a generic non-restrictive example of learning to produce a specialized version of an existing general model can occur as follows.

[0055] The AI / ML application may include a CaaS call for learning that can leverage LaaS. The base model for learning can be obtained from general model storage, as described above, and CaaS can be executed on the base model in conjunction with LaaS to train the model. The model can then be validated, and LaaS can broadcast the training results. If the new model is to be saved, the AI / ML application can, in response to the broadcast of training results, update the coefficient services and request that CaaS use the coefficient services to save the new model to the specialized model storage.

[0056] For example, models related to vehicle features, such as a mount angle, can typically be general in nature (since the angle of a mount is a kind of "fact" whose nature should not change based on a given user's usage or preference—i.e., a 30-degree angle is a 30-degree angle everywhere). On the other hand, models related to a user—such as preferred window states and vehicle feature recommendations based on user-centric observations or environmental conditions—can be specialized models that, while derived from a base model, benefit significantly from specialized training to perform according to what a user would expect.

[0057] Models may also require validation prior to deployment, which may involve aggressive testing under a variety of circumstances to help ensure that a given vehicle 100 does not behave unexpectedly under a very broad range of circumstances. Before a model is deployed in a production environment, the model, with supporting inference code, can be deployed in a cloud environment. There, test data and validation / threshold data can be used in conjunction with the model to ensure that expected results / thresholds are achieved when the test data is executed. If the model is a new version of an existing model, difference testing can be used to compare the differences in execution from one version to another when the inputs (data) are the same.

[0058] The model can be deployed for testing with processor-in-loop evaluation on a test bench, evaluation board, spare hardware, etc. The model can also be deployed on target hardware (for the model) using a hardware-in-loop simulator to evaluate performance with a simulation of the addressed hardware. The model can then be further evaluated using a test or prototype vehicle to provide vehicle-in-loop testing.

[0059] Additionally, as previously noted, there may be a shadow mode of the model that can be executed in the target environment without impacting the performance of the currently running model. This allows for prediction based on both the existing and the new version of the model. Using live data, prediction inputs and outputs for both versions can be monitored and stored, and the resulting data can be used to determine whether the new version performs better than the existing version, which can serve as a deployment trigger.

[0060] Testing may also include CPU / GPU and memory usage performance, which may include, for example, prediction and task completion time. This may also include determining processor utilization and any need for parallel processing, as well as RAM utilization, code optimization requirements, caching needs, and / or communication protocol testing. The vehicle is a complex computing environment in which countless entities may use limited resources, and where the vehicle itself must often be prioritized for these resources when performing driving tasks. Accordingly, it can be useful to understand the computational and resource burden imposed by a model before deploying the model. A limited-scope release to a small subset of intended vehicles can also help highlight any overlooked operational issues prior to large-scale deployment.

[0061] Because models can be updated both specifically for vehicles and globally, it can be difficult to robustly test all versions of trained models. While a general model can be extensively tested before deployment, a specific version of the model can only be tested onboard a vehicle. This might include, for example, validation against parameters defined by the model owner and shadow execution to ensure conformance, but neither process can continue indefinitely, or the newer model will never be deployed. At some point, often defined by the feature owner and subject to the constraints of the feature / model owner, the model will be deployed.Model monitoring, described in more detail below, can continuously monitor active versions of models to both improve performance and identify needs for rollback or reversion.

[0062] Continuous model monitoring can, for example, monitor drift data (unexpected inputs) and concept drift (unexpected outputs). When a model is initially trained, it may be based on real-world data, but significant assumptions may be made about both the consistency of that data and the fact that the initial training data is truly representative of what a given vehicle will encounter. Even when a model is retrained for a new version, there is an assumption that the new data that resulted in the new version will continue to reflect the data that is still being observed.Accordingly, and particularly because of all the variables that exist regarding vehicles in terms of both location and utilization, it is useful to monitor both sides of the data (input and output) to ensure that there is some level of maintained consistency that reflects the state of affairs when the model was actually trained.

[0063] Fortunately, the vehicles in the illustrative examples are capable of self-monitoring and do not rely on cloud-based monitoring, which would require massive cloud computing if tens of millions of active models were continuously monitored against live data. Using configuration files defined by the feature / model owner, which can help determine what constitutes an actual aberration or training opportunity from the perspective of the builder, the vehicle can actively monitor its own specialized models and identify for itself when a model would benefit from training or reversion.

[0064] Fig. Figure 3 shows an illustrative example of a continuous monitoring process and flow. An initial AI / ML model may be scheduled for deployment 301 from the cloud, which may include sending a copy of the model to a telematics control unit (TCU) 303 of the vehicle 100. The model may include the code necessary to execute the model 307, as well as a configuration file that may define, for example, any data necessary to utilize and / or train the model, expected inputs, labeling parameters, expected outputs, training thresholds, reversion thresholds, etc. The TCU deploys the model to the data repository where it may be used by the vehicle 100 for its intended purpose.

[0065] When executed, the AI / ML model 311 loads the model from the data repository, an example of which is provided earlier in this document in relation to the discussion of the model repository. The data collection 313 receives instructions from the configuration file, according to the configuration file that defines what data is needed, and provides the requested data back to the AI / ML application. The data may also be sent to the TCU 303 for redistribution back to the cloud 130. This data may be used for learning in the cloud. The AI / ML application also sends requests to Inference as a Service (IaaS) 315, which helps provide the output for the model.

[0066] Continuous monitoring in this example includes both monitoring 300 and interpretability 302. Interpretability extracts feature values ​​317 from the AI ​​model and feature statistics 319 from the configuration file and compares the current feature values ​​obtained by running the AI ​​model with the expected values ​​obtained from the AI ​​model 321. These can be, for example, features that contribute most to predictions made by a model. The model can have expected values ​​for these features, and the configuration file can have expected statistics for these features. The interpretability function 302 can extract the values ​​for the features during model runtime and compare them with the expected values.For example, a feature owner can define thresholds for features and define which features provide the greatest contribution, which can be used in interpretability to determine the impact of one or more features on a result. For example, the features can be extracted from pixel values ​​in an image as a vector of numeric values. This vector is a low-level representation of the image. Certain indices may be critical for predictions, and the values ​​at these indices would contribute the most to model predictions. A configuration file can define the relevant contributions.

[0067] This information is used to generate an interpretability message 323, which identifies which features lead to incorrect predictions or predictions that fall outside the range of acceptable deviation. The interpretability message can be delivered to both the cloud and a model owner. Metadata associated with the model and stored in the repository can define which entities should receive copies of which messages, how frequently, and / or under what conditions.

[0068] The monitoring process 300 checks the configuration file 325 for a model type and input / output expectations. This check may include verifying that the model type (e.g., regression, classification, etc.) is supported 327 and that the model input (e.g., numeric, images, etc.) is supported 329. Once this has been verified, the appropriate model functionalities may be started 331 to receive the model input from the data collection. The appropriate processing functionalities may also be started 333 to process the incoming data to produce results (output) according to the model's capabilities in response to the received input.

[0069] While the model receives input and processes output, the monitoring process can continuously check the input 335 and output 337 against the expected values ​​obtained from the configuration file. Thus, if the model produces an unexpected output, the monitoring process can identify whether or not an unexpected input is a possible cause. Likewise, if the model experiences an unexpected input, the monitoring process can determine whether or not this changes the output values ​​outside of parameters. By including both expected results and tolerances / thresholds, the model owner can define what constitutes unacceptable data, allowing the vehicle to identify opportunities for model improvement or reversion.The results of the analysis performed by monitoring can be consolidated in a performance report and distributed to the cloud and / or the model owner.

[0070] Analysis across input and inference cycles can be continuous and may involve comparing historical distribution data (provided by the cloud) with historically used data that may be stored onboard. This reveals whether the locally observed data over a larger area matches the expected historical distribution, and the local data is also shared with the cloud to update the expectations and the historical distribution. Because the data is actually observed, it can't be "wrong," but it can present an opportunity to train a model with different weights if the data is skewed relative to what was expected (based on longer-term observation).For example, with respect to something like environmental data, a recent distortion may represent a change in seasons or a movement of the vehicle to a different climate, which can affect services related to HVAC and other user-centric recommendations, as well as active system controls and feedback.

[0071] If the feature owner provides an updated dataset distribution resulting from training in the cloud, the vehicle can compare the local training set with the updated set. This may indicate an opportunity to update the local training data, collect additional data, or reweight the collected data. Drift in the data can help identify a circumstance unique to the vehicle or an expectation that may have been inappropriately set (or no longer holds). Configuration file "alert rules" and thresholds can, for example, define when drift is considered significant for a given model and / or when to warn the feature owner, when to train the model, and / or when to revert to a previous version of the model.

[0072] Before a model is deployed, a model owner can generate distributional expectations about a model's input, which can be derived from the training set. Similarly, distributional expectations about the output can be generated, and the expectations and a model type can be stored in a configuration file distributed with a given model. The procedures (e.g., mathematics, logic, etc.) used to calculate the expectations can also be provided, allowing a replica of these procedures to be used by model monitoring.

[0073] For example, a traffic sign classification model may take images of a traffic sign as input, with the expected output specifying a sign type (stop, speed limit, road, etc.). Monitoring may track the confidence rating resulting from the model, for example, an expectation that the confidence is higher than 70%. In one example, if a certain number of images in a series are predicted with confidence values ​​less than 70%, this may trigger an alert and identify an opportunity to further evaluate and / or train the model. The evaluation may yield at least temporary regression if the current model is unsatisfactory.

[0074] Fig. Figure 4 shows an illustrative example of an interface analysis of model monitoring. Cloud 130 stores at least one generalized version of a model or application 401. Data 403 may include, for example, weights, architecture, metadata, and a configuration file. An example configuration file 405 may include, for example, a model type, an input distribution, an output distribution, and evaluation metrics.

[0075] The model 401 is deployed from the cloud, for example, via the TCU 411 of the vehicle 100. The TCU stores the model 401 in the data repository 413, which may include storing the appropriate weights, metadata, architecture, and configuration file, as previously noted. The application 415 loads the model from the repository 413 and subscribes to the required data, as noted by the configuration file or elsewhere in the model. The model monitor 300 obtains the expected values ​​from parsing the configuration file 417 and thus obtains an expected input and expected output distribution.

[0076] The monitor may also subscribe to the input data stream to obtain the live input data for use in a comparison with the expected distributions, as well as subscribe to IaaS 423, which provides the model's output based on the live input data. This information may be used to compare the real output distribution with the expected output distribution 421. Based on the comparisons, process 300 may forward a message to data collection 425, which may report the performance to the relevant parties (e.g., those required by the metadata or other parameters defining the message).

[0077] As briefly discussed earlier in this paper, inference as a service (IaaS) is a capability that can be included in the illustrative framework. An inference is, for example, the result of any computation, statistical calculation, or output from an AI / ML model. IaaS is a publication of inferences to which clients can subscribe. The AI / ML framework provides a set of functionalities that can enable sharing of the inference or computation output across multiple entities under specific conditions according to a service-oriented architecture (SOA).

[0078] For example, the IaaS layer can include a standard and future-independent method for publishing inferences to the SOA. Functionality for converting computational output into computational input—converting inferences to the SOA into input—and common functionality for performing predictions. This strategy can benefit from a generic interface structure to be published and an identification of which service interface to publish inferences.

[0079] Service elements can contain fields that contain, for example, the most recent inference value, model metadata, whether inferences are actively published, error handling, and other additional information. The service elements can also handle methods for enabling and disabling inferences, for example, when the inferences are controlled by clients.

[0080] The platform can provide common functionality for converting between output / input and inference via SOA. For example, for a common kernel used for prediction, the platform can provide common code to extract data from the kernel output and publish the data via SOA. For scenarios where SOA inference is used in further computation (e.g., inter-SOC computation), the platform can provide common code that allows the use of inference in computation. For example, if a region of interest (ROI) is detected and published via SOA, it may also be needed to crop the input to another model. The ROI can crop around any pixel or group of pixels, and the ROI can be of interest to more than one model.The platform could then provide an Open VX kernel for SOA subscribers, feeding the ROI into the rest of the OpenVX pipeline. This would represent a valid inter-SOC pipeline. Or, for example, when inferences are fused to provide an inference with improved prediction or a better confidence estimate.

[0081] In general, the execution of an inference computation and the publication of the resulting inference can be determined by various factors. These can include, for example, when an inference stream needs to be active. This can be determined by various predefined scenarios, such as when a use case is active, a vehicle state, repeated feature requests, etc. A trigger condition can define which specific condition triggers an individual computation and publication of an inference on the SOA when the use case is active. Input data can define which data should be fed as inputs to the computation kernel. The kernel type can define which predictor function is used to obtain a given inference. A given model can define the parameters / coefficients / weights (e.g., trained) to be used in the predictor function.

[0082] A data broker can support the evaluation of the "when" trigger condition and input data considerations in the previous example, as these factors may be similar to the factors driving data collection. The other example conditions, kernel type, and trained model, can be stored in a data repository, as they are likely specific to a given AI / ML model.

[0083] Use cases can include, but are not limited to: fully custom code to collect and preprocess data with a custom model that can use the standard IaaS structure to publish a resulting inference to the SOA; use of data intent or predefined triggers to run a prediction process and publish the inference using a standard structure; or requests for features to kick-start a prediction process and publish the inference in a request-response manner.

[0084] For example, a diesel particulate filter monitor may include a diesel particulate filter prediction. The kernel and coefficients of this model may be black-box C code produced in MATLAB. The logic for aggregating the data, performing preprocessing, and even the model may be highly customized and therefore unable to rely on standard kernels for execution. However, the results (the inference) can still be published using the standard structure and service.

[0085] In another example, reversing assistance might be a use case where inputs are clearly defined, such as a node in an OpenVX graph and some standard signals. These inputs can be shared with many other models. Inference is required once a vehicle is in a certain state and for each frame, and the kernel can be highly standardized. For scenarios like this, the architecture can support deployment with minimal or no production code for use case inference. Platform code can be configurable with the state condition, references to image and signal inputs, and the desired model (kernel and coefficients).

[0086] The output of IaaS could be transformed into multiple forms, one for publication in a standard SOA format, allowing, for example, a binary output to be converted to the SOA format using a custom kernel. Another transformation might involve converting an IaaS output into kernel input for further computation. Furthermore, it might be useful to publish a translated output (from IaaS) as instances on the SOA so that multiple clients can subscribe to it.

[0087] In a general example, IaaS can receive a request from an AI / ML application owner and kick-start the computational graph by requesting a compute-as-a-service module. IaaS can then respond to the application owner with the inference stream topic identifier and provide a computational graph identifier as acknowledgment of the request. Based on the received identifier, the application owner can subscribe to the inference topic and request the computational graph for any custom data preprocessing. The request sent by the application owner to IaaS can include the model required to be executed, required input data, and details of custom triggers, if any, based on which the data broker subscribes to the data. The model can be loaded from persistent storage for computational purposes by making a request to the model repository.Once the input data and model are ready, or based on the trigger, the inference is computed and published to SOA via IaaS. The computation can be executed, for example, in a loop, based on the trigger request timeout, or until the stopping conditions are met, after which memory cleanup can be performed.

[0088] Fig. Figure 5 shows an illustrative example of inference as a service for a possible ML application. Application 501 may be executed when a user 503 turns on an ignition at 523, signaling a vehicle 505. The user may be identified in some way (since this may be a personalized feature in this example), and this identifier and the key-on signal may be passed to configuration files for IaaS 513, which may be loaded upon key on.

[0089] The configuration files specify that a trigger signal at 531 causes a request for inference, which may also include any relevant data for the request, such as, but not limited to, the ID, model ID, trigger details, etc. This request is forwarded to IaaS / CaaS 517, which (since it will perform the computation on the model) subscribes to input data and trigger parameters at 533 so that IaaS knows when the model is triggered and has the relevant data to execute the model. The subscription here is done with the data broker 519. IaaS / CaaS 517 also creates an inference topic on the SOA 515 to which modules and features can subscribe, which in this example are features for lowering windows 509 and for radio station prediction 511 of a packet for an adaptive window butler 507.

[0090] IaaS / CaaS also returns a response to the initial request from the configuration file at 537, which may include an identification of which inference topics were generated, so that lowering windows 509 can subscribe at 539 and radio station prediction 511 can subscribe at 541. Feature 507 may also have use cases specified for both lowering windows at 527 and radio station prediction at 529, so that these features are active and can subscribe to the relevant inference topics.

[0091] At 543, IaaS / CaaS loads the model(s) for the user, which may be user-specific or possibly regionally or demographically specific for a preference prediction model (windows down, radio). The model repository 521 returns the model at 545.

[0092] The loop, which may be based on a trigger, activates the model(s) to produce inferences usable by the various features. Once a trigger occurs (temperature, precipitation, time of day, etc.), the model(s) at 547 and 549 receive inputs from the data broker 519 and run the models using the data. Relevant data for each model may be different (and in this case, likely is), although certain weather data, for example, may be relevant to both if it influences the window position and driver mood, which produces a different radio station selection.

[0093] IaaS / CaaS 517 publishes data on the SOA 515 to the created topics at 551. Lowering windows accesses its topic data (to which it has subscribed) at 553, and the radio station prediction accesses the data to which it has subscribed at 555. Outputs from these features are passed to the Butler based on the inferences at 557 and 559, and the Butler feature 507 can control the window or radio accordingly. While the vehicle 505 is moving, the Butler detects events at 561 that may trigger another inference, and the loop can continue. If the user 501 turns off the vehicle 505 at 563, this key turn-off is passed to the configuration files at 565, which can instruct the computing to stop at 567 and the data to stop at 569.

[0094] Model monitoring can also take into account the inferences and confidence levels and, for example, determine whether a model is ready for training or needs reversion using values ​​specified in the configuration file.

[0095] Fig. Figure 6 shows an illustrative example of a monitoring process for a newly deployed model. A developer 601 prepares a model that is tested and ready for deployment, and this new model is deployed at 615 by sending it to a vehicle TCU 603. At an appropriate time, the TCU 603 stores the model at 617 in a model repository 613 onboard the vehicle. The vehicle now has a working, accessible version of the model stored onboard.

[0096] When a feature or process calls / loads the model at 621, the model is loaded from the repository 613 at 623, and the output from the model is provided to the model monitoring process 611. This allows the vehicle to continuously monitor the model while it is in use. The configuration file associated with the model defines when an input or output is outside of acceptable limits or represents an opportunity for improvement. The monitoring process checks the configuration file loaded with the model into the monitoring process 613 at 625.

[0097] The model has also subscribed to the appropriate data and therefore model inputs are received at 627. While the model uses these inputs according to its current form to produce results usable in the vehicle, the monitoring process 613 also validates the input data against the expectations from the configuration file. The model further produces the output requested by the feature calling the model produced by IaaS 609, which was invoked by the model in conjunction with CaaS. This output is also received at 631 by the monitoring process 613. That is, while the feature uses the output to perform its function, the monitoring process also receives the output to validate the output against the expectations contained in the configuration file at 633.A model owner can define these expectations and / or redefine these expectations as the model evolves, with the redefinition achieved by loading a new configuration file into the model repository.

[0098] Using the comparison of the inputs and outputs with the expectations, the model monitoring process 613 generates a performance message at 635 and publishes it to any entities that have subscribed to it. A copy of the performance message is received at 637 by the data collection process 605, which uses the TCU 603 to send the message to the cloud. The model performance may also be received by the application owner 607, who may use the information to determine whether, for example, the configuration file needs to be updated or whether the model is performing poorly at 639. Observations of poor performance provide an opportunity to decide at 641 to revert to an old version of the model, which may be stored in the model repository 613. The IaaS 609 may also be notified of the reversion at 643 so that it can use the older version of the model.

[0099] Shadow mode can also provide a similar benefit by running a new (but undeployed) version of the model against current inputs. Monitoring can verify the inputs and outputs sent to and produced by the shadow version of the model, comparing them with expectations, and also determining whether the outputs represent, for example, "better" outputs than the currently deployed model. If so, then the new model is both functional and improved (the monitoring process can also validate that the new model does not produce out-of-bounds or erroneous results, again at least based on the configuration file that defines the parameters for both improvement and error).

[0100] Deploying an improved model from the cloud or based on an (in-vehicle) online learning process may involve dependency checks for the target environment. Furthermore, the previous (working) version of the model can be stored in the model repository in case future inputs result in poor model performance. Observing poor performance, which may not have been apparent based on the data used to originally test the model, may result in a rollback to the older, previously working version.

[0101] Once all dependencies have been verified and any authentication has been performed, the deployment process can also determine a pre-deployment vehicle state, as the vehicle may be required to be in certain states (e.g., driven mode, key-off mode, moving mode, sleep mode, charging mode, etc.) before deploying a new model. If necessary, customer consent can also be obtained (via the vehicle or an application interfacing with the vehicle). If the new model originates from the cloud, it can first be deployed in a shadow mode for a given vehicle to validate the model using live data from the vehicle before live deployment.The vehicle is intended to make the newly deployed model the currently functioning model, if necessary, but the model monitoring process will continue to monitor the model over time in case reversion or training is needed.

[0102] Models in the illustrative embodiments can be trained using at least two techniques, referred to as federated learning and / or online learning. Online learning is the live improvement of a model onboard a given vehicle, where the model is trained over time based on relevant observations and inputs provided by the vehicle. This represents an efficient utilization of computational resources (taking advantage of the vehicle's computer) and also allows models to be trained for a given vehicle. Because vehicles experience widely varying variables in virtually all facets of observed and collected data, as well as utilization, initial models can evolve very differently for different vehicles used in different contexts by different users.

[0103] For example, one user may lightly drive a vehicle in weather that is consistently above 60 degrees F and observe virtually no traffic. Another user may drive the vehicle with heavy loads, hundreds of miles a day, in heavy traffic in an environment that changes aggressively with the seasons. The vehicle itself may be specifically designed for one (or neither) of the two uses, but the evolution of models according to the context (who, what, when, where, why, and how) can produce models that achieve a given goal for each user, as determined by the context. Either model installed on the other vehicle may be inappropriate (from an input and / or output perspective) for the other vehicle, but may ultimately be very well suited to the context of the vehicle in which it evolved.Accordingly, online learning allows for efficient computational utilization and context-relevant evolution without requiring massive cloud computing and significant ongoing data transfers, as well as high costs and bandwidth utilization when the cloud attempts to continuously evolve tens of millions of versions of a model based on specifically observed circumstances on a per-vehicle basis.

[0104] While it may be the case that models adapt to a specific vehicle, the online learning process also provides an essential source of input for federated learning. That is, as models evolve, common themes and improvements may emerge that can be useful across many vehicles. Instead of ingesting all data into the cloud, the evolved models themselves can be used to evolve a global model that can be adapted at least in ways that appear to represent common improvements for all vehicles or selected, but large, subsets of vehicles. Thus, when a new vehicle receives the global model as a baseline, it will have a better version of the model than originally developed. Similarly, vehicles that experience slow or no model evolution (e.g.,limited inputs and workload), receive updated model versions without having to evolve the model on board. Furthermore, some vehicles may train a model incorrectly or poorly and directly benefit from federated learning by being taught (via the deployment of an updated global model) the thing(s) these vehicles should have learned or likely would have eventually learned.

[0105] Both online learning and federated learning can be used to update some or all vehicles. Online learning as a global source can be useful for vehicles in a similar context—e.g., similarly used vehicles operating in similar environments. This may be particularly true, for example, if a fleet owner simply expands its fleet, doubling the number of each vehicle, and intends for these vehicles to perform similar services under similar conditions as their previously existing counterparts. Instead of having to wait for all new vehicles to learn what the existing vehicles have learned, the online-learned models can be distributed to the appropriate and relevant new vehicles.

[0106] As noted, federated learning can be applicable to all or a subset of vehicles. Accordingly, federated learning can be used to update a version of a model that also has associated context—e.g., this model is "better" for vehicles in this context. In other cases, federated learning can simply reveal a better aspect of a model that is globally applicable and can be used in all instances as an improvement to the global model. Cloud-based repositories can track every deployed version of every model for every vehicle, if desired, including what it includes, what was changed, when it was changed, which new model was deployed, etc.

[0107] In one example, online learning is categorized into three possible scenarios for illustrative purposes: light learning + light data (LLLD), light learning + heavy data (LLHD), and heavy learning + heavy data (HLHD). For example, LLLD can involve learning algorithms such as linear / logistic regression or k-NN with small data. LLHD can involve use cases requiring feature extraction, dimensionality reduction, etc., and can be processed in an accelerated domain, while the actual training / inference can be lightweight. The training effort can be the same as with LLLD, except that data is streamed from the dataset to the accelerated domain. HLHD can involve neural network-based methods and some methods such as SVM / Random Forest / K-NN that run directly in the high-dimensional accelerated domain.

[0108] Fig. Figure 7 shows an illustrative example of an online learning flow. In this example, an AI / ML application 701 may load a model intended for learning from the model repository 703. Real-time learning 705 may involve learning while data is collected and may be configured for training and status monitoring actions. If an inference is incorrect, real-time training may occur. It may call LaaS 707, which uses CaaS 709 for training. The results of the training (a new model) may require validation, and therefore the AI / ML application may call an automatic validation module 711, which uses IaaS 713 and test inferences, to determine the validity of the new model.Batch-based learning 715 can function in a similar way, but can use stored data from previous observations to train the model based on a batch of data for which model monitoring has suggested better results can be obtained. IaaS can return the inferences and / or run the model in shadow mode 717, which can include model monitoring taking into account data interpretability 719. The weights of retuned and / or validated models can be sent to the cloud.

[0109] When the AI / ML model is running, it receives data from data collection 721, and an onboard process 723 automatically labels the data received from the vehicle. Data labeling and automatic labeling are discussed in more detail later in this document, but generally, data collected by the vehicle is more useful to models and learning if it has some identification (label) indicating what it represents. This can often involve a human going through the data and appending labels, but waiting for this step can make online learning virtually impossible, and accordingly, the vehicle is equipped to automatically label the data in some ways.

[0110] Training can begin when the AI / ML model decides that training is appropriate, which can vary from model to model and can be defined, at least in part, by the configuration file for a given model. For example, if a model performs poorly relative to expectations, training can be initiated.

[0111] Training data can be collected by the data collection module in response to a request, and this data can be automatically labeled by the automatic labeling module for use in training. The learning process can invoke LaaS, which uses the labeled training data and generates a graph (training binaries) to be used by CaaS, which can occur, for example, during vehicle shutdown to avoid overloading the computing resources used by the vehicle while driving.

[0112] LaaS can monitor training status using preconfigured thresholds (e.g., accuracy, loss) and can abort training if the reported accuracy and / or loss exceed thresholds. Training progress can also be saved in the event of a training interruption, such as vehicle operation. A newly trained model can be saved for validation.

[0113] Following validation, the new model is versioned and saved to the model repository. Furthermore, once the model has been validated but before deployment, model inference can be tested in shadow mode, mirroring the current model (except using the new model) and using live data as inputs. The interpretability from this optional step can provide an explanation of the model's internal behavior, which can be particularly useful if the model is not performing as expected.

[0114] Fig. Figure 8 shows an illustrative example of an online learning sequence. In this example, a model is newly deployed by a developer 801 in a TCU 803 at 817. At an appropriate time, the TCU 803 saves the model to the model repository 815 at 819. When the model is ready for training, it is requested at 821 and returned from the model repository at 823. The training process may also request data from the data collection 805 at 825. The collected data may be returned to the training process at 827 in response to the request, and automatic labeling may label the data at 829. If batch-based training occurs, previous data (already labeled) may also be loaded at 831.

[0115] The requirements and configuration for a learning job are passed to LaaS 811 at 833, which forwards the requirements to the OpenVX module 813. OpenVX 813 queues for an available compute job at 835, and if compute is available at 837, the module starts the compute used for training at 839. While training occurs, the status of the model is updated at 843, and the LaaS process 811 monitors the training at 845, as discussed above. The LaaS may abort training at 847 for any convenient reason, or training may complete at 849 and the model may proceed to validation at 851. Assuming the model validates, it may be saved to the model repository at 853.

[0116] LaaS can load and configure a kernel based on an application-defined configuration. For example, a neural network-based learning process can provide training with logistic regression, a configuration for training an NNN classifier (&image, &label, batch), and a Tensorflow TrainNNRegressor configuration (&image, &label, batch). A classical ML learning process can provide training with SVM regression, a configuration for training a random forest classifier, and a configuration for Markov model training. For example, training can occur on a batch containing images with certain labels in a batch of a certain size.

[0117] The LaaS implements a data accessor for the kernel based on the configuration and hyperparameters and interfaces with the dataset to feed data to the selected kernel (directly or through a streaming graph). The dataset can be split into validation and training data, and the LaaS can interface with the AI / ML application to receive a computational graph and submit it to the CaaS. The LaaS can further interface with the CaaS to submit a graph with SOA nodes as input, SOA nodes as output, and the supplied training kernel. The message from LaaS to CaaS can include a set of API calls provided by the OpenVX standard, such as creating nodes, joining nodes, validating graphs, executing graphs / streaming graphs.Streaming can involve repeated execution over streamed data provided by input nodes until training is completed or aborted.

[0118] As training progresses, the LaaS can receive progress logs and transmit them to the AI / ML application. Training progress and kernel progress can be disclosed in an SOA-compatible format and exchanged between the CaaS and the AI / ML application. Any abort request from the AI / ML application can be forwarded from the LaaS to the CaaS, and the training progress can be saved in the event of an abort or termination.

[0119] Federated learning can occur when one or more online learning updates result in newly valid models. The weights of the new model from each successful (or a subset of successful) online learning session can be shared with the cloud, which can modify the global model based on these online learning sessions. Data used to train the individual models can be obtained from the respective vehicles.

[0120] Autolabeling is the process of automatically annotating data, such as adding metadata, with minimal or no active human intervention. When a vehicle AI / ML model is to be trained on a vehicle, it is often done using live or recently collected data. This means the data cannot undergo a traditional process where a human reviews the data and decides how to label it. Accordingly, it can be useful to have a module onboard the vehicle that can both label the data and append labels that make the data useful to a given subscriber.This means that not only can the data be labeled, but it can, for example, be labeled with relevant context based on at least one or more subscribing AI / ML models, so that the labels make the data useful to the subscribing model used to label the data.

[0121] Several illustrative examples demonstrate various non-limiting examples of types of labels that may be used in this context. In sensor-derived auto-labeling, a sensor annotates the data. For example, an image may have an associated acceleration label, and a sensor may provide an inertial measurement to assist in labeling that data. In human-derived auto-labeling, human input may be used in conjunction with the data label to label the data either directly or indirectly. Indirect auto-labeling involves annotations accessed indirectly based on human interaction, and direct auto-labeling involves annotations accessed directly based on an intended interaction with a human.Algorithm-derived auto-labeling uses an algorithm to label data, and synthetic data can be used when annotations are generated via generative CL algorithms such as GANs.

[0122] Automatic labeling can be used in several contexts, including, but not limited to: increasing cloud workflow - when precise (human) labeling is required, automatically labeling collected data to be later labeled by a human can improve throughput and initial sorts; in online learning, data that does not require human labeling can be collected and labeled for use by online learning without involving a human labeler in the process at all;and identifying a dataset as a cluster of users—for example, when data is needed that requires hyperlocal knowledge, such as lane centering to align a vehicle to a corner of a lane when overtaking a large vehicle, automatic labeling can label data showing where a vehicle is located relative to a lane marking. This can assist in determining group preferences (location relative to lanes across a group) and individual preferences (where in a lane a given driver prefers to drive). More personalized driver data might include, for example, steering wheel angles achieved by the driver when driving within a lane.

[0123] The auto-labeling mechanism may be the responsibility of an AI / ML application owner, as auto-labeling can be specific to a given use case. That is, the configuration file for the AI / ML application can define how certain data should be labeled. The following is an example.

[0124] An AI / ML application owner can monitor the application's performance based on key performance indicators (KPIs) provided by model monitoring and implement logic to determine whether automatic labeling is necessary based on the KPIs. The AI / ML application owner can identify the data required to generate a dataset (e.g., subscriptions to various vehicle data, timing, scope, etc.). The owner can then implement the logic required to generate the dataset—that is, the logic to assign a label to the data.During data collection, the generated dataset is stored in persistent storage. The AI / ML application owner can configure the dataset to perform this step and provide some dataset cleaning utilities to remove extra data not required by the specific model. Based on the input data, the AI / ML application owner generates and publishes the dataset. The dataset reads the dataset and stores it in persistent storage. The dataset can then perform data cleaning on the stored data. This process can continue until a trigger to stop it occurs, provided, for example, by the AI / ML application owner or the vehicle platform.

[0125] An illustrative automatic labeling sequence might consist of three parts: initiation, loop, and trigger / stop. Initiation, in this example, is the decision of the AI / ML application owner and involves a request to the data collection that configures the data collection to set up publishers and subscribers in the data collection and data architecture. The AI / ML feature owner can then subscribe to input data and process it to generate a dataset. The AI / ML application can then publish the dataset, and the data collection can subscribe to that dataset. The subscribed data can then be stored in persistent storage according to the above configuration. At some point, triggers can indicate a termination of the data collection, which closes the publishers and subscribers. The data collection can then provide a brief summary of the collected data—e.g., size, samples, etc.

[0126] In one example, mount angle data (derived from images of a mount) can be automatically labeled using synthetic data via generative machine learning. Inputs to the labeling process can include a real-time image of a mount, as well as a reference image with a known mount angle. The GAN process produces a rendered image, with the input image rendered as the reference image, and a mount angle for the rendered input image. This data, which is the rendered image, is now labeled with a reference angle and can be used for training, online learning, analytics, etc., but no human had to measure the angle or consider the image.

[0127] In another example, an adaptive butler process can learn whether a suggestion should be labeled as valid or invalid based on human responses to "suggestions." Inputs in this case can include a human ID supplied by the AI / ML owner, a location (GPS) supplied via a CAN signal, a timestamp supplied via the data architecture, an applicable time zone supplied by the AI / ML owner, a GPS track showing historical position data supplied via the CAN, events that are voice commands to interact with the feature, and the vehicle speed supplied via the CAN. The result is a suggested window position that attempts to move the window to a correct position as desired by that person at that time under those circumstances.

[0128] The application can monitor the performance of the deployed model using model monitoring. Model monitoring provides the application with the following metrics: true positive (TP), false positive (FP), true negative (TN), and false negative (FN). Misclassifications refer to occurrences of false positives and false negatives. Once a misclassification is identified, the signal data is collected using FNV4 data collection scripts. The resulting data is labeled to indicate whether the signal (window position) was correct or incorrect. In one example, this can be considered instances where the human left the window unchanged or manually changed the window following the suggested positioning. This is an example of indirect labeling.Direct labeling can, for example, involve asking the human whether the position was correct. Among other training reasons, false negatives and false positives can indicate opportunities to train the model.

[0129] In addition to online learning, the vehicle can benefit from being able to validate newly proposed or improved models. Just because a model has been trained doesn't mean it will result in a more intelligent model. Before deploying a working version of a model, the vehicle can validate the model, avoiding the need to send the model to the cloud for validation (although the cloud could later attempt to validate the model with a much larger dataset from many vehicles).

[0130] Shadow mode is useful for validation in that it can benchmark both the previous / current model and a retuned model using the same live data. This means that without deploying the model, shadow mode can effectively run both versions on live data and determine which model performs better. An AI / ML application owner can define what constitutes an improved model (metrics) in the configuration file, as well as provide instructions on whether or not to append training data to a validation set.

[0131] Fig.Figure 9 shows an illustrative model validation sequence. As in previous instances, a new model may be deployed from the cloud 901 to the TCU 903 at 917 and eventually saved to the model repository 915 at 919. When shadow mode attempts to validate a newly trained version of the model, the application owner may request both the original and tuned model at 921. This may involve a subscription to the online learning module 911, since a newly trained model can only be saved to the repository after validation. However, if the owner 907 has subscribed to online learning for the model, they will be able to receive the tuned version of the model at 923 when training is complete.

[0132] The owner can then request training data and / or valid data from data collection 905 at 925, which returns valid data at 927. This data and both versions of the model can be sent to the validation module 913 at 929, which can benchmark both models at 931 using the provided data. Validation results are returned to the owner at 933, where, based on the KPIs specified in the configuration file, the owner can specify either to discard the new model at 935 or to save and deploy the new model at 937. Validation data can include owner-defined validation data and randomly sampled validation data. The model validation process can further randomly sample available data at 939 and update the second validation dataset, returning this information to the model owner at 941.

[0133] Some models can be validated onboard, while others may not. For example, information models such as window position prediction may be suitable for onboard validation. On the other hand, certain security / privacy-related features can be validated offline (in the cloud).

[0134] Although exemplary embodiments are described above, these embodiments are not intended to describe all possible forms encompassed by the claims. The terms used in the specification are terms of description rather than limitation, and it is understood that various changes may be made without departing from the spirit and scope of the disclosure. As described above, the features of various embodiments may be combined to form further embodiments of the invention that may not be expressly described or illustrated.While various embodiments may have been described as advantageous or preferred over other prior art embodiments or implementations with respect to one or more desired characteristics, one of ordinary skill in the art will recognize that one or more features or characteristics may be compromised to achieve the desired overall system attributes, depending on the specific application and implementation. These attributes may include, but are not limited to, strength, durability, marketability, appearance, space requirements, size, maintainability, weight, manufacturability, ease of assembly, etc.Accordingly, embodiments that are described as less desirable than other embodiments or prior art implementations with respect to one or more characteristics are not outside the scope of the disclosure and may be desirable for particular applications.

[0135] According to the present invention, a system is provided comprising: one or more processors of a vehicle configured to: receive, via a request configured based on a configuration file associated with a machine learning model stored by the vehicle; receive an identification of how to label the data, which is defined by the configuration file; create one or more topics for publishing the data onboard the vehicle, wherein the publication includes both the collected data and any metadata usable to label the data according to the definitions in the configuration file; subscribe to the topics to receive the published data and the metadata;Appending labels to the data using the metadata to label the data according to the label definitions in the configuration file; and storing the labeled data in a vehicle repository as data associated with the model.

[0136] According to one embodiment, the metadata includes vehicle sensor data published for the topic in conjunction with assigned data through the configuration file to be labeled based on one or more sensor values ​​present at the time the data was collected.

[0137] According to one embodiment, the request includes a timing for collecting the data, wherein the timing is defined by the configuration file.

[0138] According to one embodiment, at least one of the one or more processors is further configured to remove data from the collected data that does not correspond to a region of interest in the collected data, wherein the region of interest is defined based on the configuration file.

[0139] According to one embodiment, the metadata includes an identification of a positive or negative response by a human occupant in response to an action to change a vehicle state generated based on the model, and the positive or negative response indicates whether the change in the vehicle state was accepted or modified, respectively.

[0140] According to one embodiment, the metadata includes a response to a query made by a vehicle occupant directly via a vehicle display, the query being defined by the configuration file.

[0141] According to one embodiment, the configuration file defines an algorithm usable to modify the collected data to produce a modified version of the data and to determine a property of the data used to label the data based on the modified version of the data.

[0142] According to the present invention, a method includes: receiving, via a request configured based on a configuration file associated with a machine learning model stored by the vehicle; receiving an identification of how to label the data, which is defined by the configuration file; creating one or more topics for publishing the data onboard the vehicle, the publication including both the collected data and any metadata usable to label the data according to the definitions in the configuration file; subscribing to the topics to receive the published data and the metadata; appending labels to the data using the metadata to label the data according to the labeling definitions in the configuration file;and storing the labeled data in a vehicle memory as data associated with the model;

[0143] In one aspect of the invention, the metadata includes vehicle sensor data published to the subject in association with assigned data through the configuration file to be labeled based on one or more sensor values ​​present at the time the data was collected.

[0144] In one aspect of the invention, the request includes a timing for collecting the data, the timing being defined by the configuration file.

[0145] In one aspect of the invention, the method includes removing data from the collected data that does not correspond to a region of interest in the collected data, wherein the region of interest is defined based on the configuration file.

[0146] In one aspect of the invention, the metadata includes an identification of a positive or negative response by a human occupant in response to an action to change a vehicle state generated based on the model, and the positive or negative response indicates whether the change in the vehicle state was accepted or modified, respectively.

[0147] In one aspect of the invention, the metadata includes a response to a query from a vehicle occupant made directly via a vehicle display, the query being defined by the configuration file.

[0148] In one aspect of the invention, the configuration file defines an algorithm usable to modify the collected data to produce a modified version of the data and to determine a property of the data used to label the data based on the modified version of the data.

[0149] According to the present invention, a non-transitory, computer-readable storage medium storing instructions that, when executed by one or more processors, cause the one or more processors to perform a method, includes: receiving, via a request configured based on a configuration file associated with a machine learning model stored by the vehicle; receiving an identification of how to label the data, which is defined by the configuration file; creating one or more topics for publishing the data onboard the vehicle, wherein the publication includes both the collected data and any metadata usable to label the data according to the definitions in the configuration file;Subscribing to the topics to receive the published data and metadata; appending labels to the data using the metadata to label the data according to the label definitions in the configuration file; and storing the labeled data in a vehicle repository as data associated with the model.

[0150] In one aspect of the invention, the metadata includes vehicle sensor data published to the subject in association with assigned data through the configuration file to be labeled based on one or more sensor values ​​present at the time the data was collected.

[0151] In one aspect of the invention, the request includes a timing for collecting the data, the timing being defined by the configuration file.

[0152] In one aspect of the invention, the method includes removing data from the collected data that does not correspond to a region of interest in the collected data, wherein the region of interest is defined based on the configuration file.

[0153] In one aspect of the invention, the metadata includes at least one identification of a positive or negative response by a human occupant in response to an action to change a vehicle state generated based on the model, and indicating the positive or negative response, whether the change in the vehicle state was accepted or modified, respectively, or a response to a query from a vehicle occupant made directly via a vehicle display, the query being defined by the configuration data.

[0154] In one aspect of the invention, the configuration file defines an algorithm usable to modify the collected data to produce a modified version of the data and to determine a property of the data used to label the data based on the modified version of the data.

Claims

[1] System comprising: one or more processors of a vehicle configured to: Receiving an identification of data to be collected via a request configured based on a configuration file associated with a machine learning model stored by the vehicle; Receiving an identification of how to label the data, which is defined by the configuration file; Creating one or more topics to publish the data on board the vehicle, the publication includes both the collected data and any metadata usable to label the data according to the definitions in the configuration file; Subscribe to the topics to receive the published data and metadata; Attaching labels to the data using the metadata to label the data according to the label definitions in the configuration file; and Storing the labeled data in a vehicle memory as data associated with the model. [2] The system of claim 1, wherein the metadata includes vehicle sensor data published for the topic in association with assigned data by the configuration file to be labeled based on one or more sensor values ​​present at the time the data was collected. [3] The system of claim 1, wherein the request includes a timing for collecting the data, the timing being defined by the configuration file. [4] The system of claim 1, wherein at least one of the one or more processors is further configured to remove data from the collected data that does not correspond to a region of interest in the collected data, the region of interest being defined based on the configuration file. [5] The system of claim 1, wherein the metadata includes an identification of a positive or negative response by a human occupant in response to an action to change a vehicle state generated based on the model, and the positive or negative response indicates whether the change in the vehicle state was accepted or modified, respectively. [6] The system of claim 1, wherein the metadata includes a response to a query from a vehicle occupant made directly via a vehicle display, the query being defined by the configuration file. [7] The system of claim 1, wherein the configuration file defines an algorithm usable to modify the collected data to produce a modified version of the data and to determine a property of the data used to label the data based on the modified version of the data. [8] Method comprising: Receiving an identification of data to be collected via a request configured based on a configuration file associated with a machine learning model stored by the vehicle; Receiving an identification of how to label the data, which is defined by the configuration file; Creating one or more topics to publish the data on board the vehicle, the publication includes both the collected data and any metadata usable to label the data according to the definitions in the configuration file; Subscribe to the topics to receive the published data and metadata; Attaching labels to the data using the metadata to label the data according to the label definitions in the configuration file; and Storing the labeled data in a vehicle memory as data associated with the model. [9] The method of claim 8, wherein the metadata includes vehicle sensor data published for the topic in association with assigned data by the configuration file to be labeled based on one or more sensor values ​​present at the time the data was collected. [10] The method of claim 8, wherein the request includes a timing for collecting the data, the timing being defined by the configuration file. [11] The method of claim 8, further comprising removing data from the collected data that does not correspond to a region of interest in the collected data, wherein the region of interest is defined based on the configuration file. [12] The method of claim 8, wherein the metadata includes an identification of a positive or negative response by a human occupant in response to an action to change a vehicle state generated based on the model, and the positive or negative response indicates whether the change in the vehicle state was accepted or modified, respectively. [13] The method of claim 8, wherein the metadata includes a response to a query from a vehicle occupant made directly via a vehicle display, the query being defined by the configuration file. [14] The method of claim 8, wherein the configuration file defines an algorithm usable to modify the collected data to produce a modified version of the data and to determine a property of the data used to label the data based on the modified version of the data. [15] A non-transitory, computer-readable storage medium storing instructions that, when executed by one or more processors, cause the one or more processors to perform a method comprising: Receiving an identification of data to be collected via a request configured based on a configuration file associated with a machine learning model stored by the vehicle; Receiving an identification of how to label the data, which is defined by the configuration file; Creating one or more topics to publish the data on board the vehicle, the publication includes both the collected data and any metadata usable to label the data according to the definitions in the configuration file; Subscribe to the topics to receive the published data and metadata; Attaching labels to the data using the metadata to label the data according to the label definitions in the configuration file; and Storing the labeled data in a vehicle memory as data associated with the model.

Images