Automatic tagging of data by entities collecting data

Abstract

The present disclosure provides automatic tagging of data by entities that collect data. A system receives an identification of data to gather via a request configured based on a configuration file associated with a machine learning model stored by a vehicle. The system receives an identification, defined by the profile, of how the data will be tagged, and creates one or more topics for issuing the data on the vehicle, the publishing includes both the collected data and any metadata that can be used to mark the data according to the definition in the configuration file. Moreover, the system subscribes to the topic to receive the published data and the metadata, and supplements a tag to the data using the metadata to tag the data according to the definition in the configuration file about a tag. The system saves the marked data in a vehicle memory as data associated with the model.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application is a continuation of U.S. application serial number 17 / 746,746 filed on May 17, 2022, the disclosure of which is hereby incorporated by reference in its entirety. Technical Field

[0003] The illustrative embodiments generally relate to automatic tagging of data by an entity collecting the data. Background Art

[0004] Vehicles are becoming increasingly connected and powerful. Complex computing systems and onboard microprocessors create a mobile computing environment capable of providing a suite of services, intelligent sensing, reactive and dynamic control, and advanced driver feedback. Connectivity provides the opportunity for vehicles to communicate with each other, with the infrastructure, and with cloud-based backend servers. New software updates can be pushed directly to the vehicle, updating various original equipment manufacturer (OEM) installed software and firmware, as well as optional applications included by the vehicle owner.

[0005] Continuous software development brings with it continuous opportunities for improvement. Each vehicle represents a mobile data collecting entity with countless data observations and unique experiences. Connected vehicles can report observations, yet collating and distilling such a large volume of data is a nearly impossible task for even a group of humans. Smart systems can get smarter through machine learning, but modifying and improving models for such a diverse group of vehicles or situations can be difficult, involving processing large amounts of data and requiring models to be tuned to observations that may only be relevant to a single vehicle or group of vehicles. Sensing and data may be related to habits, environment, local conditions, etc., and its applicability may vary from situation to situation. This presents incredible challenges in deciding how to tune the model and having the software learn for both global applicability and individual improvements. Continuously tuning such models for even 10,000 vehicles can place a huge burden on backend servers, let alone doing so for tens of millions of active vehicles. Summary of the invention

[0006] In a first illustrative embodiment, a system includes one or more processors of a vehicle, the one or more processors being configured to receive an identification of data to be collected via a request configured based on a configuration file associated with a machine learning model stored by the vehicle. The one or more processors are also configured to receive an identification of how the data is to be labeled as defined by the configuration file, and to create one or more topics for publishing the data on the vehicle, the publication including both the collected data and any metadata that can be used to label the data according to the definition in the configuration file. Moreover, the one or more processors are configured to subscribe to the topic to receive the published data and the metadata, and to append tags to the data using the metadata to label the data according to the definition of the relevant tags in the configuration file. The one or more processors are also configured to save the labeled data as data associated with the model in the vehicle memory.

[0007] In a second illustrative embodiment, a method includes receiving an identification of data to be collected via a request configured based on a configuration file associated with a machine learning model stored by the vehicle. The method also includes receiving an identification of how the data is to be labeled as defined by the configuration file, and creating one or more topics for publishing the data on the vehicle, the publication including both the collected data and any metadata that can be used to label the data according to the definition in the configuration file. The method also includes subscribing to the topic to receive the published data and the metadata; appending tags to the data using the metadata to label the data according to the definition of the relevant tags in the configuration file; and saving the labeled data as data associated with the model in the vehicle memory.

[0008] In a third illustrative embodiment, a non-transitory computer-readable storage medium stores instructions that, when executed by one or more processors, cause the one or more processors to perform a method comprising receiving an identification of data to be collected via a request configured based on a configuration file associated with a machine learning model stored by the vehicle. The method also includes: receiving an identification of how the data is to be labeled as defined by the configuration file, and creating one or more topics for publishing the data on the vehicle, the publication including both: the collected data and any metadata that can be used to label the data according to the definition in the configuration file. The method also includes: subscribing to the topic to receive the published data and the metadata; using the metadata to append tags to the data to label the data according to the definition of the relevant tags in the configuration file; and saving the labeled data as data associated with the model in the vehicle memory. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] Figure 1 An illustrative example of an end-to-end AI / ML operational pipeline is shown;

[0010] Figure 2 An illustrative example of a deployed model's learning process that may occur on a vehicle is shown;

[0011] Figure 3 Illustrative examples of continuous monitoring processes and procedures are shown;

[0012] Figure 4 An illustrative example of model-monitored interface analysis is shown;

[0013] Figure 5 An illustrative example of Inference as a Service (IaaS) is shown;

[0014] Figure 6 An illustrative example of model monitoring is shown;

[0015] Figure 7 An illustrative example of an online learning process is shown;

[0016] Figure 8 An illustrative example of an online learning sequence is shown; and

[0017] Fig. 9 An illustrative example of an automatic verification sequence is shown. DETAILED DESCRIPTION

[0018] Embodiments of the present disclosure are described herein. However, it should be understood that the disclosed embodiments are merely examples, and other embodiments may take on various forms and alternative forms. The drawings are not necessarily drawn to scale; some features may be enlarged or minimized to show the details of a particular component. Therefore, the specific structural details and functional details disclosed herein are not to be interpreted as limiting, but only as a representative basis for teaching those skilled in the art to adopt the present invention in various ways. As will be understood by those of ordinary skill in the art, the various features shown and described with reference to any one of the drawings may be combined with the features shown in one or more other drawings to produce embodiments that are not explicitly shown or described. The combination of features shown provides representative embodiments for typical applications. However, for specific applications or embodiments, various combinations and modifications of features consistent with the teachings of the present disclosure may be desired.

[0019] In addition to having the exemplary process performed by a vehicle computing system located in a vehicle, in certain embodiments, the exemplary process may also be performed by a computing system that is in communication with the vehicle computing system. Such a system may include, but is not limited to, a wireless device (such as, but not limited to, a mobile phone) or a remote computing system (such as, but not limited to, a server) connected via a wireless device. Such systems may be collectively referred to as a vehicle associated computing system (VACS). In certain embodiments, specific components of the VACS may perform specific portions of the process depending on the specific implementation of the system. As an example and not limitation, if a process has a step of sending or receiving information with a paired wireless device, it is likely that the wireless device did not perform that portion of the process because a wireless device does not "send and receive" information with itself. One of ordinary skill in the art will understand when it is inappropriate to apply a particular computing system to a given solution.

[0020] Execution of the processes may be facilitated through the use of one or more processors working alone or in conjunction with one another and executing instructions stored on various non-transitory storage media such as, but not limited to, flash memory, programmable memory, hard drive, etc. Communications between the system and the processes may include the use of, for example, Bluetooth, Wi-Fi, cellular communications, and other suitable wireless and wired communications.

[0021] In each of the illustrative embodiments discussed herein, exemplary, non-limiting examples of processes that may be performed by a computing system are shown. With respect to each process, the computing system that performs the process may become a dedicated processor configured to perform the process for the limited purpose of performing the process. All processes need not be performed in their entirety, and are understood to be examples of types of processes that may be performed to implement elements of the present invention. Additional steps may be added or removed from the exemplary processes as desired.

[0022] With respect to the illustrative embodiments described in the accompanying figures showing illustrative process flows, it should be noted that a general purpose processor may be temporarily enabled as a special purpose processor for the purpose of performing some or all of the exemplary methods illustrated by these figures. When executing code providing instructions to perform some or all of the steps of the method, the processor may be temporarily repurposed as a special purpose processor until the method is completed. In another example, to the extent appropriate, firmware acting in accordance with a preconfigured processor may cause the processor to act as a special purpose processor provided for performing the method or some reasonable variation thereof.

[0023] Vehicles may include fully networked vehicles, such as vehicles with both internal and external communications. Internal communications may be achieved through short-range and long-range wireless communications as well as wired communications through a vehicle bus (e.g., a control area network (CAN) bus and / or other data connections). External connections may include wireless connections to, for example, other vehicles (V2V), infrastructure (V2I), edge processors (V2E), mobile and other devices (V2D), and the cloud (V2C) through cellular or other wireless connections. These connections may be collectively referred to as V2X communications, where X is any entity that the vehicle is able to communicate with. These vehicles may include on-board distributed processing, the ability to leverage edge and cloud processing, and a dedicated architecture that may be repurposed in at least some cases for providing highly advanced computing services.

[0024] The vehicle may also include software and firmware modules, and the vehicle electronic control unit (ECU) may also include on-board processing. Vehicle features may include artificial intelligence and machine learning models that can provide advanced occupant services using vehicle sensors and shared cloud data. The AI / ML models may be capable of self-evolution (online in-vehicle learning) to tune the model to the vehicle context and user preferred experience. Sensors may include, but are not limited to, cameras, lidar, radar, RFID, NFC, suspension sensing, occupant sensors, occupant recognition, device sensing, etc.

[0025] The illustrative embodiments, among others, present an artificial intelligence (AI) / machine learning (ML) operational pipeline (collectively, the "AI / ML operational pipeline") that provides strategically timed and placed machine learning opportunities to improve both local and global models. This can leverage onboard, edge, and cloud computing, and can strategically leverage the vast computing resources represented by each connected vehicle in a fleet. Vehicles can improve their own models for their own observations, situations, and owner / occupant preferences. Improvements to the global model can be implemented and deployed to improve the experience for all vehicles in a fleet, location, subset (similar demographics, environments, etc.).

[0026] In at least one embodiment, the model owner is able to define learning parameters for when the model should be trained, as well as provide training data and expected values ​​for model performance, so models that are regressing can be identified, and new models can be tested, evaluated, validated, and deployed. When control over these variables is given to the model owner, the model owner can be responsible for setting parameters about when the model should learn without directly controlling the learning or even accessing the learning process. At the same time, the AI / ML operational pipeline control can be model agnostic, and if it receives appropriately formatted and defined information, it can act within its own scope to improve the model according to the developer parameters without directly knowing why the model or what the improvement strategy should achieve.

[0027] This paradigm allows disparate and dissatisfied engineering teams and coders to develop a set of self-improving tools and platforms without the need to constantly communicate goals and intent. If the model is suitable for deployment and includes the right information, each team can continue with their own goals, relying on the cohesive interconnected system to provide learning for improvement without directly trying to monitor and train millions of versions of the deployed model or worrying about the lack of supervision for their specific model.

[0028] Artificial intelligence (AI) can play an enabling role in OEMs’ digital transformation towards fully connected smart vehicles. AI can impact the entire lifecycle of a vehicle, from design to operation and vehicle disposal. AI-enabled experiences such as personalization, virtual assistants, driver assistance, predictive maintenance, and autonomous vehicles are likely to become increasingly important, and business operations teams want to leverage AI / ML to solve real-world problems. The Fully Connected Vehicle (FNV) AI / ML Framework provides tools, capabilities, and best practices to accelerate data collection / processing, development / training, evaluation, deployment, and operationalization of ML models at scale. To provide greater computing power, the framework pushes the enterprise to the edge, leveraging vehicles, wall boxes, mobile applications, 5G multi-access edge computing (MEC), and cloud computing to optimize performance and computing usage.

[0029] Most of the processes for creating, building, validating, deploying, retraining, and redeploying AI / ML models on vehicles are decentralized. It is rarely automated, and it involves a large number of manual steps in collaboration with multiple teams, resulting in longer lead times for launching and fixing adaptive vehicle features. These teams can suffer from decentralized AI / ML deployment strategies and lack of a single framework to monitor and manage the lifecycle of AI / ML models. In addition, many teams, especially third-party application developers, are unable to perform rapid prototyping / validation of AI / ML models on vehicles or test benches. This leads to conflicting goals and strategies, as well as significantly slowing down the improvement and deployment of models, and in some cases, can almost prohibit model improvement.

[0030] The illustrative embodiments provide a method for operating and managing the lifecycle of ML models at scale. They represent examples of overall software platforms to improve productivity and scalability in transportation, monitor and manage the lifecycle of online AI / ML models for various ML-based solutions across cloud, vehicles, mobile applications and wall boxes.

[0031] Illustrative embodiments and the like can provide a cloud online model FNV AI / ML framework that allows the integration of ML models developed independently of each other and still allows these models to be improved in an efficient and targeted manner. Through various aspects of standardization, guidelines can allow a cohesive approach to providing all the necessary information required for the framework to operate, even if the application developer does not play any role in the framework, and vice versa.

[0032] Self-service tools and dashboards may allow real-time testing and monitoring of generated insights. The framework may support structured and unstructured data collection. The framework supports federated learning (learning from deployed models) and online learning (the vehicle learns from its own observations).

[0033] In an automotive environment, the platform may be AUTOSAR compliant, allowing standardized integration of functionality and applications from non-OEM devices, thereby allowing vendors and suppliers to be relatively agnostic of the platform, beyond the requirements for controlling model data (e.g., but not limited to expected inputs and outputs, training data, validation data, training triggers, etc.). When such data is needed, it may be requested in a preformatted version that allows integration into the platform without requiring the vendor to monitor the model after deployment.

[0034] Vehicle services can be abstracted through standard application programming interfaces (APIs) and service-oriented architectures (SOA), allowing application-agnostic request and response models as long as the application can process standardized data. A common software development kit (SDK) can be abstracted from the platform, providing centralized high-level functions.

[0035] High-performance computing clusters (HPCC), shared memory architectures, hardware abstraction, and virtualization (hypervisor / containers) can provide an online (in-vehicle) learning environment suitable for model training in dedicated scenarios, where models can be trained by strategically using on-vehicle resources and do not necessarily require any cloud access to train the models.

[0036] Figure 1An illustrative example of an end-to-end AI / ML operational pipeline is shown. The example includes a portion of the pipeline that can be run "online" (on the vehicle 100) and a portion that is executed in the cloud / backend 130. The proposed pipeline allows the vehicle 100 to collect its own data, and make its own observations, and train its own models. Data is also sent back to the cloud 130 as needed, which can allow for federated learning opportunities and long-term data storage and management, as well as using the data to improve other models that may not be deployed on the vehicle 100. The cloud 130 can also track deployed versions of models and store other models for rollback, as well as provide general management for the entire software system of a given vehicle, which may be useful in determining updates and global model deployment strategies.

[0037] AI / ML features 121 may reside on the vehicle 100 and communicate with Inference and Compute as a Service 117 (IaaS and CaaS, respectively). Shadow software 119 may run a newly trained version of a model against a live data set to determine if the expected output from the new model is met, and to determine if the resulting output reflects a better version of the output than the currently deployed model 121 is achieving.

[0038] The vehicle 100 may collect data 103 from various onboard systems and sensors, as dictated by a data collection policy 105. Each feature 121 may request its own data, and data may be collected and stored specifically for a given feature according to the collection request, tags, and specific parameters associated with that feature (e.g., collection cycle, data type, etc.). The vehicle 100 may automatically clean the data 107, thereby removing any personal information and irrelevant information (not relevant to the feature 121), and may perform automatic tagging 109, as discussed in more detail herein.

[0039] Once the data is collected and labeled, it can be used as training data for online learning 109, allowing learning to be performed on the vehicle 100 based on data observed by the vehicle 100, giving a personalized training data set directly applicable to the relevant vehicle 100. The data can be stored locally until no longer needed, and uploaded to the cloud 130 by the upload manager 127 when appropriate. If a newly deployed model begins to regress, the model versioning and repository 115 can track deployed versions and older functional versions in the event that a model recovery or rollback is required, which can be determined based on information provided by the model owner (developer), etc.

[0040] Newly trained models 113 may be validated on-board, which may also involve executing the model in shadow software using a real-time data stream, where model performance may be tracked in real-time and considered against expected values ​​(and the performance of currently deployed models) to avoid deploying a model that, while newly trained, does not represent a significant improvement as defined, for example, by the model owner.

[0041] Model monitoring 123 may provide ongoing supervision of the executing model against parameters defined by the model owner to detect outliers 125 that may signal regression or further learning opportunities.

[0042] The cloud 130 receives the vehicle data 131 and stores the data in a data repository 133, which may be a large-scale data store that can store data from all vehicles to allow for federated learning opportunities and model improvements. Data lifecycle management 135 may dictate when data is saved or removed, as well as ensure that data is saved, edited, and deleted according to any overall policies.

[0043] The data repository 133 may also receive data from federated learning 137, third parties 139 (e.g., services that provide general information, such as but not limited to traffic, environmental data, etc.), and simulated data 141. Simulated data may be data that can be used for training, which represents data that will be achieved or expected to be achieved under certain conditions, and can be used instead of or in conjunction with real data when additional information is required for the data set. Simulation models may also be trained on real data to represent an improved simulation of data observed in real time when the vehicle 100 reports to the cloud 130.

[0044] The data in the repository 133 may be cleaned 143 and anonymized 145. The data may be preprocessed 147 for classification and standardization. Preprocessing may include, but is not limited to, noise removal, other filtering, downsampling, upsampling, etc. The preprocessed data may be used for feature engineering 149 to provide new features, and it may be further labeled 151 and / or initially labeled (if not already labeled). The labels on the vehicle 100 may take the form of labels relative to a given AI / ML feature (e.g., how the feature requests that the data be labeled), but a given segment of data may be available for more than one reason and may therefore benefit from a more comprehensive set of labels. Once the data is labeled, the data may be used to train any appropriate model 153 to which the labeled data may be applied. This may provide a large data input stream for cloud-based model training and may facilitate global training of models (deployed models or new models) that may benefit from observations from the entire group.

[0045] Any newly trained model may be subject to validation and verification 155 in the cloud 130, which may also be done according to parameters defined by the feature owner. This may reveal whether the newly trained model is functioning according to, for example, the expected output. The performance of the model against each deployed version (which may be vehicle-specific at some point due to online learning) may not be easy to track, but the performance against expected values ​​and globally deployed versions should be considered.

[0046] The model may further undergo hardware optimization 157 and run on a set of optimal virtual hardware 159 in simulation to verify the functionality and output of the model. In at least one example, this optimization may occur before the distribution of the model. If verified and validated by successful simulation, a functional version of the model may be saved in the model repository 161. If necessary, model management may also be provided, which attempts to use ethical AI practices to ensure that the model is not biased, and this may also occur before deployment. The model repository may also include shadow software 163, which allows the model to be executed using real-time and continuous data inputs to provide further verification before deployment, and to ensure that the model does represent an improvement over the deployed version. Feature and model owners can define parameters for improvements and / or new deployments as well as regression and recovery parameters, allowing fine-grained control based on configuration files without the need to monitor the inputs and outputs of a given model and a newly trained version of the model in real time as the model undergoes continuous integration and continuous development (CI / CD).

[0047] The model management process 169 on the cloud can determine when and whether to deploy a new version of the model, such as through the general collection of data and retraining to improve. Cloud-based monitoring 167 and explainability can continue to use real-time data stored in the repository 133 to monitor the expected performance of the deployed model, allowing regressions and unexpected outputs to be identified before the model regresses too much and / or performs far beyond operating parameters. Explainability can provide explanations about why a certain prediction was obtained, what pattern led to the result, the correlation of factors that led to the result, etc.

[0048] Policies and goals 171 may determine which vehicles (or devices, such as phones, wall boxes, etc.) should receive a newly updated model and / or need to roll back / revert to an old model, such as when monitoring shows that a previously deployed version is behaving abnormally. Deployment / rollback 173 delivers the appropriate version of the model to a given vehicle 100 (or other entity), and online model 175 may track and mirror all deployed versions of the model, so that monitoring 167 may faithfully track the performance of the model under a much larger data set than the data that the model may receive from a single entity to which it is deployed. Given the different amounts of data provided to each entity relative to the cloud 130, monitoring occurring under larger data sets may generally reveal defects or regression possibilities more quickly than the vehicle 100.

[0049] With this cohesive pipeline model, improvements are continuously made online (in the vehicle / entity) and in the cloud 130. Individual improvements can be made online in the vehicle 100, making the deployed version of the AI / ML feature 121 specific to that vehicle 100. Overall improvements can be made through federated learning and cloud-based learning and monitoring, allowing improved versions of the model to be deployed globally. Feature owners can observe performance and improvements and redefine profiles to represent more aggressive parameters for improvement, and both the vehicle 100 and the cloud 130 can monitor the deployed model for anomaly detection (again, using feature owner parameters if needed), as well as run shadow versions of the newly trained model against live datasets (both local and cloud-based) to determine if the new model performs better than the deployed version and when the new model is ready for live deployment.

[0050] Figure 2 An illustrative example of the learning process for a deployed model that may occur on a vehicle is shown. The AI / ML feature 201 may represent the currently executing version of a given ML model, which may have a configuration file that defines its own requests and parameters for data collection. The vehicle 100 may provide personalized data based on such requests, thereby collecting any allowed data 203 requested at any interval at which the data is requested. Policy management may dictate how much data is stored for a given model or service, as storage is not infinite, but each model 201 may have a personalized version of the data collected and annotated according to its own parameters, so that a highly specific version of the same general data may be stored for each model's use.

[0051] The automatic annotation / labeling process 205 can be used to label data in real time, allowing the data to be used immediately without having to wait for the manual labeling process to occur. This increases the relevance of the data from the perspective of a given feature 201, resulting in a clean data set 207 that can be stored 209 relative to the given feature 201. This data can be replaced by newer data or retained as needed and / or policy allows. Automatic labeling will be discussed in more detail below.

[0052] If needed, learning as a service (LaaS) 211 can provide online learning without resorting to cloud or edge computing. This allows the vehicle 100 to improve its own model on its own without having to wait for global improvement or deployment. This also allows the vehicle 100 architecture to be leveraged to efficiently train the model using the HPCC and target architecture of the vehicle 100, and distribute the computations used to train the model across a large fleet of deployed vehicles 100, avoiding the need to keep all computing power in a central repository, and allowing the use of existing computing resources residing in a given vehicle 100 or other entity.

[0053] The newly trained model can be validated 213 against a set of feature owner parameters to ensure general functionality and to determine whether the model represents an improved version of the current model 201. Once validated, the model can be moved to a model repository that can track deployed versions and execute the new model as shadow software against live data to ensure performance before deployment. All the while, model monitoring 203 can track the performance of the executing live model 201 to avoid regressions and identify training opportunities.

[0054] This online process allows vehicles to improve their own models using data relevant to both the models and the given vehicle. Since vehicles will observe significantly different information based on location, overall usage, specific types of usage, owner / operator tendencies, environmental changes, etc., the experience of a given vehicle is far less homogeneous than the experience of an application on a desktop computer residing in a fixed location in an office environment, for example. While the usage of any deployed model may vary based on the operator's tendencies and goals, the vehicles themselves represent a continuously changing operating environment and may require many additional changes beyond a typical model to ensure continued improvement and performance under a wider set of variables. This is not to say that vehicle-deployed models will not benefit from global updates, but they will also benefit from various individual improvements in a way that may reflect a more customized environment relative to a given vehicle rather than a simple improvement of a model trained for one type of user (common goals and usage) relative to another type of user.

[0055] The model repository may exist in both the cloud and the vehicle. The cloud version of the repository may store generic and prototype models, which may be ML models that are broadly applicable to fleets, regions, environments, etc. Both the cloud and the vehicle may store dedicated models, which are ML models that are unique to a given user or vehicle / VIN. These are models that may have been trained specifically for that user or vehicle. This may actually represent millions of models in the cloud and may need to be adjusted in a reasonable data management manner.

[0056] The source of the model can be either online (e.g., for a dedicated model) or federated learning (e.g., for a global model). The data stored about a given model may include, for example, but not limited to, a description of the model, algorithm class (e.g., linear regression, decision tree, neural network, etc.), ML framework, contact information of the model / feature owner, version of the training dataset used to implement the model, vehicle dependencies, deployment mode, dependencies, activation triggers, etc.

[0057] The cloud 130 may be configured to continuously deliver updated models, such as when new ML models arrive at a particular branch in the model repository. For a given model, updates may also be scheduled for vehicles that may benefit from an identified version change between a deployed model and a newly trained model. Other delivery triggers may include one-click deployment (instantaneous) or based on vehicle and / or lifecycle events. For example, a valid vehicle state may be required prior to deployment, which may include, but is not limited to, subscription / registration, vehicle authorization, user authorization, presence or absence of certain software or hardware features, etc.

[0058] Model versioning is a process that can be associated with any AI / ML infrastructure. Leveraging the illustrative FNV AI / ML architecture, the concepts of online learning and federated learning can benefit from modified model versioning in the vehicle to track trained, fine-tuned, validated, and updated entities in the vehicle. This can include AUTOSAR-compliant storage structures.

[0059] The model repository 115 component can introduce an in-vehicle model storage structure, for example, using the Autosar persistence structure. The model repository can be designed to store model metadata / configuration (information describing model inputs and outputs and version control information) and models / coefficients (data required to initialize the kernel) in a standardized, ara::persist-based persistence interface.

[0060] Models can be divided into two categories: general purpose or specialized. General purpose models refer to models that are applicable to a large fleet of vehicles (e.g., all vehicles of a certain model or all vehicles in a certain region). Those models can be provided by the cloud and trained in the cloud (even if the training is based on data collected in the vehicle or federated learning) and delivered to the vehicle as part of the normal software / configuration payload. Specialized models are models trained for a specific user or a specific vehicle. They may not be part of the software payload and can be trained in the vehicle or in the cloud. Specialized models are usually derived from a prototype non-specialized model.

[0061] The Ara::persist service can be utilized to access the model under the common storage, and then a coefficient service can be provided on top of the ara::persist service to retrieve and update the model in the private storage. The model and any additional data or information can be updated independently for the common version and the private version.

[0062] According to this paradigm, in a non-limiting illustrative example, model retrieval may proceed as follows. The AI / ML application may request a computation using a generic model identifier, which may be a call to CaaS. CaaS may pass the call to the ara::persist service for the generic model, which will retrieve the generic model from the generic model storage and pass the model back to CaaS. When the computation call includes a specialized version of the model (e.g., online improvements learned for this vehicle), this may result in a call to the coefficient service, which will use the ara::persist service to retrieve the specialized model from the specialized model storage and pass the model back to CaaS.

[0063] In a similar manner, a general non-limiting example of learning to create a specialized version of an existing general model may occur as follows. The AI / ML application may have a CaaS call for learning, which may utilize LaaS. The base model for learning may be obtained from a general model storage device as described above, and CaaS may be executed on the base model in conjunction with LaaS to train the model. The model may then be validated, and LaaS may broadcast the training results. If a new model is to be stored, the Ai / ML application may update the coefficient service in response to broadcasting the training results and request that CaaS utilize the coefficient service to store the new model in the specialized model storage device.

[0064] For example, models related to vehicle functionality, such as accessory angles, may generally be generic in nature (since the angle of an accessory is a "fact" whose nature should not change based on usage or preferences of a given user, i.e., a 30 degree angle is a 30 degree angle everywhere). On the other hand, user-related models, such as preferred window states and vehicle feature recommendations based on user-centric observations or environmental conditions, may be specialized models that, while built from a base model, benefit significantly from specialized training in order to function according to the user's expectations.

[0065] Models may also need to be validated prior to deployment, which can include aggressive testing in a variety of scenarios that help ensure that a given vehicle 100 does not exhibit unexpected behavior under a wide range of circumstances. The model can be deployed to a cloud environment along with the supporting reasoning code before being deployed to a production environment. At that time, test data and validation / threshold data can be used in conjunction with the model to ensure that the expected results / thresholds are achieved based on the execution test data. If the model is a new version of an existing model, differential testing can be used, which can compare the execution differences between versions when the inputs (data) are the same.

[0066] The model can be deployed to a test bench, evaluation board, proxy hardware, etc. for testing with processor-in-the-loop evaluation. The model can also be deployed to target hardware (for the model) using a hardware-in-the-loop simulator to evaluate performance through simulation of the target hardware. The model can then be further evaluated using a test or prototype vehicle to provide vehicle-in-the-loop testing.

[0067] Additionally, as mentioned previously, there may be a shadow mode of the model that can be executed in the target environment without affecting the performance of the currently executing model. This allows predictions to be made relative to both the existing and new versions of the model. Using real-time data, the predicted inputs and outputs for both versions can be monitored and saved, and the resulting data can be used to determine if the new version performs better than the existing version, which can be used as a deployment trigger.

[0068] Testing may also involve CPU / GPU and memory utilization performance, which may include, for example, prediction and task completion times. This may also include determining processor usage and the need for parallel processing, as well as RAM usage, code optimization requirements, cache requirements, and / or communication protocol testing. Vehicles are complex computing environments where countless entities may be using limited resources and where the vehicle itself must often prioritize those resources when performing driving tasks. Therefore, it may be useful to understand the computational and resource burden imposed by the model before deploying it. A limited scope release to a small subset of intended vehicles may also help reveal any overlooked operational issues before large-scale distribution.

[0069] Because models can be updated both specifically for a vehicle and globally, it can be difficult to robustly test all versions of a trained model. While a general purpose model can be extensively tested prior to deployment, a specific version of a model can only be tested on a vehicle. For example, this can include validation and shadow execution against parameters defined by the model owner to ensure compliance, but neither process can continue indefinitely or new models can never be deployed. At some point, typically defined by the feature owner and subject to the constraints of the feature / model owner, the model is deployed. Model monitoring, described in more detail below, continuously monitors the active version of the model to improve performance and identify the need for rollback or recovery.

[0070] Continuous model monitoring can monitor, for example, drifting data (unexpected inputs) and concept drift (unexpected outputs). When initially training a model, it can be trained on real data, but significant assumptions may also be made about the consistency of that data and the fact that the initial training data is actually representative of what a given vehicle will encounter. Even when a model is newly trained to a new version, there is an assumption that the new data that produced the new version will continue to reflect the data that will continue to be observed. Therefore, and especially because of all the variables that exist about the vehicle in terms of location and usage, it is useful to monitor both sides of the data (inputs and outputs) to ensure that a degree of consistency is maintained that reflects what was actually happening when the model was trained.

[0071] Fortunately, the vehicle in the illustrative example is able to monitor itself, rather than relying on cloud-based monitoring, which would require massive cloud computing if tens of millions of active models were to be continuously monitored against real-time data. From the creator's perspective, using feature / model owner-defined configuration data (which can help determine what indicates an actual aberration or training opportunity), the vehicle can proactively monitor its own dedicated models and self-identify when a model would benefit from training or recovery.

[0072] Figure 3 An illustrative example of a continuous monitoring process and flow is shown. An initial AI / ML model may be specified for deployment from the cloud 301, which may include sending a copy of the model to the vehicle 100 telematics control unit (TCU) 303. The model may include the code necessary to execute the model 307, as well as a configuration file that may define, for example, any data necessary to utilize and / or train the model, expected inputs, labeling parameters, expected outputs, training thresholds, recovery thresholds, etc. The TCU deploys the model to a data repository where the vehicle 100 can use the model for its intended purpose.

[0073] When executed, AI / ML model 311 loads the model from a data repository, the discussion above regarding model repositories provides examples of data repositories. Based on a configuration file that defines what data is needed, data collection 313 receives instructions from the configuration file and provides the requested data back to the AI / ML application. The data may also be sent to TCU 303 for redistribution back to the cloud 130. The data can be used for learning in the cloud. The AI / ML application also sends requests to Inference as a Service (IaaS) 315, which helps provide output for the model.

[0074] In this example, continuous monitoring includes both monitoring 300 and interpretability 302. Interpretability extracts feature values ​​317 from the AI ​​model and feature statistics 319 from the configuration file, and compares the current feature values ​​achieved by executing the AI ​​model with the expected values ​​obtained from the AI ​​model 321. For example, these can be the features that contribute most to the predictions made by the model. The model may have expected values ​​for these features, and the configuration file may have expected statistics for these features. The interpretability function 302 can extract the values ​​of the features during model runtime and compare them to the expected values. For example, the feature owner can define thresholds for the features and which features provide the greatest contribution, which can be used in interpretability to determine the impact of one or more features on the results. For example, features can be extracted from the pixel values ​​in an image as a vector of numerical values. The vector is a low-level representation of the image. Certain indexes may be critical to the prediction, and the values ​​at these indexes contribute most to the model prediction. The configuration file can define the relevant contributions.

[0075] This information is used to create an explainability report 323 that identifies which features lead to incorrect predictions or predictions that are outside any acceptable range of deviation. The explainability report can be delivered to both the cloud and the model owner. Metadata associated with the model and stored in the repository can define which entities will receive copies of which reports, and at what frequency and / or under what conditions.

[0076] The monitoring process 300 checks the model type and input / output expectations of the configuration file 325. This check may include verifying that the model type (e.g., regression, classification, etc.) is supported 327 and that the model input (e.g., numerical, image, etc.) is supported 329. Once this is verified, the corresponding model function may be initiated 331 to receive the model input from the data collection. The corresponding processing function may also be initiated 333 to process the incoming data according to the model's function in response to the received input to produce a result (output).

[0077] As the model is receiving inputs and processing outputs, the monitoring process may continuously check the inputs 335 and outputs 337 against expected values ​​obtained from the configuration files. Thus, if the model is producing unexpected outputs, the monitoring process may be able to identify the unexpected input as a possible cause or not a possible cause. Similarly, if the model is experiencing unexpected inputs, the monitoring process may determine whether this is or is not changing the output value outside of parameters. By including both expected results and tolerances / thresholds, the model owner can define what data does or does not constitute unacceptable data, which allows the vehicle to identify opportunities for model improvement or recovery. The results of the analysis performed by the monitoring may be incorporated in a performance report and distributed 339 to the cloud and / or model owner.

[0078] Analysis of the input and inference cycles can be continuous and can include comparing historical distribution data (provided by the cloud) with historical usage data that may be stored on the vehicle. This will reveal whether the locally observed data matches the expected historical distribution over a wider range, and the local data will also be shared with the cloud to update the expected and historical distributions. This may not be "wrong" since the data is actually being observed, but if there is a skew in the data (based on observations) relative to what was expected (based on long-term observations), this may represent an opportunity to train a model with different weights. For example, with regard to something like environmental data, a recent skew may indicate a change in seasons or the vehicle moving to a different climate, which may impact services related to HVAC and other user-centric recommendations as well as proactive system control and feedback.

[0079] If the feature owner provides an updated dataset distribution resulting from training in the cloud, the vehicle can compare the local training set to the updated set. This can reveal opportunities to refresh local training data, collect additional data, or re-weight the collected data. Data drift can help identify vehicle-specific situations or expectations that may be inappropriately set (or no longer applicable). For example, profile "alert rules" and thresholds can define when drift is significant for a given model and / or when to alert the feature owner, when to train the model, and / or when to revert to a previous version of the model.

[0080] Before deploying a model, the model owner can create distribution expectations about the model's inputs, which can be derived from the training set. Similarly, distribution expectations about the outputs can be created, and the expectations and model type can be saved in a configuration file distributed with a given model. Methods (e.g., mathematical, logical, etc.) for calculating expectations can also be provided so that model monitoring can use a copy of those methods.

[0081] For example, a traffic sign classification model may take as input an image of a traffic sign, where the expected output specifies the type of sign (stop, speed limit, street, etc.). Monitoring may track confidence scores from the model, e.g., confidence above 70% is expected. In one example, if a certain number of images in the predicted row have confidence scores below 70%, this may trigger an alert and identify an opportunity to further evaluate the model and / or train the model. If the current model is not satisfactory, the evaluation may at least temporarily lead to regression.

[0082] Figure 4 An illustrative example of an interface analysis of model monitoring is shown. Cloud 130 stores at least a general version of a model or application 401. Data 403 may include, for example, weights, architecture, metadata, and configuration files. An exemplary configuration file 405 may include, for example, model type, input distribution, output distribution, and metrics for evaluation.

[0083] The model 401 is deployed to the vehicle 100 from the cloud via, for example, the vehicle 100 TCU 411. The TCU saves the model 401 to a data repository 413, which may include saving corresponding weights, metadata, architecture, and configuration files, as previously described. The application 415 loads the model from the repository 413, subscribing to the required data as indicated by the configuration file or elsewhere in the model. The model monitoring 300 obtains the expected values ​​by parsing the configuration file 417, and thus obtains the expected input distribution and the expected output distribution.

[0084] Monitoring can also subscribe to input data streams to obtain real-time input data for comparison with expected distributions, and to IaaS 423, which will provide the output of the model based on the real-time input data. This information can be used to compare the actual output distribution with the expected output distribution 421. Based on the comparison, process 300 can pass a report to data collection 425, which can report the performance to relevant parties (e.g., metadata or other parameters required to define the report).

[0085] As briefly discussed above, Inference as a Service (IaaS) is a function that can be included in an illustrative framework. For example, an inference is the result of any computation, statistical calculation, or output of an AI / ML model. IaaS is the publication of inferences to which clients can subscribe. The AI / ML framework provides a set of functions that can enable sharing of inferences or computation outputs across multiple entities under certain conditions according to a service-oriented architecture (SOA).

[0086] For example, the IaaS layer can include a standard and future-agnostic way to publish inferences to the SOA. Functionality to convert computational outputs to computational inputs - convert inferences on the SOA into inputs. Common functionality for running predictions. This strategy can benefit from a common interface structure to publish and identification of which service interfaces to publish inferences in.

[0087] The service element may include fields with, for example, the latest inference value, model metadata, an indication of whether inference is actively published, debugging, and other additional information. For example, when inference is controlled by the client, the service element may also handle methods for activating and deactivating inference.

[0088] The platform may provide common functionality to convert between output / input and SOA-based reasoning. For example, for a common kernel for prediction, the platform may provide common code to extract data from the kernel output and publish the data through SOA. For scenarios where SOA reasoning is used in further computations (e.g., inter-SOC computations), the platform may provide common code that allows reasoning to be used in computations. For example, if a region of interest (ROI) is detected and published on an SOA, it may also be needed to crop the input to another model. The ROI may be cropped around any pixel or group of pixels, and the ROI may be of interest to more than one model. The platform may then provide an SOA subscriber OpenVX kernel that feeds the ROI to the rest of the OpenVX pipeline. This would represent an effective inter-SOC pipeline. Or, for example, when reasoning is fused to provide reasoning with improved predictions or better confidence estimates.

[0089] Generally speaking, the running of inference computations and the publication of the resulting inferences may be determined by various factors. These may include, for example, when the inference flow must be active. This may be determined by various previously set scenarios, such as when a use case is active, the state of a vehicle, repeated requests from features, etc. Trigger conditions may define what specific conditions trigger the separate computation and publication of inferences on the SOA when the use case is active. Input data may define what data should be fed as input into the computation kernel. The type of kernel may define what prediction function is used to obtain a given inference. A given model may define the parameters / coefficients / weights (e.g., trained) that will be used in the prediction function.

[0090] The data agent can support the evaluation of the "when" trigger conditions and input data considerations in the previous examples, as these factors may be similar to the factors that drive data collection. Other example conditions, kernel types, and trained models can be stored in the data repository, as they may be AI / ML-specific for a given model.

[0091] Use cases may include, but are not limited to: fully custom code for collecting and pre-processing data using custom models that can publish the resulting inferences about SOA using standard IaaS structures; using data intents or predefined triggers to run a prediction process and publish the inferences using standard structures; or a request from a feature to initiate a prediction process and publish the inferences in a request-response manner.

[0092] For example, diesel particulate filter monitoring may include diesel particulate filter prediction. The kernel and coefficients of the model may be a black box C code generated in MATLAB. The logic for accumulating data, running preprocessing, and even the model may be highly customized and therefore cannot be relied upon to execute with a standard kernel. However, the results (inference) can still be published using standard structures and services.

[0093] In another example, reverse assist can be a use case where the inputs are well defined, such as nodes in an OpenVX graph and some standard signals. These inputs can be shared with many other models. Once the vehicle is in a specific state and for each frame, reasoning needs to be done, and the kernels can be highly standardized. For scenarios such as these, the architecture can support deployment with minimal or no use case reasoning production code. The platform code can be configured with state conditions, references to image and signal inputs, and the desired model (kernel and coefficients).

[0094] The output of IaaS can be transformed into several forms, one of which is published in the standard SOA format, which allows, for example, the use of custom kernels to convert binary output to SOA format. Another transformation may include converting the IaaS output into kernel input for further calculation. In addition, it may be useful to publish the output of the transformation (from IaaS) as an instance on SOA for multiple clients to subscribe.

[0095] In a general example, IaaS may receive a request from an AI / ML application owner and initiate a computational graph by requesting a computing as a service module. IaaS may then reply to the application owner with an inference flow topic identifier, and may provide a computational graph identifier as confirmation of the request. Based on the received identifier, the application owner may subscribe to the inference topic and request a computational graph to perform any custom preprocessing on the data. The request sent from the application owner to IaaS may include the model required to run, the required input data, and the custom trigger details (if any) based on which the data is subscribed from the data agent. The model may be loaded for computational purposes by a request to a model repository from a persistent memory. Once the input data and model are ready, or based on a trigger, the inference is calculated by IaaS and published to the SOA. For example, the computation may be performed in a loop based on a trigger request time limit, or until a condition is met to stop, for example, after which a memory cleanup may be performed.

[0096] Figure 5 An illustrative example of Inference as a Service of a possible ML application is shown. When the user 503 turns on the ignition at 523, the application 501 may be executed, which signals the vehicle 505. The user may be identified in some way (as this may be a personalization feature in this example), and the id and on signal may be passed to a profile for the IaaS 513, which may be loaded at on-time.

[0097] The configuration file specifies that a trigger signal causes a request for inference at 531, which may also include any relevant data for the request, such as but not limited to ID, model ID, trigger details, etc. The request is passed to IaaS / CaaS517, which (because it will be computing on the model) subscribes to the input data and trigger parameters at 533, so the IaaS will know when the model is triggered and has the relevant data to execute the model. The subscription here is to the data broker 519. IaaS / CaaS 517 also creates inference topics on SOA515 that modules and features can subscribe to, in this example, the inference topics will be the window scrolling 509 and radio station prediction 511 features of the adaptive window butler 507 package.

[0098] IaaS / CaaS also returns a response to the initial request from the configuration file at 537, which may include an identification of which inference topics were created, such that window scrolling 509 may be subscribed to at 539 and radio station predictions 511 may be subscribed to at 541. Feature 507 may also have stated use cases for both window scrolling at 527 and radio station predictions at 529, such that those features are active and may subscribe to the relevant inference topics.

[0099] At 543 , the IaaS / CaaS loads the model for the user, which may be user-specific, or perhaps region-specific or demographic-specific for preference prediction models (window scrolling, radio). The model repository 521 returns the model at 545 .

[0100] Models can be activated based on a trigger cycle to generate inferences that can be used by various features. Once a trigger occurs (temperature, precipitation, time of day, etc.), the model receives input from the data agent 519 at 547 and 549 and executes the model using the data. The relevant data for each model can be (and in this case probably is) different, but for example some weather data may be relevant to both if it affects window position and driver mood, resulting in different radio station selections.

[0101] IaaS / CaaS 517 publishes data about SOA515 to the created topic at 551. Window scrolling accesses its topic data (the topic data it subscribes to) at 553, and radio station prediction accesses its subscribed data at 555. Based on the inference, the output from those features is passed to the housekeeper at 557 and 559, and the housekeeper feature 507 can control the window or radio accordingly. When the vehicle 505 is driving, the housekeeper records events at 561 that may trigger another inference, and the loop can continue. When the user 501 turns off the vehicle 505 at 563, the key turn off is passed to the configuration file at 565, which can instruct the computer to stop at 567 and the data to stop at 569.

[0102] Model monitoring may also take into account reasoning and confidence levels and use, for example, values ​​indicated in a configuration file to determine whether the model is ready for training or needs to be restored.

[0103] Figure 6 An illustrative example of the monitoring process for a newly deployed model is shown. A developer 601 prepares a model that is reviewed and ready to deploy, and deploys the new model by sending it to the vehicle TCU 603 at 615. At the appropriate time, the TCU 603 stores the model in the model repository 613 on the vehicle at 617. The vehicle now has a working, accessible version of the model stored on board.

[0104] When a feature or process calls / loads a model at 621, the model is loaded from repository 613 at 623 and the output from the model is provided to the model monitoring process 611. This allows the vehicle to continuously monitor the model as it is used. The profile associated with the model defines when an input or output is out of bounds or indicates an opportunity for improvement. The monitoring process checks the profile at 625, which was loaded into the monitoring process 613 along with the model.

[0105] The model also subscribes to appropriate data, and thus receives model inputs at 627. When the model uses these inputs according to its current form to produce results that can be used in the vehicle, the monitoring process 613 also checks the input data against expectations from the configuration file. The model further generates outputs requested by features that call the model generated by IaaS 609, which is called by the model in conjunction with CaaS. This output is also received by the monitoring process 613 at 631. That is, when the feature uses the output to perform its function, the monitoring process also receives the output to check the output against expectations included in the configuration file at 633. The model owner can define these expectations and / or redefine these expectations as the model evolves, where the redefinition is achieved by loading new configuration files into the model repository.

[0106] Using the comparison of inputs and outputs against expectations, the model monitoring process 613 creates a performance report at 635 and publishes it to any subscribed entities. A copy of the performance report is received at 637 by the data collection process 605, which sends the report to the cloud using the TCU 603. Model performance may also be received by the application owner 607, who may use the information at 639 to determine, for example, whether a configuration file needs to be updated or whether the model is exhibiting poor performance. Poor performance observations provide an opportunity to decide at 641 to revert to an older version of the model, which may be stored in the model repository 613. The IaaS 609 may also be notified of the reversion at 643 so that it may use the older version of the model.

[0107] Shadow mode can also provide similar utility, running a new version of the model (but not deployed) against current inputs. Monitoring can look at the inputs and outputs sent to and produced by the shadow version of the model by comparing them to expectations and also by determining whether the outputs, for example, represent "better" outputs than those achieved by the current deployed model. If this is the case, the new model is both functional and improved (the monitoring process can also verify that the new model does not produce out-of-bounds or erroneous results, again based at least on a configuration file that defines improvement and error parameters).

[0108] Deploying improved models from the cloud or based on an online (in-car) learning process may include dependency checks on the target environment. Additionally, previous (working) versions of the model may be saved in a model repository in case future inputs result in poor model performance. Observations of poor performance may not be obvious based on the data originally used to review the model, which may result in rolling back to an older version that previously worked.

[0109] When all dependencies have been checked and any certifications have been performed, the deployment process can also determine the vehicle state before deployment, as the vehicle may need to be in certain states (e.g., power mode, key off, sport mode, sleep mode, charging mode, etc.). Customer consent can also be obtained (via the vehicle or an application that interfaces with the vehicle) if necessary. If the new model comes from the cloud, it can first be deployed to a shadow mode of a given vehicle so that the model can be reviewed using live data from the vehicle before live deployment. Where appropriate, the vehicle should make the newly deployed model the current running model, but the model monitoring process will continue to observe the model over time in case recovery or training is needed.

[0110] The models in the illustrative embodiments may be trained by at least two techniques referred to as federated learning and / or online learning. Online learning is the real-time improvement of the model on a given vehicle, where the model is trained over time due to relevant observations and inputs made by the vehicle. This is both an efficient use of computing resources (leveraging the vehicle computer) and allows the model to be trained for a given vehicle. Since vehicles experience vastly different variations in observed and collected data and in nearly all aspects of usage, the initial model may evolve very differently for different vehicles used by different users in different circumstances.

[0111] For example, one user may drive the vehicle lightly in weather that is consistently above 60 degrees Fahrenheit and observes little traffic. Another user may drive the vehicle heavily loaded at hundreds of miles per day in heavy traffic in an environment that changes dramatically with the seasons. The vehicle itself may be specifically designed for one of the use cases (or neither), but the evolution of the model based on the context (who, what, when, where, why, and how) may produce a model that achieves a given goal for each user, which is generated by the context. Any model installed on another vehicle may be inappropriate (from an input and / or output perspective) for the corresponding other vehicle, but may ultimately be highly appropriate for the context of the vehicle in which it is evolved. Thus, online learning allows efficient computational utilization and context-dependent evolution without having to have large-scale cloud computing and large amounts of continuous data transfer and high costs and bandwidth usage as the cloud attempts to continuously evolve 10 million versions of the model based on specific observed situations on a per-vehicle basis.

[0112] While the model may adapt to the vehicle, the online learning process also provides a large source of input for federated learning. That is, as the model evolves, common themes and improvements may emerge that may be useful in many vehicles. Instead of absorbing all the data in the cloud, the evolving model itself can be used to evolve the global model, which can at least be adjusted in a way that appears to represent the common improvements of all vehicles or a larger subset of selected vehicles. Therefore, when a new vehicle receives the global model as a baseline, it has a better version of the model than the one originally designed. Similarly, vehicles that are experiencing slow model evolution or no model evolution (e.g., limited input and use) can receive updated model versions without having to evolve the model on the vehicle. In addition, some vehicles may train models incorrectly or poorly, and may directly benefit from federated learning by being taught (via deployment of an updated global model) one or more things that those vehicles should have learned or may have eventually learned.

[0113] Both online learning and federated learning can be used to update some or all vehicles. Online learning as a global source can be useful for vehicles in similar contexts (e.g., similarly used vehicles operating in similar environments), for example, if a fleet owner simply expands their fleet, doubling the count of each vehicle and intending to have those vehicles perform similar services under similar conditions to their pre-existing counterparts. The online learned model can be distributed to corresponding and related new vehicles, rather than having to wait for all new vehicles to learn what the already existing vehicles have learned.

[0114] As noted, federated learning can be applicable to all vehicles or a subset of vehicles. Thus, federated learning can be used to update a version of a model that also has a context associated with it, e.g., the model is "better" for the vehicle in that context. In other cases, federated learning may only reveal better aspects of a model that is globally applicable and can be used as an improvement to the global model in all cases. If desired, a cloud-based information repository can track every deployed version of every model for every vehicle, including what was changed, when it was changed, new models deployed, etc.

[0115] In one example, for illustration, online learning is categorized into three possible scenarios. Light learning + light data (LLLD), light learning + heavy data (LLHD), heavy learning + heavy data (HLHD). LLLD may include, for example, learning algorithms such as linear / logistic regression or k-NN with small data. LLHD may include use cases that require feature extraction, dimensionality reduction, etc., and may be processed in the acceleration domain, while the actual training / inference may be light. The training work may be the same as LLLD, except that the data will be streamed from the data collection to the acceleration domain. HLHD may include neural network-based methods and some methods such as SVM / random forest / K-NN that run directly in the high-dimensional acceleration domain.

[0116] Figure 7An illustrative example of an online learning process is shown. In this example, an AI / ML application 701 can load a model designated for learning from a model repository 703. Real-time learning 705 can include learning as data is collected, and can be configured for training and status monitoring actions. If the reasoning is wrong, real-time training can be performed. It can call LaaS707, which will use CaaS 709 for training. The result of the training (new model) may need to be verified, so the AI / ML application can call an automatic verification module 711, which uses IaaS 713 and tests the reasoning to determine the validity of the new model. Batch-based learning 715 can work in a similar way, but can use saved data from previous observations to train the model based on batches of data that model monitoring indicates can obtain better results. IaaS can return the reasoning and / or execution model in shadow mode 717, which can involve model monitoring 719 with consideration of the interpretability of the data. The newly adjusted and / or verified model can send its weights to the cloud.

[0117] As the AI / ML model executes, it receives data from data collection 721, and the onboard process 723 automatically labels the data received from the vehicle. Labeling and automatic labeling of data is discussed in more detail below, but generally speaking, data collected by a vehicle is more useful for models and learning when the vehicle has some kind of identification (label) indicating what it represents. This may typically involve a human going through the data and attaching labels, but waiting for that step may make online learning nearly impossible, and therefore, the vehicle is equipped to automatically label the data in some way.

[0118] Training may begin when the AI / ML model determines that training is appropriate, which may vary from model to model and may be defined at least in part by a profile of a given model in terms of appropriateness. For example, training may be performed if the model is performing poorly relative to expectations.

[0119] Training data may be collected from the data collection module in response to a request, and the data may be automatically labeled by the automatic labeling module for use in training. The learning process may invoke LaaS, which uses the labeled training data and creates a graph (training binary) for use by CaaS, which may occur during vehicle rest time, for example, to avoid straining the computing resources used by the vehicle while driving.

[0120] LaaS can monitor the training status using pre-configured thresholds (e.g., accuracy, loss), and can cancel training if the reported accuracy and / or loss exceeds the threshold. The training process can also be saved in the event of a training stop event (such as operation of a vehicle). The newly trained model can be saved for validation.

[0121] After validation, the new model is versioned and stored in the model repository. Additionally, once the model has been validated, but before deployment, model inference can be tested in a shadow mode that mirrors the current model (except with the new model) and uses live data as input. Explainability from this optional step can provide explanations of the model's internal behavior, which can be particularly useful if the model is not performing as expected.

[0122] Figure 8 An illustrative example of an online learning sequence is shown. In this example, a model is newly deployed from a developer 801 to a TCU 803 at 817. At the appropriate time, the TCU 803 stores the model in a model repository 815 at 819. When the model is ready for training, the model is requested at 821 and returned from the model repository at 823. The training process may also request data from the data collection 805 at 825. In response to the request, the collected data may be returned to the training process at 827, and automatic labeling may label the data at 829. If batch-based training occurs, previous data (already labeled) may also be loaded at 831.

[0123] The requirements and configuration of the learning job are passed to LaaS 811 at 833, which passes the requirements to the OpenVX 813 module. OpenVX 813 queues for available compute jobs at 835, and when compute is available at 837, the module starts the compute for training at 839. As training occurs, the state of the model is updated at 843, and the LaaS 811 process monitors the training at 845, as discussed above. LaaS may cancel the training at 847 for any appropriate reason, or the training is completed at 849, and the model is validated at 851. Assuming the model is validated, it may be stored in the model repository at 853.

[0124] LaaS can load and configure kernels based on configurations defined by the application. For example, a neural network-based learning process can provide logistic regression training, configurations for training NNN classifiers (& images, & tags, batches), and Tensorflow-TrainNNNRegressor configurations (& images, & tags, batches). Classical ML learning processes can provide SVM regression training, configurations for training random forest classifiers, and configurations for Markov model training. For example, in the above, training can occur on batches of images with specific labels in batches of specific sizes.

[0125] LaaS implements data accessors for the kernel based on configuration and hyperparameters, and interfaces with data collection to supply data to the selected kernel (directly or through a streaming graph). The dataset can be divided into validation data and training data, and LaaS can interface with the Ai / ML application to receive the computational graph and submit it to CaaS. LaaS can also interface with CaaS to submit a graph with SOA nodes as input, SOA nodes as output, and the supplied training kernel. Messages from LaaS to CaaS may include a set of API calls provided by the OpenVX standard, such as: create node, link node, verify graph, execution graph / streaming graph. Streaming may include repeated execution of the streaming data provided by the input node until the training is completed or canceled.

[0126] As training progresses, LaaS can receive process logs and submit the logs to the Ai / ML application. The training process and kernel processes can be exposed in a SOA-compatible format and exchanged between CaaS and the AI / ML application. Any cancellation request from the AI / ML application can be passed from LaaS to CaaS, and the training process can be stored in case of cancellation or termination.

[0127] Federated learning may occur when one or more online learning updates produce a new valid model. The weights of the new model from each successful online learning session (or a subset of successful online learning sessions) may be shared with the cloud, which may modify the global model based on those online learning sessions. The data used to train each model may be obtained from the respective vehicle.

[0128] Auto-tagging is the process of automatically annotating data (e.g., adding metadata) with minimal or no active human intervention. When a vehicle Ai / ML model is to be trained on a vehicle, the model will typically use real-time or recently collected data. This means that the data cannot go through the traditional process of a human reviewing the data and deciding how to label it. Therefore, it may be useful to have a module on the vehicle that can both label the data and attach labels that make the data useful to a given subscriber. That is, not only can the data be labeled, but the data can be labeled with relevant context, for example, based at least on one or more subscribing AI / ML models, so that the labels present data that is useful to the subscription model used to label the data.

[0129] Several illustrative examples show various non-limiting examples of the types of labeling that may be used in this context. In sensor-derived auto-labeling, sensors annotate the data. For example, an image may have an acceleration tag associated with it, and a sensor may provide inertial measurements to help label the data. In human-derived auto-labeling, human input may be used in conjunction with data labeling to directly or indirectly label the data. Indirect auto-labeling includes annotations that are indirectly accessed based on human interaction, and direct auto-labeling includes annotations that are directly accessed based on intentional interaction with humans. Algorithm-derived auto-labeling uses an algorithm to label the data, and may use synthetic data where the annotations are generated via a generative CL algorithm (such as a GAN).

[0130] Automatic labeling can be used in several contexts, including but not limited to: augmenting cloud workflows, where accurate (manual) labeling is required, automatically labeling the collected data for later manual labeling can improve throughput and initial classification; in online learning, data that does not require manual labeling can be collected and labeled for use in online learning without manual labeling; and identifying datasets into user clusters, for example, when data that requires hyper-local knowledge is needed, such as lane centering to adjust the vehicle to the corner of the lane when passing a large vehicle, automatic labeling can label data showing the position of the vehicle relative to the lane markings. This can help determine group preferences (position relative to the lane in the group) and individual preferences (position within the lane that a given driver prefers to drive). More driver-personalized data can include, for example, the steering wheel angle achieved by the driver when the driver is driving within the lane.

[0131] The mechanism for automatic labeling may fall within the purview of the AI / ML application owner, as automatic labeling may be specific to a given use. That is, the configuration file of the AI / ML application may define how certain data should be labeled. The following is an example of this.

[0132] The AI / ML application owner can monitor the performance of the application based on the key performance indicators (KPIs) provided by model monitoring, and implement logic to determine whether automatic labeling is required based on the KPIs. The AI / ML application owner can identify the data required for dataset creation (e.g., subscribing to various vehicle data, timing, range, etc.). The owner can then implement the logic required to create the dataset, that is, the logic to assign labels to the data. Data collection saves the generated dataset to persistent storage, and the AI / ML application owner can configure data collection to perform this step and provide some dataset cleaning utilities to remove auxiliary data that is not required for a specific model. Based on the input data, the AI / ML application owner generates a dataset and publishes it, data collection reads the dataset and saves it to persistent storage, and data collection can perform data cleaning on the saved data. For example, the process may continue until a trigger to stop is provided by the AI / ML application owner or the vehicle platform.

[0133] An illustrative auto-tagging sequence may consist of three parts: start, loop, and trigger / stop. In this example, the initiation is a decision of the AI / ML application owner and includes a request for data collection, which configures the data collection to set up publishers and subscribers on the data collection and data architecture. The AI / ML feature owner can then subscribe to the input data and process it to generate a dataset. The AI / ML application can then publish the dataset and the data collection can subscribe to it. The subscribed data can then be stored to persistent storage according to the above configuration. At some point, the trigger may indicate to stop the data collection, and this will shut down the publisher and subscriber. The data collection can then provide a brief summary of the collected data, such as size, samples, etc.

[0134] In one example, synthetic data can be used via generative machine learning to automatically label accessory angle data (derived from images of accessories). The input to the labeling process can include images of accessories collected in real time, and reference images with known accessory angles. The GAN process produces a rendered image, where the input image is rendered in the form of a reference image, and the accessory angles of the rendered input image. This data, as a rendered image, is now labeled with reference angles and can be used for training, online learning, analysis, etc., but no one has to measure the angles or think about the image.

[0135] In another example, an adaptive butler process may learn whether to mark a suggestion as valid or invalid based on a human's reaction to a "suggestion". In this case, the inputs may be a person's ID provided by the AI / ML owner, location (GPS) provided by a CAN signal, a timestamp provided by the data architecture, applicable time zone provided by the AI / ML owner, a GPS track showing historical location data provided by CAN, an event that is a voice command interacting with a feature, and vehicle speed provided by CAN. The result is a suggested window position that attempts to put the window in the correct position that the person needs at that time in these circumstances.

[0136] Applications can use model monitoring to monitor the performance of deployed models. Model monitoring provides the following metrics to the application: True Positives (TP), False Positives (FP), True Negatives (TN), and False Negatives (FN). Misclassification refers to the incidence of false positives and false negatives. Once misclassifications are identified, the signal data is collected using the FNV4 data collection script. The resulting data is labeled with whether the signal (window position) is accurate, which in one example can be considered to be the case where a human kept the window unchanged or manually changed the window after the suggested positioning. This is an example of indirect labeling. For example, a direct labeling could be asking the human if the position is correct. False negatives and false positives may indicate opportunities for training the model, in addition to other training reasons.

[0137] In addition to online learning, vehicles may also benefit from being able to validate newly proposed or improved models. Just because a model is trained does not mean it will produce a smarter model. The vehicle can validate the model before deploying it as a functional version of the model, which avoids having to send the model to the cloud for validation (even though the cloud may later try to validate the model using a larger dataset from many vehicles).

[0138] Shadow mode is useful for validation because it can benchmark both the previous / current model and the newly tuned model using the same live data. That is, without deploying the model, shadow mode can effectively run both versions on live data and determine which model performs better. The AI / ML application owner can define what constitutes an improved model in the configuration file (metrics) and provide instructions on whether to append training data to the validation set.

[0139] Fig. 9An illustrative model validation sequence is shown. As in the previous example, the new model can be deployed from the cloud 901 to the TCU 903 at 917 and eventually saved to the model repository 915 at 919. When the shadow mode attempts to validate the newly trained model version, the application owner can request both the original model and the adjusted model at 921. This may include a subscription to the online learning module 911, as the newly trained model may not be saved in the repository until after validation. But in the case where the owner 907 subscribes to the online learning of the model, it will be able to receive the adjusted version of the model when training is complete at 923.

[0140] The owner may then request both training data and / or valid data from data collection 905 at 925, which returns valid data at 927. The data and two versions of the model may be sent to validation module 913 at 929, which may benchmark the two models using the provided data at 931. The validation results are sent back to the owner at 933, where the owner may specify to discard the new model at 935 or store and deploy the new model at 937 based on the KPIs indicated in the configuration file. The validation data may include owner-defined validation data and randomly sampled validation data. The model validation process may further randomly sample available data and update a second validation data set at 939, and send this information back to the model owner at 941.

[0141] Some models may be validated on-board, while others may not. For example, information models (such as window position prediction) may be suitable for on-board validation. On the other hand, certain security / privacy related features may be validated offline (in the cloud).

[0142] Although exemplary embodiments are described above, these embodiments are not intended to describe all possible forms covered by the claims. The words used in the specification are descriptive words rather than restrictive words, and it should be understood that various changes can be made without departing from the spirit and scope of the present disclosure. As previously described, the features of various embodiments can be combined to form other embodiments of the present invention that may not be explicitly described or shown. Although various embodiments may have been described as providing advantages or being superior to other embodiments or prior art implementations in terms of one or more desired characteristics, it will be appreciated by those of ordinary skill in the art that one or more features or characteristics may be compromised to achieve the desired overall system properties, depending on the specific application and implementation. These properties may include, but are not limited to, strength, durability, marketability, appearance, packaging, size, maintainability, weight, manufacturability, ease of assembly, etc. For this reason, embodiments described as less desirable than other embodiments or prior art implementations in terms of one or more characteristics are within the scope of the present disclosure and may be desirable for specific applications.

[0143] According to the present invention, a system is provided, which has: one or more processors of a vehicle, the one or more processors being configured to: receive an identification of data to be collected via a request configured based on a configuration file associated with a machine learning model stored by the vehicle; receive an identification of how to tag the data as defined by the configuration file; create one or more topics for publishing the data on the vehicle, the publication including both of the collected data and any metadata that can be used to tag the data according to the definition in the configuration file; subscribe to the topic to receive the published data and the metadata; use the metadata to append tags to the data to tag the data according to the definition of the relevant tags in the configuration file; and save the labeled data in the vehicle memory as data associated with the model.

[0144] According to an embodiment, the metadata includes vehicle sensor data published to the topic and data specified by the configuration file to be tagged based on one or more sensor values ​​present when the data was collected.

[0145] According to an embodiment, the request comprises a timing for collecting the data, the timing being defined by the configuration file.

[0146] According to an embodiment, at least one of the one or more processors is further configured to remove data from the collected data that does not correspond to a relevant area in the collected data, the relevant area being defined based on the configuration file.

[0147] According to an embodiment, the metadata includes an identification of a positive response or a negative response of a human occupant in response to a vehicle state change action generated based on the model, and the positive response or negative response indicates whether the vehicle state change is accepted or modified, respectively.

[0148] According to an embodiment, the metadata comprises responses to queries directly asked of a vehicle occupant via a vehicle display, the queries being defined by the configuration file.

[0149] According to an embodiment, the configuration file defines an algorithm that can be used to modify the collected data to produce a modified version of the data, and determine a characteristic of the data for marking the data based on the modified version of the data.

[0150] According to the present invention, a method includes: receiving an identification of data to be collected via a request configured based on a configuration file associated with a machine learning model stored by the vehicle; receiving an identification of how to tag the data as defined by the configuration file; creating one or more topics for publishing the data on the vehicle, the publication including both of the collected data and any metadata that can be used to tag the data according to the definition in the configuration file; subscribing to the topic to receive the published data and the metadata; using the metadata to append tags to the data to tag the data according to the definition of the relevant tags in the configuration file; and saving the tagged data as data associated with the model in the vehicle memory.

[0151] In one aspect of the invention, the metadata includes vehicle sensor data published to the topic and data specified by the configuration file to be tagged based on one or more sensor values ​​present when the data was collected.

[0152] In one aspect of the invention, the request includes a timing for collecting the data, the timing being defined by the configuration file.

[0153] In one aspect of the invention, the method comprises removing from the collected data data that does not correspond to a relevant area in the collected data, the relevant area being defined based on the configuration file.

[0154] In one aspect of the invention, the metadata includes an identification of a positive response or a negative response by a human occupant in response to a vehicle state change action generated based on the model, and the positive response or negative response indicates whether the vehicle state change is accepted or modified, respectively.

[0155] In one aspect of the invention, the metadata includes responses to queries posed directly to a vehicle occupant via a vehicle display, the queries being defined by the configuration file.

[0156] In one aspect of the invention, the configuration file defines an algorithm that can be used to modify the collected data to produce a modified version of the data, and determine characteristics of the data for marking the data based on the modified version of the data.

[0157] According to the present invention, a non-temporary computer-readable storage medium stores instructions, which, when executed by one or more processors, cause the one or more processors to perform a method, the method comprising: receiving an identification of data to be collected via a request configured based on a configuration file associated with a machine learning model stored by the vehicle; receiving an identification of how to label the data as defined by the configuration file; creating one or more topics on the vehicle for publishing the data, the publication including both the collected data and any metadata that can be used to label the data according to the definition in the configuration file; subscribing to the topic to receive the published data and the metadata; using the metadata to attach tags to the data to label the data according to the definition for labeling in the configuration file; and saving the labeled data in the vehicle memory as data associated with the model.

[0158] In one aspect of the invention, the metadata includes vehicle sensor data published to the topic and data specified by the configuration file to be tagged based on one or more sensor values ​​present when the data was collected.

[0159] In one aspect of the invention, the request includes a timing for collecting the data, the timing being defined by the configuration file.

[0160] In one aspect of the invention, the method comprises removing from the collected data data that does not correspond to a relevant area in the collected data, the relevant area being defined based on the configuration file.

[0161] In one aspect of the invention, the metadata includes at least one of: an identification of a positive response or a negative response of a human occupant to a vehicle state change action generated based on the model, and the positive response or the negative response respectively indicates whether the vehicle state change is accepted or modified; or a response to a query directly asked of the vehicle occupant via a vehicle display, the query being defined by the configuration file.

[0162] In one aspect of the invention, the configuration file defines an algorithm that can be used to modify the collected data to produce a modified version of the data, and determine characteristics of the data for marking the data based on the modified version of the data.

Claims

1. A system comprising: One or more processors of a vehicle, the one or more processors being configured to: receiving, via a request configured based on a configuration file associated with a machine learning model stored by the vehicle, an identification of data to be collected; Receiving an identification of how to mark the data as defined by the configuration file; creating one or more topics for publishing the data on the vehicle, the publication including both: the collected data and any metadata that can be used to tag the data according to the definition in the configuration file; Subscribing to the topic to receive the published data and the metadata; appending tags to the data using the metadata to tag the data according to the definition of the relevant tags in the configuration file; as well as The labeled data is stored in a vehicle memory as data associated with the model.

2. The system of claim 1, wherein the metadata includes vehicle sensor data published to the topic and data specified by the configuration file to be tagged based on one or more sensor values ​​present when the data was collected.

3. The system of claim 1, wherein the request includes a timing for collecting the data, the timing being defined by the configuration file.

4. The system of claim 1, wherein at least one of the one or more processors is further configured to remove data from the collected data that does not correspond to a relevant region in the collected data, the relevant region being defined based on the configuration file.

5. The system of claim 1 , wherein the metadata includes an identification of a positive response or a negative response by a human occupant in response to a vehicle state change action generated based on the model, and the positive response or the negative response indicates whether the vehicle state change is accepted or modified, respectively.

6. The system of claim 1, wherein the metadata includes responses to queries posed directly to a vehicle occupant via a vehicle display, the queries being defined by the configuration file.

7. The system of claim 1, wherein the configuration file defines an algorithm that can be used to modify the collected data to produce a modified version of the data, and determine characteristics of the data for marking the data based on the modified version of the data.

8. A method comprising: receiving, via a request configured based on a configuration file associated with a machine learning model stored by the vehicle, an identification of data to be collected; Receiving an identification of how to mark the data as defined by the configuration file; creating one or more topics for publishing the data on the vehicle, the publication including both: the collected data and any metadata that can be used to tag the data according to the definition in the configuration file; Subscribing to the topic to receive the published data and the metadata; appending tags to the data using the metadata to tag the data according to the definition of the relevant tags in the configuration file; as well as The labeled data is stored in a vehicle memory as data associated with the model.

9. The method of claim 8, wherein the metadata includes vehicle sensor data published to the topic and data specified by the configuration file to be tagged based on one or more sensor values ​​present when the data was collected.

10. The method of claim 8, wherein the request includes a timing for collecting the data, the timing being defined by the configuration file.

11. The method of claim 8, further comprising removing data from the collected data that does not correspond to a relevant area in the collected data, the relevant area being defined based on the configuration file.

12. The method of claim 8, wherein the metadata includes an identification of a positive response or a negative response by a human occupant in response to a vehicle state change action generated based on the model, and the positive response or the negative response indicates whether the vehicle state change is accepted or modified, respectively.

13. The method of claim 8, wherein the metadata includes responses to queries posed directly to a vehicle occupant via a vehicle display, the queries being defined by the configuration file.

14. The method of claim 8, wherein the configuration file defines an algorithm that can be used to modify the collected data to produce a modified version of the data, and determine characteristics of the data for marking the data based on the modified version of the data.

15. A non-transitory computer-readable storage medium storing instructions that, when executed by one or more processors, cause the one or more processors to perform a method comprising: receiving, via a request configured based on a configuration file associated with a machine learning model stored by the vehicle, an identification of data to be collected; Receiving an identification of how to mark the data as defined by the configuration file; creating one or more topics for publishing the data on the vehicle, the publication including both: the collected data and any metadata that can be used to tag the data according to the definition in the configuration file; Subscribing to the topic to receive the published data and the metadata; appending tags to the data using the metadata to tag the data according to the definition of the relevant tags in the configuration file; as well as The labeled data is stored in a vehicle memory as data associated with the model.

Images