Continuous update of driving system for incident avoidance

Abstract

To automatically generate a compliance requirement of a vehicle application from information collected from an Internet.SOLUTION: Continuous update of a driving system for incident avoidance is performed by the steps of: collecting a plurality of incident samples from an Internet and identifying the plurality of incident samples by an identification machine-learning model to involve one or more vehicles; clustering, by a clustering machine- learning model, the plurality of incident samples into a plurality of incident clusters; and defining, by a requirement defining machine- learning model, a vehicle application compliance requirement according to an incident cluster among the plurality of incident clusters.SELECTED DRAWING: Figure 6

Description

[Background technology]

[0001] Requirements as Code (RaC) is used to define the requirements for a vehicle application. RaC encompasses various types of information, including requirements that define the characteristics and behavior of the vehicle application, the metrics and criteria used to verify that the requirements are met, the conditions under which these metrics and criteria are evaluated, and the data or test scenarios used for such evaluation.

[0002] Creating a RaC file requires consideration of the functional and non-functional requirements of the vehicle application and applicable vehicle, as well as the vehicle application specification, including features that are valuable to users, edge cases to consider during testing, and issues identified for tracking with regression testing.

[0003] The RaC files are used to test vehicle applications and the machine learning models they employ to determine whether requirements are met and to determine whether test criteria are met. [Brief explanation of the drawings]

[0004] [Figure 1] FIG. 1 is a schematic diagram of a system for continuous updating of a driving system for incident avoidance, according to at least some embodiments of the present disclosure. [Figure 2] FIG. 2 is a schematic diagram of a generator according to at least some embodiments of the present disclosure. [Figure 3] FIG. 3 is a schematic diagram of a model updater in accordance with at least some embodiments of the present disclosure. [Figure 4] FIG. 4 is a schematic diagram of a vehicle in accordance with at least some embodiments of the present disclosure. [Figure 5] FIG. 5 is an operational flow for continuous updating of a driving system for incident avoidance, according to at least some embodiments of the present disclosure. [Figure 6] FIG. 6 is an operational flow for requirements generation according to at least some embodiments of the present disclosure. [Figure 7] FIG. 7 is an operational flow for model training according to at least some embodiments of the present disclosure. [Figure 8] FIG. 8 is an operational flow for vehicle operation according to at least some embodiments of the present disclosure. [Figure 9] FIG. 9 is a schematic diagram of a requirements file in accordance with at least some embodiments of the present disclosure. [Figure 10] FIG. 10 is a block diagram of a hardware configuration for continuous updating of a driving system for incident avoidance, according to at least some embodiments of the present disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0005] Aspects of the present disclosure are best understood from the following detailed description when read in conjunction with the accompanying drawings. It should be noted that, according to standard practice in the industry, various features have not been drawn to scale. In fact, the dimensions of various features may be arbitrarily increased or decreased for clarity of discussion.

[0006] The following disclosure provides various embodiments or examples for implementing various features of the provided subject matter. Specific examples of components, values, operations, materials, arrangements, or equivalents thereof are described below to simplify the disclosure. It should be understood that these are merely examples and are not intended to be limiting. Other components, values, operations, materials, arrangements, or equivalents thereof are contemplated. In addition, the disclosure may repeat reference numerals and / or letters in various examples. This repetition is for simplicity and clarity and does not, in itself, dictate a relationship between the various embodiments and / or configurations discussed.

[0007] Vehicle applications and their machine learning models are not always updated in a timely manner to resolve new issues, which may be in the form of challenges, weaknesses, or edge cases, because updating requires notifying developers of new problems and requiring them to manually write requirements and design tests accordingly to resolve each issue.

[0008] In at least some embodiments of the present disclosure, to update the vehicle application and its machine learning model, incident samples are collected from the internet using a discriminative machine learning model, clustered by a clustering machine learning model, and used as a basis for defining compliance requirements for the vehicle application by a requirements definition machine learning model. In at least some embodiments, the requirements definition machine learning model defines rules for annotating sensor samples that are used as training samples.

[0009] In at least some embodiments, incident samples are collected from the internet using a collection machine learning (ML) model. In at least some embodiments, the incident samples are clustered using a clustering ML model. In at least some embodiments, requirements for the incident clusters are defined using a requirements definition ML model.

[0010] By automatically generating requirements from information collected from the internet, at least some embodiments continuously update the vehicle application and its machine learning model for incident prevention without developer notification or manual input. By automatically defining annotation rules according to the requirements, at least some embodiments automatically prepare training samples.

[0011] 1 is a schematic diagram of a system for continuous updating of a driving system for incident avoidance, according to at least some embodiments of the present disclosure. The system includes a server 100, an internet 109, and a vehicle 140.

[0012] Server 100 communicates with Internet 109 and vehicles 140 and includes generator 110 and model updater 120. In at least some embodiments, server 100 hosts machine learning models and processes data for continuous updates of driving systems for incident avoidance. In at least some embodiments, server 100 is configured to communicate with vehicles 140 to exchange data and updates. In at least some embodiments, server 100 is configured to perform general server tasks such as data storage and network management. In at least some embodiments, server 100 is configured to connect to Internet 109 for data collection and distribution. In at least some embodiments, server 100 comprises multiple physical servers and computing resources. In at least some embodiments, server 100 is a physical server in a data center or a virtual server in the cloud. In at least some embodiments, server 100 is a type of server used in many fields, from web hosting to database management.

[0013] Generator 110 retrieves incident samples 111 from the internet 109 and sends requirements files, such as requirements file 123, and annotation rules, such as annotation rules 124, to model updater 120. In at least some embodiments, generator 110 is configured to generate requirements files and annotation rules from incident samples. In at least some embodiments, generator 110 is configured to generate many types of requirements files based on different types of vehicle applications and their machine learning models. In at least some embodiments, generator 110 is a software module running on server 100. In at least some embodiments, generator 110 is one of many servers that comprise server 100.

[0014] Model updater 120 receives requirements files, such as requirements file 123, and annotation rules, such as annotation rules 124, from generator 110, sensor samples, such as sensor samples 126, and application logs, such as application log 142, from vehicle 140, and transmits vehicle application models, such as vehicle application machine learning model 130, to vehicle 140. In at least some embodiments, model updater 120 is configured to update vehicle application machine learning model 130 based on provided training samples and to test vehicle application machine learning model 130 according to vehicle application compliance requirements, such as the vehicle application compliance requirements in requirements file 123. In at least some embodiments, model updater 120 is configured to update other types of machine learning models. In at least some embodiments, model updater 120 is a software module operating on server 100. In at least some embodiments, model updater 120 is one of many servers that comprise server 100.

[0015] Vehicle 140 receives vehicle application models, such as vehicle application machine learning model 130, from model updater 120 and sends sensor samples, such as sensor samples 126, and application logs, such as application log 142, to model updater 120. In at least some embodiments, vehicle 140 is configured to deploy updated versions of vehicle application machine learning model 130 and generate application log 142 based on the output of vehicle application machine learning model 130. In at least some embodiments, vehicle 140 is configured to perform normal vehicle functions, such as transportation. In at least some embodiments, vehicle 140 is configured to interact with the physical world through sensors and actuators. In at least some embodiments, vehicle 140 is any vehicle equipped with a compatible system, such as a car, truck, boat, aircraft, submarine, etc.

[0016] The Internet 109 communicates with the server 100. In at least some embodiments, the Internet 109 is configured to provide a source of incident samples, such as incident samples 111, for the system. In at least some embodiments, the Internet 109 is configured to communicate with and exchange data with the server 100. In at least some embodiments, the Internet 109 is configured to provide a network for various other applications. In at least some embodiments, the Internet 109 is configured to connect to various systems and devices around the world. In at least some embodiments, the Internet 109 is a network of global networks. In at least some embodiments, the Internet 109 is used for a wide range of applications, from communications to entertainment.

[0017] 2 is a schematic diagram of a generator according to at least some embodiments of the present disclosure. Generator 210 includes incident collector 212, incident database 214, incident clusterer 215, incident cluster database 217, and requirements definition model 219. Generator 210 is substantially similar in structure and function to generator 110 of FIG. 1, unless otherwise noted.

[0018] Incident collector 212 includes discriminative model 213. In at least some embodiments, incident collector 212 identifies and retrieves incident samples, such as incident sample 211, and stores the incident samples in incident database 214. In at least some embodiments, incident collector 212 is configured to collect incident samples from the internet. In at least some embodiments, incident samples, such as incident sample 211, are natural language text samples that include descriptions of incidents related to one or more vehicles. In at least some embodiments, incident samples, such as incident sample 211, include images. In at least some embodiments, incident samples, such as incident sample 211, include a combination of natural language and image data. In at least some embodiments, incident collector 212 is configured to remove personally identifiable information from incident samples using techniques such as filtering, generalization, obfuscation, etc. In at least some embodiments, incident collector 212 is configured to replace specific vehicle identification information in incident samples with generic descriptions such as “large trailer,” “tank truck,” etc., as needed. In at least some embodiments, incident collector 212 is configured to store collected incident samples for further processing. In at least some embodiments, incident collector 212 is configured to interact with the internet to collect incident samples, hi at least some embodiments, incident collector 212 is a real-world web crawler or data scraping tool.

[0019] The discrimination model 213 identifies incident samples for the incident collector 212. In at least some embodiments, the discrimination model 213 is configured to identify incident samples associated with one or more vehicles. In at least some embodiments, the discrimination model 213 is configured to process incident samples collected by the incident collector 212. In at least some embodiments, the discrimination model 213 is configured to distinguish incident samples from other types of data samples. In at least some embodiments, the discrimination model 213 is a machine learning model trained to perform a task. In at least some embodiments, the discrimination model 213 is a large-scale linguistic machine learning model trained for natural language processing.

[0020] The incident database 214 receives incident samples from the generator 210 and sends the incident samples to the incident clusterer 215. In at least some embodiments, the incident database 214 is configured to store incident samples collected from the internet. In at least some embodiments, the incident database 214 is configured to store other types of data. In at least some embodiments, the incident database 214 is a file system, a relational database, a NoSQL database, etc. In at least some embodiments, the incident database 214 is a type of database used in many fields, such as data analysis and web development.

[0021] The incident clusterer 215 includes a clustering model 216. In at least some embodiments, the incident clusterer 215 is configured to receive incident samples from the incident database 214 and send the incident clusters to the incident cluster database 217. In at least some embodiments, the incident clusterer 215 is configured to cluster the incident samples into incident clusters. In at least some embodiments, the incident clusterer 215 is configured to cluster many types of data.

[0022] The clustering model 216 is used by the incident clusterer 215. In at least some embodiments, the clustering model 216 is used by the incident clusterer 215 to cluster incident samples. In at least some embodiments, the clustering model 216 is a machine learning model trained to perform clustering tasks for vehicle incidents. In at least some embodiments, the clustering model 216 is a type of model used in many fields, such as data mining and market segmentation.

[0023] Incident cluster database 217 receives incident clusters from incident clusterer 215 and sends the incident clusters to requirements definition model 219. In at least some embodiments, incident cluster database 217 is configured to store the incident clusters generated by incident clusterer 215. In at least some embodiments, incident cluster database 217 is configured to store other types of data. In at least some embodiments, incident cluster database 217 is a file system, a relational database, a NoSQL database, etc. In at least some embodiments, incident cluster database 217 is a type of database used in many fields, such as data analysis and web development.

[0024] Requirements definition model 219 receives incident clusters from incident cluster database 217, vehicle information 221, and application logs from application log database 226, and sends compliance requirements for vehicle applications to requirements database 222. In at least some embodiments, requirements definition model 219 is configured to define compliance requirements for vehicle applications, such as compliance requirements for vehicle applications in requirements file 223, according to the incident clusters and the vehicle information. In at least some embodiments, requirements definition model 219 is configured to define compliance requirements for vehicle applications, such as compliance requirements for vehicle applications in requirements file 223, according to the incident clusters, the vehicle information, and the application logs. In at least some embodiments, requirements definition model 219 is configured to process incident clusters stored in incident cluster database 217. In at least some embodiments, requirements definition model 219 is configured to define vehicle application compliance requirements for many types of vehicle applications. In at least some embodiments, requirements definition model 219 is a machine learning model trained to perform RaC-defined tasks. In at least some embodiments, requirements definition model 219 is configured to define rules, such as annotation rules 224, for annotating sensor samples to be used as training samples.

[0025] Vehicle information 221 is utilized by requirements definition model 219. In at least some embodiments, vehicle information 221 is utilized by requirements definition model 219 to tailor the compliance requirements of a vehicle application to a specific vehicle or type of vehicle. In at least some embodiments, vehicle information 221 includes a vehicle application specification, a vehicle application design document, a vehicle application source code, or any combination thereof. In at least some embodiments, vehicle information 221 is configured to provide information about the vehicle on which the compliance requirements of the vehicle application are deployed. In at least some embodiments, vehicle information 221 is configured to provide information about many types of vehicles. In at least some embodiments, vehicle information 221 is a database or file that includes vehicle specifications in real-world form. In at least some embodiments, vehicle information is a type of information used in many fields, such as automotive engineering and automotive manufacturing.

[0026] Requirements database 222 receives vehicle application compliance requirements from requirements definition model 219. In at least some embodiments, requirements database 222 is configured to store vehicle application compliance requirements defined by requirements definition model 219. In at least some embodiments, requirements database 222 is configured to provide these requirements to other components for processing. In at least some embodiments, requirements database 222 is configured to store vehicle application compliance requirements for many types of vehicle applications. In at least some embodiments, requirements database 222 is a file system, a relational database, a NoSQL database, etc.

[0027] Requirements file 223 is generated by generator 210. In at least some embodiments, requirements file 223 is configured to include compliance requirements for the vehicle application in a computer-readable format. In at least some embodiments, requirements file 223 is used by other components to understand the compliance requirements for the vehicle application. In at least some embodiments, requirements file 223 is a text file, a JSON file, an XML file, etc. In at least some embodiments, requirements file 223 is a RaC file, such as those used in software development and project management.

[0028] The annotation rules 224 are generated by the generator 210. In at least some embodiments, the annotation rules 224 are used to label training samples. In at least some embodiments, the annotation rules 224 are a set of rules defined in a programming language.

[0029] Application log database 226 communicates with requirements definition model 219. In at least some embodiments, application log database 226 is configured to store output application logs of the vehicle application machine learning model. In at least some embodiments, application log database 226 is configured to receive application logs from the vehicle. In at least some embodiments, application log database 226 is configured to provide application logs to requirements definition model 219. In at least some embodiments, application log database 226 is configured to store other types of logs. In at least some embodiments, application log database 226 is configured to interact with other components and provide application logs to other components. In at least some embodiments, application log database 226 is a file system, a relational database, a NoSQL database, etc.

[0030] 3 is a schematic diagram of a model updater according to at least some embodiments of the present disclosure. Model updater 320 includes a sample labeler 328, a vehicle application model 330, training samples 331, a trainer 332, a tester 333, and a deployer 335. Model updater 320 is substantially similar in structure and function to model updater 120 of FIG. 1, unless otherwise noted. Each of requirements database 322, requirements file 323, and annotation rules 324 is substantially similar in structure and function to requirements database 222, requirements file 223, and annotation rules 224 of FIG. 2, respectively, unless otherwise noted.

[0031] Sensor samples 326 are samples of data collected from sensors. In at least some embodiments, sensor samples 326 are samples of data collected from sensors on a vehicle. In at least some embodiments, sensor samples 326 are labeled by sample labeler 328 into training samples. In at least some embodiments, sensor samples 326 have the form of values, strings of characters, images, videos, audio clips, or other digital formats generated by sensors.

[0032] Sensor sample database 327 stores and provides sensor samples, such as sensor sample 326. In at least some embodiments, sensor sample database 327 is configured to store any type of data sample. In at least some embodiments, sensor sample database 327 provides sensor samples to other components.

[0033] Sample labeler 328 receives sensor samples from sensor sample database 327, receives annotation rules 324, and provides training samples to trainer 332 and tester 333. In at least some embodiments, sample labeler 328 applies annotation rules 324 to the sensor samples and provides labeled samples, such as training samples 331A and 331B, to trainer 332 and tester 333, respectively. In at least some embodiments, sample labeler 328 is configured to label sensor samples according to annotation rules 324. In at least some embodiments, sample labeler 328 is configured to label any type of data sample. In at least some embodiments, sample labeler 328 determines which training samples are used for training and which are used for testing. In at least some embodiments, sample labeler 328 is a software module within model updater 320. In at least some embodiments, sample labeler 328 is a type of labeler used in any system employing supervised machine learning.

[0034] Vehicle application machine learning model 330 is a machine learning model for a vehicle application. In at least some embodiments, vehicle application machine learning model 330 is trained or updated using training samples, such as training sample 331A. In at least some embodiments, vehicle application machine learning model 330 is one of many types of autonomous driving models, such as an image classification model. In at least some embodiments, vehicle application machine learning model 330 is a data structure that encapsulates parameters of the machine learning model.

[0035] Trainer 332 receives training samples, such as training sample 331A, from sample labeler 328 and trains vehicle application machine learning model 330. In at least some embodiments, trainer 332 is configured to train the vehicle application machine learning model using a portion of the training samples. In at least some embodiments, trainer 332 is configured to update vehicle application machine learning model 330. In at least some embodiments, trainer 332 is not limited to training vehicle application machine learning models. In at least some embodiments, trainer 332 is configured to train any machine learning model using supervised learning. In at least some embodiments, trainer 332 stores iterations of vehicle application model 330 during training. In at least some embodiments, trainer 332 is a software module within model updater 320.

[0036] Tester 333 is configured to receive test samples, such as training samples 331B, from sample labeler 328. In at least some embodiments, tester 333 is configured to apply vehicle application machine learning model 330 to the training samples. In at least some embodiments, tester 333 is configured to test the vehicle application machine learning model using a portion of the training samples. In at least some embodiments, tester 333 is not limited to the vehicle application machine learning model. In at least some embodiments, tester 333 is a software module within model updater 320.

[0037] Deployer 335 is in communication with vehicle application machine learning model 330 and the vehicle. In at least some embodiments, deployer 335 is configured to deploy the vehicle application machine learning model to the vehicle. In at least some embodiments, deployer 335 is configured to receive vehicle application machine learning model 330. In at least some embodiments, deployer 335 deploys vehicle application machine learning model 330 in response to validation by tester 333. In at least some embodiments, deployer 335 is a software module within model updater 320.

[0038] 4 is a schematic diagram of a vehicle according to at least some embodiments of the present disclosure. Vehicle 440 includes vehicle application model 430, application log 442, application log collector 443, sensors 445, and sensor sample collector 446. Vehicle 440 is substantially similar in structure and function to vehicle 140 of FIG. 1, unless otherwise noted. Each of vehicle application machine learning model 430 and sensor samples 426 is substantially similar in structure and function to vehicle application machine learning model 330 and sensor samples 326 of FIG. 3, respectively, unless otherwise noted.

[0039] Application log 442 is an output log of vehicle application machine learning model 430. In at least some embodiments, application log 442 includes the sequential output of inferences performed by vehicle application model 430. In at least some embodiments, application log 442 is input with image classification results. In at least some embodiments, application log 442 is a text file, a CSV file, or the like.

[0040] Application log collector 443 receives application logs, such as application logs 442. In at least some embodiments, application log collector 443 is configured to collect application log output from vehicle application machine learning model 430. In at least some embodiments, application log collector 443 is configured to interact with vehicle application machine learning model 430 to facilitate the collection. In at least some embodiments, application log collector 443 is a software component of vehicle 440.

[0041] Sensors 445 are configured to send sensor samples to vehicle application model 430 and sensor data collector 446. In at least some embodiments, sensors 445 are configured to convert real-world stimuli into digital signals and data. In at least some embodiments, sensors 445 are configured to collect real-time data about the vehicle's surroundings. In at least some embodiments, sensors 445 are one of many sensors included on vehicle 440. In at least some embodiments, sensors 445 are cameras, lidar, radar, microphones, GPS sensors, accelerometers, thermometers, barometers, etc.

[0042] Sensor sample collector 446 communicates with sensor 445. In at least some embodiments, sensor sample collector 446 is configured to collect sensor samples, such as sensor sample 426, from sensor 445. In at least some embodiments, sensor sample collector 446 is configured to interact with sensor 445. In at least some embodiments, sensor sample collector 446 is also configured to interact with vehicle application machine learning model 430 to verify the eligibility of collected sensor samples. In at least some embodiments, sensor sample collector 446 is a software component within vehicle 440.

[0043] 5 is an operational flow for continuous updating of a driving system for incident avoidance, according to at least some embodiments of the present disclosure. In at least some embodiments, the operational flow provides a method for continuous updating of a driving system for incident avoidance, according to at least some embodiments of the present disclosure. In at least some embodiments, the method is performed by a controller of a server, such as controller 1002 of server 1000 of FIG. 10 , described below.

[0044] At S550, the controller, or a portion thereof, generates compliance requirements for the vehicle application. In at least some embodiments, the controller instructs the requirements definition machine learning model to define the compliance requirements for the vehicle application based on the incident clusters. In at least some embodiments, the result of this operation by the controller is the defined compliance requirements for the vehicle application. In at least some embodiments, the controller performs this operation to set standards that the vehicle application machine learning model is trained to meet. In at least some embodiments, the controller performs the operational flow of FIG. 6, described below.

[0045] At S552, the controller, or a portion thereof, updates the vehicle application machine learning model. In at least some embodiments, the controller trains the vehicle application machine learning model to meet compliance requirements for the generated vehicle application. In at least some embodiments, the result of this operation by the controller is an updated version of the vehicle application machine learning model. In at least some embodiments, the controller performs this operation to improve the performance of the vehicle application machine learning model based on the most recent incidents. In at least some embodiments, the controller performs the operational flow of FIG. 7, described below.

[0046] At S553, the controller, or a portion thereof, deploys the updated model. In at least some embodiments, the controller deploys the vehicle application machine learning model to the vehicle. In at least some embodiments, in response to this deployment, the controller instructs the vehicle to begin using the updated version of the vehicle application machine learning model. In at least some embodiments, this action by the controller is performed only if the vehicle application machine learning model meets compliance requirements for the vehicle application. In at least some embodiments, the controller performs this action so that the vehicle system can benefit from improvements made to the model.

[0047] At S555, the controller, or a portion thereof, receives an output log. In at least some embodiments, the controller receives an output log of the vehicle application machine learning model from the vehicle. In at least some embodiments, in response to receiving the output log, the controller analyzes the output log to validate the vehicle application machine learning model. In at least some embodiments, this operation requires deploying the vehicle application machine learning model to the vehicle.

[0048] At S556, the controller, or a portion thereof, determines whether the vehicle application machine learning model meets the compliance requirements of the vehicle application. In response to the vehicle application machine learning model not meeting the compliance requirements of the vehicle application, the operational flow proceeds to updating the requirements at S558. In response to the vehicle application machine learning model meeting the compliance requirements of the vehicle application, the operational flow ends. In at least some embodiments, the controller performs this operation to verify that the vehicle application machine learning model continues to meet the compliance requirements of the vehicle application after deployment.

[0049] At S558, the controller updates the compliance requirements for the vehicle application. After updating the requirements, the operational flow returns to S552 to update the model. In at least some embodiments, the controller updates the compliance requirements for the vehicle application according to the results of the application log analysis. In at least some embodiments, in response to updating the requirements, the controller again updates the vehicle application machine learning model based on the updated requirements.

[0050] 6 is an operational flow for requirements generation according to some embodiments of the present disclosure. In at least some embodiments, the operational flow provides a method of requirements generation according to at least some embodiments of the present disclosure. In at least some embodiments, the method is performed by a controller of a server, such as controller 1002 of server 1000 of FIG. 10 , described below.

[0051] At S660, the controller, or a portion thereof, collects incident samples. In at least some embodiments, the controller utilizes a discriminative machine learning model to collect incident samples from the internet. In at least some embodiments, the controller collects various incident samples associated with one or more vehicles. In at least some embodiments, the controller interacts the discriminative machine learning model with various data sources on the internet, such as web news articles, web pages, Twitter, Facebook, Instagram, TikTok, or other internet content. In at least some embodiments, the controller stores the incident samples in an incident database.

[0052] At S662, the controller, or a portion thereof, clusters the incident samples. In at least some embodiments, the controller uses a clustering machine learning model to organize the collected incident samples into incident clusters. In at least some embodiments, this operation changes the composition and classification of the incident samples through the clustering machine learning model. In at least some embodiments, the controller forms sets or clusters of incident samples. In at least some embodiments, the controller employs a clustering machine learning model to identify patterns and trends in the incident samples.

[0053] In step S664, the controller, or a portion thereof, determines whether the incident cluster exceeds a threshold. In response to the condition not being met, the operational flow returns to collecting incident samples in step S660. In response to the condition being met, the operational flow proceeds to defining compliance requirements for the vehicle application in step S667. In at least some embodiments, the controller determines whether a priority value assigned to the incident cluster exceeds a priority threshold. In at least some embodiments, the priority value is related to the number of incident samples in the incident cluster. In at least some embodiments, the controller compares the priority value of the incident cluster to the priority threshold. In at least some embodiments, the controller ensures that only significant incident clusters, as determined by the priority threshold, are used in defining the compliance requirements for the vehicle application.

[0054] In step S667, the controller, or a portion thereof, defines compliance requirements for the vehicle application. In at least some embodiments, the controller utilizes a requirements definition machine learning model to generate compliance requirements for the vehicle application based on an incident cluster of the plurality of incident clusters. In at least some embodiments, the controller applies the requirements definition machine learning model to incident samples in the incident cluster, the vehicle information, and the sensor samples to generate compliance requirements for the vehicle application.

[0055] 7 is an operational flow for model training according to at least some embodiments of the present disclosure. In at least some embodiments, the operational flow provides a method of model training according to at least some embodiments of the present disclosure. In at least some embodiments, the method is performed by a controller of a server, such as controller 1002 of server 1000 of FIG. 10 described below.

[0056] At S770, the controller, or a portion thereof, defines annotation rules. In at least some embodiments, the controller creates a set of guidelines or rules. In at least some embodiments, these rules determine which sensor samples should be labeled or annotated. In at least some embodiments, the controller utilizes a requirements definition machine learning model to generate annotation rules for annotating sensor samples for supervised learning to meet compliance requirements for the vehicle application. In at least some embodiments, the controller defines annotation rules for consistent and accurate labeling of sensor samples.

[0057] At S772, the controller, or a portion thereof, labels the sensor samples. In at least some embodiments, the controller applies one or more annotation rules to annotate the sensor samples for supervised learning to meet compliance requirements for the vehicle application. In at least some embodiments, the controller retrieves available sensor samples from a database of sensor samples. In at least some embodiments, the controller generates a training set of labeled sensor samples based on the annotation rules. In at least some embodiments, the annotation rules enable the controller to generate training samples without human intervention.

[0058] At S774, the controller, or a portion thereof, trains the model. In at least some embodiments, the controller trains a vehicle application machine learning model. In at least some embodiments, the controller adjusts weights and biases of the vehicle application machine learning model in response to the correct or incorrect outputs. In at least some embodiments, the controller uses a portion of the labeled sensor samples for training. In at least some embodiments, the controller divides the set of training samples into a portion used for training and a portion used for testing. In at least some embodiments, the controller performs several iterations of training to generate a learned machine learning model.

[0059] At S776, the controller, or a portion thereof, tests the model. In at least some embodiments, the controller tests the trained vehicle application machine learning model. In at least some embodiments, the controller uses different portions of the labeled sensor samples for testing. In at least some embodiments, the controller applies the model to the training samples within the conditions of the metrics specified in the requirements file of the compliance requirements for the vehicle application. In at least some embodiments, the controller evaluates the performance of the vehicle application machine learning model. In at least some embodiments, the controller generates a performance evaluation of the vehicle application machine learning model.

[0060] At S778, the controller, or a portion thereof, determines whether the model satisfies the requirements. In response to the model not satisfying the requirements, the operational flow returns to one of training the model at S774, labeling the sensor samples at S772, and defining the annotation rules at S770. In response to the model satisfying the requirements, the operational flow ends. In at least some embodiments, the controller determines whether the performance of the vehicle application machine learning model satisfies the compliance requirements of the vehicle application. In at least some embodiments, the controller determines whether the model meets the metric criteria specified in the requirements file of the compliance requirements of the vehicle application. In at least some embodiments, in response to the vehicle application machine learning model satisfying the compliance requirements of the vehicle application, the controller deploys the vehicle application machine learning model. In at least some embodiments, in response to the vehicle application machine learning model not satisfying the requirements, the controller determines whether to modify the compliance requirements of the vehicle application, provide more training samples, or retrain the vehicle application machine learning model.

[0061] 8 is an operational flow for vehicle operation according to at least some embodiments of the present disclosure. In at least some embodiments, the operational flow provides a method of vehicle operation according to at least some embodiments of the present disclosure. In at least some embodiments, the method is performed by a vehicle controller, such as an electronic control unit (ECU) of vehicle 1040 of FIG. 10, described below.

[0062] At S880, the controller implements the updated model. In at least some embodiments, the controller implements an updated version of the vehicle application machine learning model. In at least some embodiments, the controller initiates use of the updated model in the vehicle. In at least some embodiments, the vehicle initiates use of the updated model for actual operation. In at least some embodiments, the controller causes the vehicle to use the most recent and accurate version of the vehicle application machine learning model.

[0063] At S882, the controller logs the model output. In at least some embodiments, the controller logs the output of the vehicle application machine learning model in an application log. In at least some embodiments, the log can be used for analysis and validation.

[0064] At S884, the controller collects sensor samples. In at least some embodiments, the controller collects sensor samples from one or more sensors in the vehicle. In at least some embodiments, the controller generates a set of sensor samples. In at least some embodiments, the controller preserves sufficient quality and resolution of the sensor samples for training and testing vehicle application machine learning models.

[0065] At S886, the controller determines whether the predetermined output has been logged. In response to the predetermined output not being logged, the operational flow returns to logging the model output at S882. In response to the predetermined output being logged, the operational flow proceeds to sending the output log and sensor samples at S888. In at least some embodiments, the predetermined output is a classification of a scene to be avoided according to compliance requirements of the vehicle application.

[0066] At S888, the controller transmits the output log and sensor samples. In at least some embodiments, the controller transmits the output log and sensor samples. In at least some embodiments, this operation transmits the collected data to a location. In at least some embodiments, the location is a location where the data can be analyzed and used to update the model. In at least some embodiments, the predetermined outputs are logged and the sensor samples are collected. In at least some embodiments, the output log and sensor samples are transmitted for further processing. In at least some embodiments, transmitting the data allows for further analysis and validation as needed for model updates.

[0067] 9 is a schematic diagram of a requirements file according to at least some embodiments of the present disclosure. Requirements file 923 includes a requirements identifier 990, a requirements summary 991, a training set identifier 992, and metrics 994. In at least some embodiments, requirements file 923 is a document that includes all of the necessary information about a requirement, including the requirements identifier, summary, training set identifier, and metrics. In at least some embodiments, requirements file 923 includes structured data formatted in a computer-readable format, such as YAML format, JSON format, protocol buffers, text files, table data, machine-readable data, etc. In at least some embodiments, the requirements file includes details of compliance requirements for a particular machine learning model, such as a vehicle application machine learning model.

[0068] Requirements identifier 990 identifies requirements file 923. In at least some embodiments, requirements identifier 990 is a unique code that distinguishes requirements file 923 from other requirements files. In at least some embodiments, requirements identifier 990 is a unique alphanumeric string, such as "XYZ-1234." In at least some embodiments, requirements identifier 990 is suitable as a search key for querying a requirements database.

[0069] Requirements summary 991 is a summary of requirements file 923. In at least some embodiments, requirements summary 991 provides a concise description of the compliance requirements for the vehicle application in requirements file 923. In at least some embodiments, requirements summary 991 includes background and context for the compliance requirements for the vehicle application in requirements file 923. In at least some embodiments, requirements summary 991 is a string of text, such as "Evasive action to avoid being pinned between trucks." In at least some embodiments, requirements summary 991 includes one or more specific incident examples.

[0070] The training set identifier 992 identifies the training samples. In at least some embodiments, the training set identifier 992 is a unique code that identifies a set of training samples used to train the applicable vehicle application machine learning model to meet the vehicle application's compliance requirements in the requirements file 923. In at least some embodiments, the training set identifier 992 is a unique alphanumeric string, such as "TRAINSET123456," or metadata suitable for querying. In at least some embodiments, the training set identifier 992 is a uniform resource locator (URL).

[0071] Metrics 994 includes one or more metrics from requirements file 923. Each metric in metrics 994 includes a metric type 995, metric criteria 996, and metric conditions 997. In at least some embodiments, each metric in metrics 994 identifies a measure used to evaluate the performance of the vehicle application machine learning model with the vehicle application compliance requirements defined in requirements file 923. In at least some embodiments, metric type 995 represents a type of metric. In at least some embodiments, metric type 995 is one of mean average precision (mAP), F1 score, precision, recall, accuracy, Jaccard, Intersection over Union (IoU), mean squared error (MSE), mean absolute error (MAE), etc. In at least some embodiments, metric criteria 996 outlines the criteria used for the metric. In at least some embodiments, metric criteria 996 is a string or a number, such as ">0.80," "<0.80," "=0.80," ">=0.80," "<=0.80," etc. In at least some embodiments, metric conditions 997 specify a condition for testing the metric. In at least some embodiments, metric conditions 997 narrow the range of training samples used to test metric criteria 996. In at least some embodiments, metric conditions 997 are strings, such as "weather == clear," "target == passenger car," "road == highway," etc.

[0072] FIG. 10 is a block diagram of a hardware configuration for continuous updating of a driving system for incident avoidance, according to at least some embodiments of the present disclosure.

[0073] An exemplary hardware configuration includes a server 1000 that interacts with input devices 1007 and vehicles 1040 directly or via the Internet 1009. In at least some embodiments, the input devices 1007 are touchscreens, microphones, cameras, or other devices configured to detect tactile input, auditory input, visual input, etc. In at least some embodiments, the Internet 1009 is an Ethernet network, other wired or wireless network, or a combination thereof. In at least some embodiments, the server 1000 is a computer or other computing device that receives input or commands from the input devices 1007. In at least some embodiments, the server 1000 is integrated with the input devices 1007. In at least some embodiments, the server 1000 is a computer system that executes computer-readable instructions to perform operations for continuously updating a driving system for incident avoidance.

[0074] The server 1000 includes a controller 1002, storage 1004, an input / output interface 1006, and a communication interface 1008. In at least some embodiments, the controller 1002 includes a processor or programmable circuit that executes instructions to perform operations in accordance with the instructions. In at least some embodiments, the controller 1002 includes analog or digital programmable circuitry or any combination thereof. In at least some embodiments, the controller 1002 includes physically separate storage or circuitry that interacts through communications. In at least some embodiments, the storage 1004 includes non-volatile computer-readable media capable of storing executable and non-executable data for access by the controller 1002 during execution of instructions. In at least some embodiments, the communication interface 1008 transmits data to and receives data from the Internet 1009. In at least some embodiments, the input / output interface 1006 connects to various input / output units, such as input devices 1007 via parallel ports, serial ports, keyboard ports, mouse ports, monitor ports, and the like, to accept commands and present information. In some embodiments, storage 1004 is external to server 1000 .

[0075] The controller 1002 includes a generator 1002A, an updater 1002B, a deployer 1002C, and a receiver 1002D. The storage 1004 includes a sample 1004A, a requirements file 1004B, a model 1004C, and a log 1004D.

[0076] Generator 1002A is circuitry or instructions of controller 1002 configured to generate compliance requirements for a vehicle application. In at least some embodiments, generator 1002A is configured to define the compliance requirements for the vehicle application through a requirements definition machine learning model according to an incident cluster of a plurality of incident clusters. In at least some embodiments, generator 1002A utilizes information in storage 1004, such as samples 1004A and models 1004D, and records information in storage 1004, such as requirements file 1004B. In at least some embodiments, generator 1002A includes subsections for performing additional functions, such as those described in the preceding flowcharts. In at least some embodiments, such subsections are referenced by names associated with the corresponding functions.

[0077] The update unit 1002B is circuitry or instructions of the controller 1002 configured to update the vehicle application machine learning model. In at least some embodiments, the update unit 1002B is configured to train the vehicle application machine learning model using a first portion of a plurality of training samples. In at least some embodiments, the update unit 1002B uses information in the storage 1004, such as the samples 1004A and the requirements file 1004B, and records information in the storage 1004, such as the model 1004C. In at least some embodiments, the update unit 1002B includes subsections for performing additional functions, such as those described in the preceding flowcharts. In at least some embodiments, such subsections are referenced by names associated with the corresponding functions.

[0078] The deployer 1002C is circuitry or instructions of the controller 1002 configured to deploy the vehicle application machine learning model. In at least some embodiments, the deployer 1002C is configured to deploy the vehicle application machine learning model to the vehicle system in response to determining that the vehicle application machine learning model meets compliance requirements for the vehicle application. In at least some embodiments, the updater 1002B utilizes information in the storage 1004, such as the model 1004C. In at least some embodiments, the deployer 1002C includes subsections for performing additional functions, such as those described in the preceding flowcharts. In at least some embodiments, such subsections are referenced by names associated with the corresponding functions.

[0079] Receiver 1002D is circuitry or instructions in controller 1002 configured to receive application logs and other output logs. In at least some embodiments, receiver 1002D is configured to receive vehicle application machine learning model output logs from vehicle systems. In at least some embodiments, receiver 1002D records information in storage 1004, such as log 1004D. In at least some embodiments, receiver 1002D includes subsections for performing additional functions, such as those described in the preceding flowcharts. In at least some embodiments, such subsections are referenced by names associated with the corresponding functions.

[0080] In at least some embodiments, the appliance is a separate device capable of processing logical functions to perform the operations herein. In at least some embodiments, the controller and storage unit need not be entirely separate devices, and in some embodiments share circuitry or one or more computer-readable media. In at least some embodiments, the storage unit includes a hard drive that stores both computer-executable instructions and data accessed by the controller, and the controller includes a central processing unit (CPU) and RAM combination in which the computer-executable instructions can be replicated, in whole or in part, for execution by the CPU during performance of the operations herein.

[0081] In at least some embodiments where the device is a computer, a program installed on the computer can cause the computer to function as the device of the embodiments described herein or to perform operations associated with the device of the embodiments described herein, and in at least some embodiments, such a program can be executed by a processor to cause the computer to perform certain operations associated with some or all of the blocks in the flowcharts and block diagrams described herein.

[0082] At least some embodiments are described with reference to flowcharts and block diagrams in which blocks represent (1) steps in a process in which an operation is performed or (2) sections of a controller responsible for performing an operation. In at least some embodiments, certain steps and sections are implemented by dedicated circuitry, programmable circuitry provided with computer-readable instructions stored on a computer-readable medium, and / or a processor provided with computer-readable instructions stored on a computer-readable medium. In at least some embodiments, dedicated circuitry includes digital and / or analog hardware circuitry, including integrated circuits (ICs) and / or discrete circuits. In at least some embodiments, programmable circuitry includes reconfigurable hardware circuitry, such as field programmable gate arrays (FPGAs), programmable logic arrays (PLAs), etc., including logical AND, OR, XOR, NAND, NOR, and other logic operations, flip-flops, registers, memory elements, etc.

[0083] In at least some embodiments, a computer-readable storage medium comprises a tangible device capable of holding and storing instructions for use by an instruction execution device. In some embodiments, a computer-readable storage medium comprises, for example, but not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof. A non-exhaustive list of more specific examples of computer-readable storage media includes portable computer diskettes, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disk read-only memory (CD-ROM), digital versatile disk (DVD), memory sticks, floppy disks, mechanically encoded devices such as punch cards or ridge structures in grooves having instructions recorded thereon, and any suitable combination thereof. Computer-readable storage medium, as used herein, should not be construed as a transitory signal itself, such as an electric wave or other freely propagating electromagnetic wave, an electromagnetic wave propagating through a waveguide or other transmission medium (e.g., an optical pulse passing through a fiber optic cable), or an electrical signal transmitted through a wire.

[0084] In at least some embodiments, the computer-readable program instructions described herein can be downloaded to a respective computing / processing device from a computer-readable storage medium or can be downloaded to an external computer or external storage device over a network, e.g., the Internet, a local area network, a wide area network, and / or a wireless network. In at least some embodiments, the network includes copper transmission cables, optical fiber transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. In at least some embodiments, a network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in a computer-readable storage medium in the respective computing / processing device.

[0085] In at least some embodiments, the computer-readable program instructions for performing the operations described above are assembler instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state-setting data, or source or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, or equivalents, and traditional procedural programming languages ​​such as the "C" programming language or similar programming languages. In at least some embodiments, the computer-readable program instructions execute entirely on the user's computer, partially on the user's computer, partially on the user's computer as a standalone software package and partially on a remote computer, or entirely on a remote computer or server. In at least some embodiments, in the latter scenario, the remote computer is connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or a connection is made to an external computer (e.g., through the Internet using an Internet Service Provider). In at least some embodiments, electronic circuitry, including, for example, programmable logic circuitry, field programmable gate arrays (FPGAs), or programmable logic arrays, executes computer-readable program instructions by utilizing state information in the computer-readable program instructions to individualize the electronic circuitry to perform aspects of the present invention.

[0086] Although embodiments of the present invention have been described, the technical scope of all subject matter of the claims is not limited to the above-described embodiments. Those skilled in the art will understand that various modifications and improvements to the above-described embodiments are possible. Furthermore, those skilled in the art will understand from the claims that embodiments to which such modifications or improvements have been made are included within the technical scope of the present invention.

[0087] The operations, procedures, steps, and stages of each process performed by the devices, systems, programs, and methods shown in the claims, embodiments, or figures can be performed in any order, unless the order is indicated by "before" or the like, and unless output from a previous process is used in a later process. Even if the flow of a process is described in the claims, embodiments, or figures using phrases such as "first" or "next," such description does not necessarily mean that the process must be performed in the order described.

[0088] In at least some embodiments, continuously updating the driving system for incident avoidance is performed by collecting a plurality of incident samples from the internet, where the plurality of incident samples are identified by a discriminative machine learning model as being associated with one or more vehicles; clustering the plurality of incident samples into a plurality of incident clusters by a clustering machine learning model; and defining compliance requirements for the vehicle application according to the incident clusters of the plurality of incident clusters by a requirements definition machine learning model.

[0089] In at least some embodiments, defining compliance requirements for the vehicle application includes defining metric types, metric criteria, and metric evaluation conditions. In at least some embodiments, continuously updating the driving system for incident avoidance further includes preparing a plurality of training samples in accordance with the compliance requirements for the vehicle application. In at least some embodiments, the preparing includes defining annotation rules in accordance with the compliance requirements for the vehicle application, labeling a plurality of sensor samples in accordance with the annotation rules, and selecting a plurality of training samples from among the labeled plurality of sensor samples. In at least some embodiments, continuously updating the driving system for incident avoidance further includes training a vehicle application machine learning model using a first portion of the plurality of training samples. In at least some embodiments, continuously updating the driving system for incident avoidance further includes testing the vehicle application machine learning model using a second portion of the plurality of training samples in accordance with the compliance requirements for the vehicle application. In at least some embodiments, continuously updating the driving system for incident avoidance further includes determining whether the vehicle application machine learning model satisfies the compliance requirements for the vehicle application. In at least some embodiments, continuously updating the driving system for incident avoidance further includes deploying the vehicle application machine learning model to the vehicle system in response to determining that the vehicle application machine learning model meets compliance requirements for the vehicle application. In at least some embodiments, continuously updating the driving system for incident avoidance further includes receiving an output log of the vehicle application machine learning model from the vehicle system, analyzing the output log to validate the vehicle application machine learning model, and updating the compliance requirements for the vehicle application according to results of the analysis.In at least some embodiments, the vehicle application machine learning model is configured for scene classification, the output log includes a plurality of scene classification results, and receiving the output log occurs in response to the vehicle application machine learning model classifying the scene into a predetermined class. In at least some embodiments, the providing step includes receiving a sensor data log corresponding to a plurality of sensor samples. In at least some embodiments, the compliance requirements for the vehicle application include a training set identifier. In at least some embodiments, the continuous updating of the driving system for incident avoidance further includes assigning a priority value to each incident cluster of the plurality of incident clusters, and the defining step occurs in response to determining that the priority value assigned to the incident cluster exceeds a priority threshold. In at least some embodiments, the priority value is based on a size of the incident cluster. In at least some embodiments, the compliance requirements for the vehicle application are further defined according to a vehicle type constraint. In at least some embodiments, the plurality of incident samples include natural language text. In at least some embodiments, the collecting step includes applying the plurality of incident samples to a large-scale language model. In at least some embodiments, the compliance requirements for the vehicle application include structured data in a computer-readable format.

[0090] In at least some embodiments, the continuous updating of the driving system for incident avoidance is performed by a device having a processor that executes instructions in accordance with the operations described above, or a controller that includes circuitry configured to perform the operations described above.

[0091] The foregoing has outlined features of some embodiments so that those skilled in the art may better understand aspects of the present disclosure. Those skilled in the art should appreciate that they may readily use this disclosure as a basis for designing or modifying other processes and structures to carry out the same purposes and / or achieve the same advantages as the embodiments introduced herein. Those skilled in the art should also recognize that such equivalent constructions do not depart from the spirit and scope of the present disclosure, and that various changes, substitutions, and alterations can be made herein without departing from the spirit and scope of the present disclosure.

Claims

1. collecting a plurality of incident samples from the internet, the plurality of incident samples being identified by a discriminative machine learning model as being associated with one or more vehicles; clustering the plurality of incident samples into a plurality of incident clusters using a clustering machine learning model; defining compliance requirements for a vehicle application according to an incident cluster of the plurality of incident clusters using a requirements definition machine learning model; A computer program that causes one or more processors to execute the program.

2. The computer program product of claim 1 , wherein defining the compliance requirements for the vehicle application includes defining types of metrics, criteria for the metrics, and evaluation conditions for the metrics.

3. The computer program product of claim 1 or 2, further comprising causing the one or more processors to perform the step of preparing a plurality of training samples according to compliance requirements of the vehicle application.

4. The preparing step includes: defining annotation rules according to compliance requirements of the vehicle application; labeling a plurality of sensor samples according to the annotation rules; selecting the plurality of training samples from the plurality of labeled sensor samples; 4. The computer program of claim 3, comprising:

5. The computer program product of claim 4 , further causing the one or more processors to perform the step of training a vehicle application machine learning model using a first portion of the plurality of training samples.

6. 6. The computer program product of claim 5, further causing the one or more processors to perform the step of testing the vehicle application machine learning model using a second portion of the plurality of training samples according to compliance requirements of the vehicle application.

7. 7. The computer program product of claim 6, further causing the one or more processors to perform the step of determining whether the vehicle application machine learning model meets compliance requirements for the vehicle application.

8. 8. The computer program product of claim 7, further causing the one or more processors to perform the step of deploying the vehicle application machine learning model to a vehicle system in response to determining that the vehicle application machine learning model meets compliance requirements for the vehicle application.

9. receiving an output log of the vehicle application machine learning model from the vehicle system; analyzing the output log to validate the vehicle application machine learning model; updating compliance requirements for the vehicle application according to results of the analysis; The computer program product of claim 8 , further comprising causing the one or more processors to execute:

10. the vehicle application machine learning model is configured for scene classification; the output log includes a plurality of scene classification results; 10. The computer program product of claim 9, wherein receiving the output log occurs in response to the vehicle application machine learning model classifying a scene into a predetermined class.

11. The computer program product of claim 4 , wherein the providing step includes receiving a sensor data log corresponding to the plurality of sensor samples.

12. The computer program of claim 1 or 2, wherein the compliance requirements for the vehicle application include a training set identifier.

13. further causing the one or more processors to assign a priority value to each incident cluster of the plurality of incident clusters; The computer program product of claim 1 or 2, wherein the defining step is performed in response to a determination that the priority value assigned to the incident cluster exceeds a priority threshold.

14. The computer program of claim 13 , wherein the priority value is based on a size of the incident cluster.

15. The computer program product of claim 1 or 2, wherein the compliance requirements for the vehicle application are further defined according to vehicle type constraints.

16. The computer program of claim 1 or 2, wherein the plurality of incident samples comprises natural language text.

17. The computer program of claim 1 or 2, wherein the collecting step comprises applying the plurality of incident samples to a large-scale language model.

18. The computer program product of claim 1 or 2, wherein the vehicle application compliance requirements comprise structured data in a computer-readable format.

19. 1. A method executed by one or more processors, comprising: collecting a plurality of incident samples from the internet, the plurality of incident samples being identified by a discriminative machine learning model as being associated with one or more vehicles; clustering the plurality of incident samples into a plurality of incident clusters using a clustering machine learning model; defining compliance requirements for a vehicle application according to an incident cluster of the plurality of incident clusters using a requirements definition machine learning model; A method comprising:

20. collecting a plurality of incident samples from the internet, the plurality of incident samples being identified by a discriminative machine learning model as being associated with one or more vehicles; clustering the plurality of incident samples into a plurality of incident clusters using a clustering machine learning model; defining compliance requirements for a vehicle application according to an incident cluster of the plurality of incident clusters using a requirements definition machine learning model; 12. An apparatus comprising: a controller including circuitry configured to perform

Images