Continuous updates to the operating system to prevent incidents.

Abstract

To automatically generate a compliance requirement of a vehicle application from information collected from an Internet.SOLUTION: Continuous update of a driving system for incident avoidance is performed by the steps of: collecting a plurality of incident samples from an Internet and identifying the plurality of incident samples by an identification machine-learning model to involve one or more vehicles; clustering, by a clustering machine- learning model, the plurality of incident samples into a plurality of incident clusters; and defining, by a requirement defining machine- learning model, a vehicle application compliance requirement according to an incident cluster among the plurality of incident clusters.SELECTED DRAWING: Figure 6

Description

Background Art

[0001] Requirements as Code (RaC) are used to define the requirements of a vehicle application. RaC includes various types of information that define the characteristics and behaviors of the vehicle application, the metrics and their criteria used to check whether the requirements are met, the conditions under which these metrics and criteria are evaluated, and the data or test scenarios used for such evaluation.

[0002] Creating a RaC file requires considering the functional and non-functional requirements of the vehicle application and applicable vehicle functions, the features valuable to the user, the edge cases to be considered during testing, and the specifications of the vehicle application including the issues identified for tracking in regression testing.

[0003] The RaC file is used to test the vehicle application and the machine learning model adopted by the vehicle application to determine whether the requirements are met and to judge whether the test criteria are met.

Brief Description of the Drawings

[0004] [Figure 1] FIG. 1 is a schematic diagram of a system for continuous update of a driving system for incident avoidance according to at least some embodiments of the present disclosure. [Figure 2] FIG. 2 is a schematic diagram of a generator according to at least some embodiments of the present disclosure. [Figure 3] FIG. 3 is a schematic diagram of a model updater according to at least some embodiments of the present disclosure. [Figure 4] FIG. 4 is a schematic diagram of a vehicle according to at least some embodiments of the present disclosure. [Figure 5] FIG. 5 is an operation flow for continuous update of a driving system for incident avoidance according to at least some embodiments of the present disclosure. [Figure 6] Figure 6 shows an operational flow for requirement generation according to at least some embodiments of this disclosure. [Figure 7] Figure 7 shows an operational flow for model training according to at least some embodiments of this disclosure. [Figure 8] Figure 8 shows an operation flow for vehicle operation according to at least some embodiments of this disclosure. [Figure 9] Figure 9 is a schematic diagram of a requirements file according to at least some embodiments of this disclosure. [Figure 10] Figure 10 is a block diagram of a hardware configuration for continuous updating of an operating system for incident avoidance, according to at least some embodiments of the present disclosure. [Modes for carrying out the invention]

[0005] The aspects of this disclosure will be best understood from the following detailed description when read in conjunction with the accompanying drawings. Note that, in accordance with standard practice in the industry, various features are not depicted to scale. In fact, the dimensions of various features may be arbitrarily increased or decreased for the sake of clarity in the discussion.

[0006] The following disclosure provides various embodiments or examples for implementing various features of the subject matter provided. Specific examples of components, values, operations, materials, arrangements, or equivalents thereof are described below for the sake of simplification of this disclosure. Naturally, these are merely examples and are not intended to be limiting. Other components, values, operations, materials, arrangements, or equivalents thereof are contemplated. In addition, this disclosure may repeat reference numerals and / or letters in various examples. This repetition is for simplification and clarity and is not in itself intended to indicate relationships between the various embodiments and / or configurations discussed.

[0007] Vehicle applications and their machine learning models are not always updated in a timely manner to address new problems that may arise in the form of challenges, weaknesses, or edge cases. This is because updating requires notifying developers of new issues, and for developers to manually describe requirements and design tests accordingly to resolve each issue.

[0008] In at least some embodiments of this disclosure, incident samples are collected from the internet using an identification machine learning model, clustered by a clustering machine learning model, and used as a basis for defining compliance requirements for the vehicle application by a requirements definition machine learning model, in order to update the vehicle application and its machine learning model. In at least some embodiments, the requirements definition machine learning model defines rules for annotating sensor samples used as training samples.

[0009] In at least some embodiments, incident samples are collected from the internet using a collection machine learning (ML) model. In at least some embodiments, incident samples are clustered by a clustering ML model. In at least some embodiments, requirements for incident clusters are defined using a requirements definition ML model.

[0010] By automatically generating requirements from information collected from the internet, at least some embodiments continuously update vehicle applications and their machine learning models for incident avoidance without notifying developers or requiring manual input. By automatically defining annotation rules according to the requirements, at least some embodiments automatically prepare training samples.

[0011] Figure 1 is a schematic diagram of a system for continuous updating of an operating system for incident avoidance, according to at least some embodiments of the present disclosure. The system includes a server 100, an internet 109, and a vehicle 140.

[0012] Server 100 communicates with the Internet 109 and vehicles 140 and includes a generator 110 and a model updater 120. In at least some embodiments, Server 100 hosts machine learning models and processes data for continuous updates of the driving system for incident avoidance. In at least some embodiments, Server 100 is configured to communicate with vehicles 140 to exchange data and update information. In at least some embodiments, Server 100 is configured to perform common server tasks such as data storage and network management. In at least some embodiments, Server 100 is configured to connect to the Internet 109 for data collection and distribution. In at least some embodiments, Server 100 comprises multiple physical servers and computing resources. In at least some embodiments, Server 100 is a physical server in a data center or a virtual server in the cloud. In at least some embodiments, Server 100 is a type of server used in many areas, from web hosting to database management.

[0013] The generator 110 searches for incident samples 111 from the internet 109 and sends requirement files, such as requirement file 123, and annotation rules, such as annotation rule 124, to the model updater 120. In at least some embodiments, the generator 110 is configured to generate requirement files and annotation rules from incident samples. In at least some embodiments, the generator 110 is configured to generate many types of requirement files based on various types of vehicle applications and their machine learning models. In at least some embodiments, the generator 110 is a software module running on server 100. In at least some embodiments, the generator 110 is one of many servers comprising server 100.

[0014] The model updater 120 receives requirement files such as requirement file 123 and annotation rules such as annotation rule 124 from the generator 110, sensor samples such as sensor sample 126 and application logs such as application log 142 from the vehicle 140, and sends a vehicle application model such as vehicle application machine learning model 130 to the vehicle 140. In at least some embodiments, the model updater 120 is configured to update the vehicle application machine learning model 130 based on prepared training samples and to test the vehicle application machine learning model 130 according to vehicle application compliance requirements such as vehicle application compliance requirements in requirement file 123. In at least some embodiments, the model updater 120 is configured to update other types of machine learning models. In at least some embodiments, the model updater 120 is a software module running on server 100. In at least some embodiments, the model updater 120 is one of many servers comprising server 100.

[0015] Vehicle 140 receives a vehicle application model, such as a vehicle application machine learning model 130, from a model updater 120 and sends sensor samples, such as a sensor sample 126, and application logs, such as an application log 142, to the model updater 120. In at least some embodiments, vehicle 140 is configured to deploy an updated version of the vehicle application machine learning model 130 and to generate an application log 142 based on the output of the vehicle application machine learning model 130. In at least some embodiments, vehicle 140 is configured to perform normal vehicle functions, such as transportation. In at least some embodiments, vehicle 140 is configured to interact with the physical world through sensors and actuators. In at least some embodiments, vehicle 140 is any vehicle equipped with a compatible system, such as a car, truck, boat, aircraft, submarine, etc.

[0016] The Internet 109 communicates with the Server 100. In at least some embodiments, the Internet 109 is configured to provide a source of incident samples, such as incident sample 111, for the system. In at least some embodiments, the Internet 109 is configured to communicate with the Server 100 to exchange data. In at least some embodiments, the Internet 109 is configured to provide a network for various other applications. In at least some embodiments, the Internet 109 is configured to connect to various systems and devices around the world. In at least some embodiments, the Internet 109 is a global network. In at least some embodiments, the Internet 109 is used for a wide range of applications, from communications to entertainment.

[0017] Figure 2 is a schematic diagram of a generator according to at least some embodiments of the present disclosure. The generator 210 includes an incident collector 212, an incident database 214, an incident clusterer 215, an incident cluster database 217, and a requirements definition model 219. Unless otherwise noted, the generator 210 is substantially the same in structure and function as the generator 110 in Figure 1.

[0018] The incident collector 212 includes an identification model 213. In at least some embodiments, the incident collector 212 identifies and retrieves incident samples, such as incident sample 211, and stores the incident samples in the incident database 214. In at least some embodiments, the incident collector 212 is configured to collect incident samples from the internet. In at least some embodiments, an incident sample, such as incident sample 211, is a natural language text sample containing a description of an incident related to one or more vehicles. In at least some embodiments, an incident sample, such as incident sample 211, includes an image. In at least some embodiments, an incident sample, such as incident sample 211, includes a combination of natural language and image data. In at least some embodiments, the incident collector 212 is configured to remove personally identifiable information from incident samples using techniques such as filtering, generalization, and ambiguation. In at least some embodiments, the incident collector 212 is configured to replace specific vehicle identification information in incident samples with general descriptions such as "heavy trailer" or "tanker truck," as needed. In at least some embodiments, the incident collector 212 is configured to store the collected incident samples for further processing. In at least some embodiments, the incident collector 212 is configured to interact with the internet to collect incident samples. In at least some embodiments, the incident collector 212 is a web crawler or data scraping tool in real-world form.

[0019] The identification model 213 identifies incident samples for the incident collector 212. In at least some embodiments, the identification model 213 is configured to identify incident samples associated with one or more vehicles. In at least some embodiments, the identification model 213 is configured to process incident samples collected by the incident collector 212. In at least some embodiments, the identification model 213 is configured to distinguish incident samples from other types of data samples. In at least some embodiments, the identification model 213 is a machine learning model trained to perform a task. In at least some embodiments, the identification model 213 is a large-scale language machine learning model trained for natural language processing.

[0020] The incident database 214 receives incident samples from the generator 210 and sends the incident samples to the incident cluster unit 215. In at least some embodiments, the incident database 214 is configured to store incident samples collected from the internet. In at least some embodiments, the incident database 214 is configured to store other types of data. In at least some embodiments, the incident database 214 is a file system, a relational database, a NoSQL database, etc. In at least some embodiments, the incident database 214 is a type of database used in many fields such as data analysis and web development.

[0021] The incident clusterer 215 includes a clustering model 216. In at least some embodiments, the incident clusterer 215 is configured to receive incident samples from the incident database 214 and send incident clusters to the incident cluster database 217. In at least some embodiments, the incident clusterer 215 is configured to cluster incident samples into incident clusters. In at least some embodiments, the incident clusterer 215 is configured to cluster many types of data.

[0022] The clustering model 216 is used by the incident clusterer 215. In at least some embodiments, the clustering model 216 is used by the incident clusterer 215 to cluster incident samples. In at least some embodiments, the clustering model 216 is a machine learning model trained to perform clustering tasks for vehicle incidents. In at least some embodiments, the clustering model 216 is a type of model used in many fields such as data mining and market segmentation.

[0023] The incident cluster database 217 receives an incident cluster from the incident cluster device 215 and sends the incident cluster to the requirement definition model 219. In at least some embodiments, the incident cluster database 217 is configured to store the incident cluster generated by the incident cluster device 215. In at least some embodiments, the incident cluster database 217 is configured to store other types of data. In at least some embodiments, the incident cluster database 217 is a file system, a relational database, a NoSQL database, etc. In at least some embodiments, the incident cluster database 217 is a type of database used in many fields such as data analysis and web development.

[0024] The requirements definition model 219 receives incident clusters from the incident cluster database 217, receives vehicle information 221, receives application logs from the application log database 226, and sends the compliance requirements for the vehicle application to the requirements database 222. In at least some embodiments, the requirements definition model 219 is configured to define compliance requirements for the vehicle application, such as the compliance requirements for the vehicle application in the requirements file 223, based on the incident clusters and vehicle information. In at least some embodiments, the requirements definition model 219 is configured to define compliance requirements for the vehicle application, such as the compliance requirements for the vehicle application in the requirements file 223, based on the incident clusters, vehicle information, and application logs. In at least some embodiments, the requirements definition model 219 is configured to process incident clusters stored in the incident cluster database 217. In at least some embodiments, the requirements definition model 219 is configured to define compliance requirements for the vehicle application for many types of vehicle applications. In at least some embodiments, the requirements definition model 219 is a machine learning model trained to perform the RaC definition task. In at least some embodiments, the requirements definition model 219 is configured to define rules such as annotation rules 224 for annotating sensor samples to be used as training samples.

[0025] Vehicle information 221 is used by the requirements definition model 219. In at least some embodiments, vehicle information 221 is used by the requirements definition model 219 to tailor the compliance requirements of a vehicle application to a specific vehicle or a specific type of vehicle. In at least some embodiments, vehicle information 221 includes the specifications of the vehicle application, the design documents of the vehicle application, the source code of the vehicle application, or any combination thereof. In at least some embodiments, vehicle information 221 is configured to provide information about the vehicle to which the compliance requirements of the vehicle application are deployed. In at least some embodiments, vehicle information 221 is configured to provide information about many types of vehicles. In at least some embodiments, vehicle information 221 is a database or file containing vehicle specifications in real-world form. In at least some embodiments, vehicle information is a type of information used in many fields such as automotive engineering and automotive manufacturing.

[0026] The requirements database 222 receives compliance requirements for the vehicle application from the requirements definition model 219. In at least some embodiments, the requirements database 222 is configured to store the compliance requirements for the vehicle application as defined by the requirements definition model 219. In at least some embodiments, the requirements database 222 is configured to provide these requirements to other components for processing. In at least some embodiments, the requirements database 222 is configured to store compliance requirements for the vehicle application for many types of vehicle applications. In at least some embodiments, the requirements database 222 is a file system, a relational database, a NoSQL database, etc.

[0027] The requirements file 223 is generated by the generator 210. In at least some embodiments, the requirements file 223 is configured to contain compliance requirements for the vehicle application in a computer-readable format. In at least some embodiments, the requirements file 223 is used by other components to understand the compliance requirements for the vehicle application. In at least some embodiments, the requirements file 223 is a text file, a JSON file, an XML file, etc. In at least some embodiments, the requirements file 223 is an RaC file, such as those used in software development and project management.

[0028] Annotation rules 224 are generated by the generator 210. In at least some embodiments, annotation rules 224 are used to label training samples. In at least some embodiments, annotation rules 224 are a set of rules defined in a programming language.

[0029] The application log database 226 communicates with the requirements definition model 219. In at least some embodiments, the application log database 226 is configured to store output application logs of a vehicle application machine learning model. In at least some embodiments, the application log database 226 is configured to receive application logs from the vehicle. In at least some embodiments, the application log database 226 is configured to provide application logs to the requirements definition model 219. In at least some embodiments, the application log database 226 is configured to store other types of logs. In at least some embodiments, the application log database 226 is configured to interact with other components and provide application logs to those components. In at least some embodiments, the application log database 226 is a file system, a relational database, a NoSQL database, etc.

[0030] Figure 3 is a schematic diagram of a model updater according to at least some embodiments of the present disclosure. The model updater 320 includes a sample labeler 328, a vehicle application model 330, a training sample 331, a trainer 332, a tester 333, and a deployer 335. Unless otherwise noted, the model updater 320 is substantially the same in structure and function as the model updater 120 in Figure 1. The requirements database 322, the requirements file 323, and the annotation rules 324 are substantially the same in structure and function as the requirements database 222, the requirements file 223, and the annotation rules 224 in Figure 2, respectively, unless otherwise noted.

[0031] Sensor sample 326 is a sample of data collected from a sensor. In at least some embodiments, sensor sample 326 is a sample of data collected from a vehicle sensor. In at least some embodiments, sensor sample 326 is labeled by sample labeler 328 to become a training sample. In at least some embodiments, sensor sample 326 may take the form of a value, string, image, video, audio clip, or other digital format generated by the sensor.

[0032] The sensor sample database 327 stores and provides sensor samples such as sensor sample 326. In at least some embodiments, the sensor sample database 327 is configured to store any type of data sample. In at least some embodiments, the sensor sample database 327 provides sensor samples to other components.

[0033] The sample labeler 328 receives sensor samples from the sensor sample database 327, receives annotation rules 324, and provides training samples to the trainer 332 and tester 333. In at least some embodiments, the sample labeler 328 applies the annotation rules 324 to the sensor samples and provides labeled samples, such as training samples 331A and 331B, to the trainer 332 and tester 333, respectively. In at least some embodiments, the sample labeler 328 is configured to label sensor samples according to the annotation rules 324. In at least some embodiments, the sample labeler 328 is configured to label any type of data sample. In at least some embodiments, the sample labeler 328 determines which training samples are used for training and which are used for testing. In at least some embodiments, the sample labeler 328 is a software module within the model updater 320. In at least some embodiments, the sample labeler 328 is a type of labeler used in any system employing supervised machine learning.

[0034] The vehicle application machine learning model 330 is a machine learning model for a vehicle application. In at least some embodiments, the vehicle application machine learning model 330 is trained or updated using training samples such as training sample 331A. In at least some embodiments, the vehicle application machine learning model 330 is one of many types of autonomous driving models, such as an image classification model. In at least some embodiments, the vehicle application machine learning model 330 is a data structure that encapsulates the parameters of the machine learning model.

[0035] The trainer 332 receives training samples, such as training sample 331A, from the sample labeler 328 and trains the vehicle application machine learning model 330. In at least some embodiments, the trainer 332 is configured to train the vehicle application machine learning model using a portion of the training samples. In at least some embodiments, the trainer 332 is configured to update the vehicle application machine learning model 330. In at least some embodiments, the trainer 332 is not limited to training the vehicle application machine learning model. In at least some embodiments, the trainer 332 is configured to train any machine learning model using supervised learning. In at least some embodiments, the trainer 332 stores iterations of the vehicle application model 330 during training. In at least some embodiments, the trainer 332 is a software module within the model updater 320.

[0036] The tester 333 is configured to receive test samples, such as training sample 331B, from the sample labeler 328. In at least some embodiments, the tester 333 is configured to apply the vehicle application machine learning model 330 to the training samples. In at least some embodiments, the tester 333 is configured to test the vehicle application machine learning model using a portion of the training samples. In at least some embodiments, the tester 333 is not limited to the vehicle application machine learning model. In at least some embodiments, the tester 333 is a software module in the model updater 320.

[0037] The deployer 335 communicates with the vehicle application machine learning model 330 and the vehicle. In at least some embodiments, the deployer 335 is configured to deploy the vehicle application machine learning model to the vehicle. In at least some embodiments, the deployer 335 is configured to receive the vehicle application machine learning model 330. In at least some embodiments, the deployer 335 deploys the vehicle application machine learning model 330 in response to verification by the tester 333. In at least some embodiments, the deployer 335 is a software module within the model updater 320.

[0038] Figure 4 is a schematic diagram of a vehicle according to at least some embodiments of the present disclosure. The vehicle 440 includes a vehicle application model 430, an application log 442, an application log collector 443, a sensor 445, and a sensor sample collector 446. Unless otherwise noted, the vehicle 440 is substantially identical in structure and function to the vehicle 140 of Figure 1. The vehicle application machine learning model 430 and the sensor sample 426 are substantially identical in structure and function to the vehicle application machine learning model 330 and the sensor sample 326 of Figure 3, respectively, unless otherwise noted.

[0039] The application log 442 is the output log of the vehicle application machine learning model 430. In at least some embodiments, the application log 442 includes the sequential output of inferences performed by the vehicle application model 430. In at least some embodiments, the application log 442 is input with image classification results. In at least some embodiments, the application log 442 is a text file, a CSV file, etc.

[0040] The application log collector 443 receives application logs such as application log 442. In at least some embodiments, the application log collector 443 is configured to collect application log output from the vehicle application machine learning model 430. In at least some embodiments, the application log collector 443 is configured to interact with the vehicle application machine learning model 430 to facilitate collection. In at least some embodiments, the application log collector 443 is a software component of the vehicle 440.

[0041] Sensor 445 is configured to transmit sensor samples to the vehicle application model 430 and the sensor data collector 446. In at least some embodiments, sensor 445 is configured to convert real-world stimuli into digital signals and data. In at least some embodiments, sensor 445 is configured to collect real-time data about the vehicle's surroundings. In at least some embodiments, sensor 445 is one of many sensors included in vehicle 440. In at least some embodiments, sensor 445 is a camera, lidar, radar, microphone, GPS sensor, accelerometer, thermometer, barometer, etc.

[0042] The sensor sample collector 446 communicates with the sensor 445. In at least some embodiments, the sensor sample collector 446 is configured to collect sensor samples from the sensor 445, such as sensor sample 426. In at least some embodiments, the sensor sample collector 446 is configured to interact with the sensor 445. In at least some embodiments, the sensor sample collector 446 is also configured to interact with a vehicle application machine learning model 430 to verify the eligibility of the collected sensor samples. In at least some embodiments, the sensor sample collector 446 is a software component within the vehicle 440.

[0043] Figure 5 shows an operation flow for continuous updating of an operating system for incident avoidance, according to at least some embodiments of the present disclosure. In at least some embodiments, the operation flow provides a method for continuous updating of an operating system for incident avoidance, according to at least some embodiments of the present disclosure. In at least some embodiments, the method is performed by a server controller, such as the controller 1002 of the server 1000 in Figure 10 described below.

[0044] In S550, the controller or a part thereof generates compliance requirements for the vehicle application. In at least some embodiments, the controller instructs a requirements definition machine learning model to define compliance requirements for the vehicle application based on incident clusters. In at least some embodiments, the result of this operation by the controller is the defined compliance requirements for the vehicle application. In at least some embodiments, the controller performs this operation to set criteria that the vehicle application machine learning model is trained to meet. In at least some embodiments, the controller performs the operation flow shown in Figure 6 below.

[0045] In S552, the controller or a part thereof updates the vehicle application machine learning model. In at least some embodiments, the controller trains the vehicle application machine learning model to meet the compliance requirements of the generated vehicle application. In at least some embodiments, the result of this operation by the controller is an updated version of the vehicle application machine learning model. In at least some embodiments, the controller performs this operation to improve the performance of the vehicle application machine learning model based on the most recent incident. In at least some embodiments, the controller performs the operation flow shown in Figure 7 below.

[0046] In S553, the controller or a part thereof deploys the updated model. In at least some embodiments, the controller deploys the vehicle application machine learning model to the vehicle. In at least some embodiments, in response to this deployment, the controller instructs the vehicle to begin using the updated version of the vehicle application machine learning model. In at least some embodiments, this operation by the controller is performed only if the vehicle application machine learning model meets the compliance requirements of the vehicle application. In at least some embodiments, the controller performs this operation so that the vehicle system can benefit from the improvements made to the model.

[0047] In S555, the controller or a part thereof receives output logs. In at least some embodiments, the controller receives output logs of a vehicle application machine learning model from the vehicle. In at least some embodiments, upon receiving the output logs, the controller analyzes the output logs to validate the vehicle application machine learning model. In at least some embodiments, this operation requires the deployment of the vehicle application machine learning model to the vehicle.

[0048] In S556, the controller or a part thereof determines whether the vehicle application machine learning model meets the vehicle application's compliance requirements. If the vehicle application machine learning model does not meet the vehicle application's compliance requirements, the operation flow proceeds to updating the requirements in S558. If the vehicle application machine learning model meets the vehicle application's compliance requirements, the operation flow terminates. In at least some embodiments, the controller performs this operation to verify that the vehicle application machine learning model continues to meet the vehicle application's compliance requirements after deployment.

[0049] In S558, the controller updates the compliance requirements for the vehicle application. After updating the requirements, the operation flow returns to S552 to update the model. In at least some embodiments, the controller updates the compliance requirements for the vehicle application according to the results of the application log analysis. In at least some embodiments, in response to the requirement update, the controller updates the vehicle application machine learning model again based on the updated requirements.

[0050] Figure 6 shows an operational flow for requirements generation according to some embodiments of the present disclosure. In at least some embodiments, the operational flow provides a method for requirements generation according to at least some embodiments of the present disclosure. In at least some embodiments, the method is performed by a server controller, such as the controller 1002 of the server 1000 in Figure 10 described below.

[0051] In the S660, the controller or a part thereof collects incident samples. In at least some embodiments, the controller uses an identification machine learning model to collect incident samples from the internet. In at least some embodiments, the controller collects various incident samples related to one or more vehicles. In at least some embodiments, the controller interacts the identification machine learning model with various data sources on the internet, such as web news articles, web pages, X (Twitter), Facebook, Instagram, TikTok, or other internet content. In at least some embodiments, the controller stores the incident samples in an incident database.

[0052] In S662, the controller or a part thereof clusters the incident samples. In at least some embodiments, the controller uses a clustering machine learning model to organize the collected incident samples into several incident clusters. In at least some embodiments, this operation results in changes to the composition and classification of the incident samples through the clustering machine learning model. In at least some embodiments, the controller forms a set or cluster of incident samples. In at least some embodiments, the controller allows the clustering machine learning model to identify patterns and trends in the incident samples.

[0053] In step S664, the controller or a part thereof determines whether the incident cluster exceeds a threshold. Depending on whether the condition is not met, the operation flow returns to collecting incident samples in step S660. Depending on whether the condition is met, the operation flow proceeds to defining the compliance requirements for the vehicle application in step S667. In at least some embodiments, the controller determines whether the priority value assigned to the incident cluster exceeds a priority threshold. In at least some embodiments, the priority value relates to the number of incident samples in the incident cluster. In at least some embodiments, the controller compares the priority value of the incident cluster to a priority threshold. In at least some embodiments, the controller ensures that only critical incident clusters determined by the priority threshold are used to define the compliance requirements for the vehicle application.

[0054] In step S667, the controller or a part thereof defines the compliance requirements for the vehicle application. In at least some embodiments, the controller utilizes a requirements definition machine learning model to generate the compliance requirements for the vehicle application based on incident clusters among multiple incident clusters. In at least some embodiments, the controller generates the compliance requirements for the vehicle application by applying the requirements definition machine learning model to incident samples in the incident cluster, vehicle information, and sensor samples.

[0055] Figure 7 shows an operational flow for model training according to at least some embodiments of the present disclosure. In at least some embodiments, the operational flow provides a method for model training according to at least some embodiments of the present disclosure. In at least some embodiments, the method is performed by a server controller, such as the controller 1002 of the server 1000 in Figure 10 described below.

[0056] In S770, the controller or a part thereof defines annotation rules. In at least some embodiments, the controller creates a set of guidelines or rules. In at least some embodiments, these rules determine which sensor samples should be labeled or annotated. In at least some embodiments, the controller utilizes a requirements-defined machine learning model to generate annotation rules for annotating sensor samples for supervised learning to meet the compliance requirements of the vehicle application. In at least some embodiments, the controller defines annotation rules for consistent and accurate labeling of sensor samples.

[0057] In S772, the controller or a part thereof labels the sensor samples. In at least some embodiments, the controller applies one or more annotation rules to annotate the sensor samples for supervised learning to meet the compliance requirements of the vehicle application. In at least some embodiments, the controller searches for available sensor samples from a database of sensor samples. In at least some embodiments, the controller generates a training set of labeled sensor samples based on the annotation rules. In at least some embodiments, the annotation rules enable the controller to generate training samples without human intervention.

[0058] In S774, the controller or a part thereof trains a model. In at least some embodiments, the controller trains a vehicle application machine learning model. In at least some embodiments, the controller adjusts the weights and biases of the vehicle application machine learning model depending on whether the output is correct or incorrect. In at least some embodiments, the controller uses a portion of the labeled sensor samples for training. In at least some embodiments, the controller divides a set of training samples into a portion used for training and a portion used for testing. In at least some embodiments, the controller runs the training several times to produce a trained machine learning model.

[0059] In S776, the controller or a part thereof tests the model. In at least some embodiments, the controller tests a trained vehicle application machine learning model. In at least some embodiments, the controller uses different portions of labeled sensor samples for testing. In at least some embodiments, the controller applies the model to the training samples within the range of metric conditions specified in the requirements file for compliance requirements of the vehicle application. In at least some embodiments, the controller evaluates the performance of the vehicle application machine learning model. In at least some embodiments, the controller generates a performance evaluation of the vehicle application machine learning model.

[0060] In S778, the controller or part thereof determines whether the model meets the requirements. Depending on whether the model does not meet the requirements, the operation flow returns to one of the following: training the model in S774, labeling sensor samples in S772, or defining annotation rules in S770. Depending on whether the model meets the requirements, the operation flow terminates. In at least some embodiments, the controller determines whether the performance of the vehicle application machine learning model meets the compliance requirements of the vehicle application. In at least some embodiments, the controller determines whether the model meets the criteria for metrics identified in the requirements file for the compliance requirements of the vehicle application. In at least some embodiments, depending on whether the vehicle application machine learning model meets the compliance requirements of the vehicle application, the controller deploys the vehicle application machine learning model. In at least some embodiments, depending on whether the vehicle application machine learning model does not meet the requirements, the controller determines whether the compliance requirements of the vehicle application should be modified, more training samples should be provided, or the vehicle application machine learning model should be retrained.

[0061] Figure 8 is an operation flow for vehicle operation according to at least some embodiments of the present disclosure. In at least some embodiments, the operation flow provides a method of vehicle operation according to at least some embodiments of the present disclosure. In at least some embodiments, the method is performed by a vehicle controller, such as the electronic control unit (ECU) of vehicle 1040 in Figure 10 described below.

[0062] In S880, the controller implements the updated model. In at least some embodiments, the controller implements an updated version of the vehicle application machine learning model. In at least some embodiments, the controller initiates the use of the updated model in the vehicle. In at least some embodiments, the vehicle initiates the use of the updated model for actual operation. In at least some embodiments, the controller causes the vehicle to use the latest and most accurate version of the vehicle application machine learning model.

[0063] In S882, the controller logs the model output. In at least some embodiments, the controller logs the application log of the output of the vehicle application machine learning model. In at least some embodiments, the log can be used for analysis and verification.

[0064] In S884, the controller collects sensor samples. In at least some embodiments, the controller collects sensor samples from one or more sensors in the vehicle. In at least some embodiments, the controller generates a set of sensor samples. In at least some embodiments, the controller maintains sufficient quality and resolution of the sensor samples for training and testing vehicle application machine learning models.

[0065] In S886, the controller determines whether a predetermined output has been logged. If the predetermined output has not been logged, the operation flow returns to logging the model output in S882. If the predetermined output has been logged, the operation flow proceeds to sending the output log and sensor sample in S888. In at least some embodiments, the predetermined output is a classification of scenes to be avoided according to the compliance requirements of the vehicle application.

[0066] In S888, the controller transmits output logs and sensor samples. In at least some embodiments, the controller transmits output logs and sensor samples. In at least some embodiments, this operation transmits the collected data to a location. In at least some embodiments, that location is a location where the data can be used to analyze and update the model. In at least some embodiments, predetermined outputs are logged and sensor samples are collected. In at least some embodiments, the output logs and sensor samples are transmitted for further processing. In at least some embodiments, transmitting the data enables further analysis and verification required for updating the model.

[0067] Figure 9 is a schematic diagram of a requirements file according to at least some embodiments of this disclosure. The requirements file 923 includes a requirements identifier 990, a requirements summary 991, a training set identifier 992, and metrics 994. In at least some embodiments, the requirements file 923 is a document containing all the necessary information about the requirements, including the requirements identifier, summary, training set identifier, and metrics. In at least some embodiments, the requirements file 923 includes structured data formatted in a computer-readable format, such as YAML format, JSON format, protocol buffer, text file, tabular data, machine-readable data, etc. In at least some embodiments, the requirements file includes details of compliance requirements for a specific machine learning model, such as a vehicle application machine learning model.

[0068] The requirement identifier 990 identifies the requirement file 923. In at least some embodiments, the requirement identifier 990 is a unique code that distinguishes the requirement file 923 from other requirement files. In at least some embodiments, the requirement identifier 990 is a unique alphanumeric string such as "XYZ-1234". In at least some embodiments, the requirement identifier 990 is suitable as a lookup key for querying the requirement database.

[0069] Requirement summary 991 is an overview of requirement file 923. In at least some embodiments, requirement summary 991 provides a brief description of the compliance requirements for the vehicle application in requirement file 923. In at least some embodiments, requirement summary 991 includes the background and context of the compliance requirements for the vehicle application in requirement file 923. In at least some embodiments, requirement summary 991 is a string of text such as "avoidance actions to avoid being sandwiched between trucks". In at least some embodiments, requirement summary 991 includes one or more specific incident examples.

[0070] The training set identifier 992 identifies the training samples. In at least some embodiments, the training set identifier 992 is a unique code that identifies a set of training samples used to train an applicable vehicle application machine learning model to meet the compliance requirements of the vehicle application in the requirements file 923. In at least some embodiments, the training set identifier 992 is a unique alphanumeric string such as "TRAINSET123456" or metadata suitable for querying. In at least some embodiments, the training set identifier 992 is a uniform resource locator (URL).

[0071] Metric 994 includes one or more metrics from requirements file 923. Each metric in Metric 994 includes a metric type 995, a metric criterion 996, and a metric condition 997. In at least some embodiments, each metric in Metric 994 identifies a measure used to evaluate the achievement of a vehicle application machine learning model using the compliance requirements for the vehicle application defined in requirements file 923. In at least some embodiments, metric type 995 represents the type of metric. In at least some embodiments, metric type 995 is one of the following: mean mean precision (mAP), F1 score, precision, recall, accuracy, Jaccard, Intersection over Union (IoU), mean squared error (MSE), mean absolute error (MAE), etc. In at least some embodiments, metric criterion 996 provides an overview of the criterion used for the metric. In at least some embodiments, the metric criterion 996 is a string or number such as ">0.80", "<0.80", "=0.80", ">=0.80", "<=0.80", etc. In at least some embodiments, the metric condition 997 specifies the conditions for testing the metric. In at least some embodiments, the metric condition 997 narrows the range of training samples used to test the metric criterion 996. In at least some embodiments, the metric condition 997 is a string such as "weather==sunny", "target==passenger car", "road==main road", etc.

[0072] Figure 10 is a block diagram of a hardware configuration for continuous updating of an operating system for incident avoidance, according to at least some embodiments of the present disclosure.

[0073] An exemplary hardware configuration includes an input device 1007 and a server 1000 that interacts with the vehicle 1040 directly or via the internet 1009. In at least some embodiments, the input device 1007 is a touchscreen, microphone, camera, or other device configured to detect tactile, auditory, visual, etc. In at least some embodiments, the internet 1009 is an Ethernet network, other wired or wireless network, or a combination thereof. In at least some embodiments, the server 1000 is a computer or other computing device that receives input or commands from the input device 1007. In at least some embodiments, the server 1000 is integrated with the input device 1007. In at least some embodiments, the server 1000 is a computer system that executes computer-readable instructions to perform operations for continuous updates of the driving system for incident avoidance.

[0074] The server 1000 includes a controller 1002, storage 1004, an input / output interface 1006, and a communication interface 1008. In at least some embodiments, the controller 1002 includes a processor or programmable circuitry that executes instructions and performs operations according to those instructions. In at least some embodiments, the controller 1002 includes analog or digital programmable circuitry or any combination thereof. In at least some embodiments, the controller 1002 includes physically separated storage or circuitry that interacts through communication. In at least some embodiments, the storage 1004 includes a non-volatile computer-readable medium capable of storing executable and non-executable data for access by the controller 1002 during instruction execution. In at least some embodiments, the communication interface 1008 transmits data to and receives data from the internet 1009. In at least some embodiments, the input / output interface 1006 is connected to various input / output units such as an input device 1007 via a parallel port, serial port, keyboard port, mouse port, monitor port, and equivalents thereof to accept commands and present information. In some embodiments, the storage 1004 is located outside the server 1000.

[0075] The controller 1002 includes a generation unit 1002A, an update unit 1002B, a deployment unit 1002C, and a receiving unit 1002D. The storage 1004 includes a sample 1004A, a requirements file 1004B, a model 1004C, and a log 1004D.

[0076] The generation unit 1002A is a circuit or instruction of the controller 1002 configured to generate compliance requirements for a vehicle application. In at least some embodiments, the generation unit 1002A is configured to define compliance requirements for a vehicle application by a requirements definition machine learning model according to an incident cluster among a plurality of incident clusters. In at least some embodiments, the generation unit 1002A utilizes information in storage 1004, such as sample 1004A and model 1004D, and records information in storage 1004, such as requirements file 1004B. In at least some embodiments, the generation unit 1002A includes subsections for performing additional functions, such as those described in the flowchart above. In at least some embodiments, such subsections are referred to by names associated with the corresponding functions.

[0077] The update unit 1002B is a circuit or instruction of the controller 1002 configured to update the vehicle application machine learning model. In at least some embodiments, the update unit 1002B is configured to train the vehicle application machine learning model using a first portion of a plurality of training samples. In at least some embodiments, the update unit 1002B utilizes information in storage 1004, such as sample 1004A and requirements file 1004B, and records information in storage 1004, such as model 1004C. In at least some embodiments, the update unit 1002B includes subsections for performing additional functions, such as those described in the flowchart above. In at least some embodiments, such subsections are referred to by names associated with the corresponding functions.

[0078] The deployment unit 1002C is a circuit or instruction of the controller 1002 configured to deploy a vehicle application machine learning model. In at least some embodiments, the deployment unit 1002C is configured to deploy the vehicle application machine learning model to the vehicle system in response to a determination that the vehicle application machine learning model meets the compliance requirements of the vehicle application. In at least some embodiments, the update unit 1002B utilizes information in storage 1004, such as model 1004C. In at least some embodiments, the deployment unit 1002C includes subsections for performing additional functions as described in the flowchart above. In at least some embodiments, such subsections are referred to by names associated with the corresponding functions.

[0079] The receiver 1002D is a circuit or instruction of the controller 1002 configured to receive application logs and other output logs. In at least some embodiments, the receiver 1002D is configured to receive output logs of a vehicle application machine learning model from the vehicle system. In at least some embodiments, the receiver 1002D records information in storage 1004, such as log 1004D. In at least some embodiments, the receiver 1002D includes subsections for performing additional functions, such as those described in the flowchart above. In at least some embodiments, such subsections are referred to by names associated with the corresponding functions.

[0080] In at least some embodiments, the device is another device capable of processing logical functions to perform the operations described herein. In at least some embodiments, the controller and the storage unit do not need to be entirely separate devices, and in some embodiments they share circuitry or one or more computer-readable media. In at least some embodiments, the storage unit includes a hard drive that stores both computer-executable instructions and data accessed by the controller, and the controller includes a combination of a central processing unit (CPU) and RAM, in which computer-executable instructions can be duplicated whole or partially for execution by the CPU during the performance of the operations described herein.

[0081] In at least some embodiments where the device is a computer, a program installed on the computer may cause the computer to function as the device of the embodiments described herein, or to perform operations associated with the device of the embodiments described herein. In at least some embodiments, such a program may be processor-executable and cause the computer to perform predetermined operations associated with some or all of the blocks of the flowcharts and block diagrams described herein.

[0082] At least some embodiments are described with reference to flowcharts and block diagrams in which blocks represent (1) steps in a process in which an operation is performed or (2) sections of a controller responsible for performing an operation. In at least some embodiments, given steps and sections are implemented by dedicated circuits, programmable circuits supplied with computer-readable instructions stored on a computer-readable medium, and / or processors supplied with computer-readable instructions stored on a computer-readable medium. In at least some embodiments, the dedicated circuits include digital and / or analog hardware circuits, including integrated circuits (ICs) and / or discrete circuits. In at least some embodiments, the programmable circuits include reconfigurable hardware circuits, such as field-programmable gate arrays (FPGAs), programmable logic arrays (PLAs), etc., which include logical AND, OR, XOR, NAND, NOR and other logic operations, flip-flops, registers, memory elements, etc.

[0083] In at least some embodiments, a computer-readable storage medium includes a tangible device capable of holding and storing instructions used by an instruction execution device. In some embodiments, the computer-readable storage medium includes, for example, but not limited to, electronic storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination thereof. A non-exhaustive list of more specific examples of computer-readable storage mediums includes portable computer diskettes, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disk read-only memory (CD-ROM), digital multipurpose disks (DVDs), memory sticks, floppy disks, mechanically encoded devices such as punch cards or grooved raised structures having instructions recorded thereon, and any suitable combination thereof. When used herein, computer-readable storage media should not be construed as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses passing through optical fiber cables), or transient signals themselves, such as electrical signals transmitted through wires.

[0084] In at least some embodiments, the computer-readable program instructions described herein are downloadable from a computer-readable storage medium to each computing / processing device, or downloadable to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. In at least some embodiments, the network includes copper transmission cables, optical transmission fibers, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. In at least some embodiments, a network adapter card or network interface within each computing / processing device receives computer-readable program instructions from the network and transfers the computer-readable program instructions for storage in a computer-readable storage medium within each computing / processing device.

[0085] In at least some embodiments, computer-readable program instructions for performing the operations described above are assembler instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++ or equivalents, and conventional procedural programming languages ​​such as the C programming language or similar programming languages. In at least some embodiments, computer-readable program instructions are fully executed on the user's computer, partially executed on the user's computer, partially executed on the user's computer and partially executed on a remote computer as a standalone software package, or fully executed on a remote computer or server. In at least some embodiments, in the latter scenario, the remote computer is connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection is made to an external computer (for example, through the Internet using an Internet Service Provider). In at least some embodiments, for example, an electronic circuit including a programmable logic circuit, a field-programmable gate array (FPGA), or a programmable logic array executes a computer-readable program instruction by individualizing the electronic circuit using state information of the computer-readable program instruction in order to carry out an aspect of the present invention.

[0086] While embodiments of the present invention have been described, the technical scope of any subject matter of the claims is not limited to the embodiments described above. Those skilled in the art will understand that various modifications and improvements are possible to the embodiments described above. Furthermore, those skilled in the art will understand from the claims that such modified or improved embodiments fall within the technical scope of the present invention.

[0087] The operations, procedures, steps, and stages of each process performed by the apparatus, systems, programs, and methods shown in the claims, embodiments, or drawings can be performed in any order, unless the order is indicated by “before” or similar and the output from a previous process is not used in a later process. Even when the flow of a process is described in the claims, embodiments, or drawings using phrases such as “first” or “next,” such description does not necessarily mean that the processes must be performed in the order described.

[0088] In at least some embodiments, the continuous updating of the driving system for incident avoidance is performed by the steps of: collecting multiple incident samples from the internet, wherein the multiple incident samples are identified by an identification machine learning model so as to be related to one or more vehicles; clustering the multiple incident samples into multiple incident clusters by a clustering machine learning model; and defining compliance requirements for the vehicle application according to the incident clusters among the multiple incident clusters by a requirements definition machine learning model.

[0089] In at least some embodiments, the step of defining compliance requirements for a vehicle application includes defining the type of metric, the criteria for the metric, and the conditions for evaluating the metric. In at least some embodiments, the continuous updating of the driving system for incident avoidance further includes the step of preparing multiple training samples in accordance with the compliance requirements of the vehicle application. In at least some embodiments, the preparation step includes defining annotation rules in accordance with the compliance requirements of the vehicle application, labeling multiple sensor samples in accordance with the annotation rules, and selecting multiple training samples from the labeled sensor samples. In at least some embodiments, the continuous updating of the driving system for incident avoidance further includes the step of training a vehicle application machine learning model using a first portion of the multiple training samples. In at least some embodiments, the continuous updating of the driving system for incident avoidance further includes the step of testing the vehicle application machine learning model using a second portion of the multiple training samples in accordance with the compliance requirements of the vehicle application. In at least some embodiments, the continuous updating of the driving system for incident avoidance further includes the step of determining whether the vehicle application machine learning model meets the compliance requirements of the vehicle application. In at least some embodiments, the continuous updating of the driving system for incident avoidance further includes the step of deploying the vehicle application machine learning model to the vehicle system in response to a determination that the vehicle application machine learning model meets the compliance requirements of the vehicle application. In at least some embodiments, the continuous updating of the driving system for incident avoidance further includes the steps of receiving output logs of the vehicle application machine learning model from the vehicle system, analyzing the output logs to validate the vehicle application machine learning model, and updating the compliance requirements of the vehicle application in accordance with the results of the analysis.In at least some embodiments, a vehicle application machine learning model is configured for scene classification, the output log includes multiple scene classification results, and the step of receiving the output log is performed depending on whether the vehicle application machine learning model has classified the scenes into predetermined classes. In at least some embodiments, the prepare step includes receiving sensor data logs corresponding to multiple sensor samples. In at least some embodiments, the compliance requirements for the vehicle application include a training set identifier. In at least some embodiments, the continuous updating of the driving system for incident avoidance further includes the step of assigning a priority value to each incident cluster among multiple incident clusters, and the define step is performed depending on the determination that the priority value assigned to an incident cluster exceeds a priority threshold. In at least some embodiments, the priority value is based on the size of the incident cluster. In at least some embodiments, the compliance requirements for the vehicle application are further defined according to vehicle type constraints. In at least some embodiments, multiple incident samples include natural language text. In at least some embodiments, the collect step includes applying multiple incident samples to a large language model. In at least some embodiments, the compliance requirements for the vehicle application include computer-readable structured data.

[0090] In at least some embodiments, continuous updates of the operating system for incident avoidance are performed by a device comprising a processor that executes instructions in accordance with the aforementioned operations, or a controller that includes circuits configured to perform the aforementioned operations.

[0091] The above description outlines the features of several embodiments so that those skilled in the art may better understand aspects of the disclosure. Those skilled in the art should understand that the disclosure can be readily used as a basis for designing or modifying other processes and structures to perform the same purposes and / or achieve the same advantages as the embodiments introduced herein. Furthermore, those skilled in the art should recognize that such equivalent configurations do not depart from the spirit and scope of the disclosure, and that various changes, substitutions and modifications are possible herein without departing from the spirit and scope of the disclosure.

Claims

1. A step of collecting multiple incident samples from the internet, wherein the multiple incident samples are identified by an identification machine learning model so as to be related to one or more vehicles, The steps include: clustering the multiple incident samples into multiple incident clusters using a clustering machine learning model; The requirements definition machine learning model defines compliance requirements for the vehicle application according to the incident cluster among the multiple incident clusters, and A computer program that causes one or more processors to execute.

2. The computer program according to claim 1, wherein the step of defining compliance requirements for the vehicle application includes defining the type of metric, the criteria for the metric, and the evaluation conditions for the metric.

3. The computer program according to claim 1 or 2, further comprising the step of causing one or more processors to perform the step of preparing a plurality of training samples in accordance with the compliance requirements of the vehicle application.

4. The steps to be prepared as described above are: Define annotation rules in accordance with the compliance requirements of the aforementioned vehicle application, Labeling multiple sensor samples according to the annotation rules, Selecting the multiple training samples from the multiple labeled sensor samples. The computer program according to claim 3, including the computer program described in claim 3.

5. The computer program according to claim 4, further causing one or more processors to perform the step of training a vehicle application machine learning model using a first portion of the plurality of training samples.

6. The computer program according to claim 5, further comprising the step of causing one or more processors to perform a step of testing the vehicle application machine learning model using a second portion of the plurality of training samples in accordance with the compliance requirements of the vehicle application.

7. The computer program according to claim 6, further comprising the step of causing one or more processors to perform the step of determining whether the vehicle application machine learning model satisfies the compliance requirements of the vehicle application.

8. The computer program according to claim 7, further causing one or more processors to perform the step of deploying the vehicle application machine learning model to a vehicle system in response to a determination that the vehicle application machine learning model satisfies the compliance requirements of the vehicle application.

9. The steps include receiving the output log of the vehicle application machine learning model from the vehicle system, The steps include: analyzing the output log to verify the vehicle application machine learning model, The steps include updating the compliance requirements for the vehicle application in accordance with the results of the analysis, and The computer program according to claim 8, further causing one or more processors to execute the above.

10. The aforementioned vehicle application machine learning model is configured for scene classification. The output log includes multiple scene classification results, The computer program according to claim 9, wherein the step of receiving the output log is performed in accordance with the vehicle application machine learning model classifying the scene into a predetermined class.

11. The computer program according to claim 4, wherein the preparation step includes receiving sensor data logs corresponding to the plurality of sensor samples.

12. The computer program according to claim 1 or 2, wherein the compliance requirements for the vehicle application include a training set identifier.

13. The step of assigning a priority value to each incident cluster among the plurality of incident clusters is further performed by one or more processors, The computer program according to claim 1 or 2, wherein the defined step is performed in response to a determination that the priority value assigned to the incident cluster exceeds a priority threshold.

14. The computer program according to claim 13, wherein the priority value is based on the size of the incident cluster.

15. The computer program according to claim 1 or 2, wherein the compliance requirements for the vehicle application are further defined according to vehicle type constraints.

16. The computer program according to claim 1 or 2, wherein the plurality of incident samples include natural language text.

17. The computer program according to claim 1 or 2, wherein the step of collecting the incident samples includes applying the plurality of incident samples to a large-scale language model.

18. The computer program according to claim 1 or 2, wherein the compliance requirements for the vehicle application include structured data in a computer-readable format.

19. A method that is executed by one or more processors, A step of collecting multiple incident samples from the internet, wherein the multiple incident samples are identified by an identification machine learning model so as to be related to one or more vehicles, The steps include: clustering the multiple incident samples into multiple incident clusters using a clustering machine learning model; The requirements definition machine learning model defines compliance requirements for the vehicle application according to the incident cluster among the multiple incident clusters, and Methods that include...

20. A step of collecting multiple incident samples from the internet, wherein the multiple incident samples are identified by an identification machine learning model so as to be related to one or more vehicles, The steps include: clustering the multiple incident samples into multiple incident clusters using a clustering machine learning model; The requirements definition machine learning model defines compliance requirements for the vehicle application according to the incident cluster among the multiple incident clusters, and A device comprising a controller including a circuit configured to perform a certain action.

Images