Automated vehicle test system based on requirements written in natural language

Abstract

To provide a method.SOLUTION: A method, a system, and a device for automated vehicle tests are provided. The method comprises: generating a ticket, wherein the ticket comprises at least one natural language test requirement and at least one natural language incident scenario description; determining whether the sensor data collected from a vehicle matches the at least one natural language incident scenario description in the ticket; generating a requirement file (RaC file) based on the ticket and the sensor data collected from the vehicle based on a determination that the sensor data collected from the vehicle matches the at least one natural language incident scenario description; and evaluating an ML model based on the RaC file to determine whether the ML model meets the test requirements, wherein the ML model is used to implement a vehicle application.SELECTED DRAWING: Figure 4

Description

[Technical field]

[0001] Systems and methods consistent with exemplary embodiments of the present disclosure relate to automated vehicle testing based on requirements written in natural language. [Background technology]

[0002] In the related art, testing of automated vehicle systems may be implemented using machine learning (ML) models that may be used to automate various tasks related to the operation of the automated vehicle. For this purpose, it is necessary to collect test data to test the vehicle application.

[0003] Typically, the vehicle may be implemented with sensors (e.g., cameras, accelerometers, etc.) to collect data, which the operator / manufacturer may then process for testing purposes. To this end, the data collected by the sensors may be input into a vehicle application (which may be implemented, for example, using an ML model).

[0004] In the related art, data is collected from a vehicle and input to a test vehicle application based on a rule-based trigger condition. In particular, the rule-based trigger condition requires an explicit, definitive, and clear definition of the conditions under which sensor data is collected from the vehicle to a database / server. The rule-based trigger condition needs to be clearly specified with numerical criteria and strict conditional branching, and therefore cannot contain any ambiguity. Such strict conditions for collecting data may result in lack of collection of data required for testing, which may lead to false positives / false negatives during collection.

[0005] Therefore, there is a need for a system for collecting data that can allow test requirements to be specified in a less rigid manner. Summary of the Invention

[0006] According to one or more exemplary embodiments, an apparatus and method for automated vehicle testing based on natural language is provided. In particular, a ticket for an issue management system may be generated based on test requirements and incident scenario descriptions written in natural language. Based on a match of the ticket to sensor data collected from a vehicle, a coded file (e.g., a Requirement as Code (RaC) file) may be generated, and a machine learning (ML) model (which may implement a vehicle application) may be evaluated based on the coded file for whether the ML model meets the test requirements. Thus, since the test requirements and incident scenarios are written in a human-readable format and then converted into a coded file, an operator / developer may easily and broadly specify conditions and requirements under which data may be collected by specifying the test requirements and incident scenarios in a natural language that allows for ambiguity. Thus, the conditions for collecting data may not be too strict, and false positives / false negatives may be avoided.

[0007] According to an embodiment, a method for testing a vehicle application may be provided. The method may include: generating a ticket, the ticket comprising at least one human-readable test requirement and at least one natural language incident scenario description; determining whether sensor data collected from the vehicle matches the at least one human-readable incident scenario description in the ticket; and based on the determination that the sensor data collected from the vehicle matches the at least one human-readable incident scenario description, generating a requirements file (RaC file) based on the ticket and the sensor data collected from the vehicle, and evaluating an ML model based on the RaC file to determine whether the ML model achieves the test requirement, the ML model being used to implement the vehicle application. According to an embodiment, the ticket may be generated based on an incident record (IR) ticket written in natural language and a requirements description (RD) ticket written in natural language. The ticket may be stored in a ticket database, such as an issue management system (e.g., a system that may open and manage tickets when a new issue needs to be resolved), and the collected sensor data is obtained from a database of vehicle data. Determining whether the sensor data collected from the vehicle matches the description in the ticket may be done using image classifiers and / or neural networks.

[0008] According to an embodiment, the RaC file may include a ticket identifier, a file identifier, the coded test requirements, and a link to the collected sensor data. Generating the RaC file may include converting at least one human readable test requirement into the coded test requirements.

[0009] According to an embodiment, once the evaluation of the ML model based on the RaC file is completed, the vehicle application is deployed in the vehicle. Once the evaluation of the ML model based on the RaC file is completed, the status of the ticket may be updated based on the results of the evaluation.

[0010] According to an embodiment, once generation of the ticket is complete, the ticket is received by a ticket receiver in the vehicle, and an image classifier and / or neural network is implemented in the vehicle. Based on a determination that the sensor data collected from the vehicle matches at least one human readable incident scenario description, the collected sensor data may be transmitted by a data transmitter in the vehicle to a database of vehicle data.

[0011] Additional aspects will be set forth in part in the description that follows, and in part will be apparent from the description, or may be realized by practice of the presented embodiments of the present disclosure. [Brief description of the drawings]

[0012] Features, aspects, and advantages of certain preferred embodiments of the present disclosure are described below with reference to the accompanying drawings, in which like reference numerals refer to like elements, and in which: [Figure 1] FIG. 1 is a diagram of example components of a device according to an example embodiment. [Diagram 2] FIG. 2 is a system architecture diagram in accordance with one or more exemplary embodiments. [Diagram 3] FIG. 3 is an alternative system architecture diagram in accordance with one or more exemplary embodiments. [Figure 4] FIG. 4 is a flow chart diagram illustrating a method for testing vehicle applications based on tickets according to one or more exemplary embodiments. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] The following detailed description of the exemplary embodiments refers to the accompanying drawings. The present disclosure provides illustration and description, but is not intended to be exhaustive or to limit one or more exemplary embodiments to the precise form disclosed. Modifications and variations are possible in light of the disclosure or may be learned from the practice of one or more exemplary embodiments. Moreover, one or more features or components of one exemplary embodiment may be incorporated in or combined with another exemplary embodiment (or one or more features of another exemplary embodiment). Furthermore, in the flowcharts and descriptions of operations provided herein, it is understood that one or more operations may be omitted, one or more operations may be added, one or more operations may be performed (at least partially) simultaneously, and the order of one or more operations may be switched.

[0014] It will be apparent that the exemplary embodiments of the systems and / or methods and / or non-transitory computer-readable storage medium described herein may be implemented in various forms of hardware, firmware, or a combination of hardware and software. The actual dedicated control hardware or software code used to implement the systems and / or methods is not a limitation of one or more exemplary embodiments. Thus, the operation and behavior of the systems and / or methods and / or non-transitory computer-readable storage medium are described herein without reference to any specific software code. It is understood that software and hardware may be designed to implement the systems and / or methods based on the description herein.

[0015] Although a particular combination of features is recited in a claim and / or disclosed herein, such combination is not intended to limit the disclosure of possible example embodiments. Indeed, many of the features may be combined in ways not specifically recited in the claims and / or disclosed herein. Although each dependent claim listed below may depend directly on only one claim, the disclosure of possible example embodiments includes each dependent claim in combination with all other claims in the claim set.

[0016] No element, act, or instruction used herein should be construed as critical or required unless expressly stated otherwise. Additionally, the articles "a" and "an" as used herein are intended to include one or more items and may be used interchangeably with "one or more." When only one item is intended, the term "a" or similar terms are used. Additionally, the terms "has," "have," "having," "include," "including," or the like as used herein are intended to be open-ended terms. Additionally, the phrase "based on" is intended to mean "based at least in part on," unless expressly stated otherwise. Additionally, phrases such as "at least one of [A] and [B]" or "at least one of [A] or [B]" should be understood as including only A, only B, or both A and B.

[0017] 1 is a diagram of example components of a vehicle test device 100. As shown in FIG. 1, the vehicle test device 100 may include a bus 110, a processor 120, a memory 130, a storage unit 140, an input unit 150, an output unit 160, and a communication interface 170.

[0018] The bus 110 includes components that enable communication between components of the vehicle test device 100. The processor 120 may be implemented in hardware, firmware, or a combination of hardware and software. The processor 120 may be a central processing unit (CPU), a graphics processing unit (GPU), an accelerated processing unit (APU), a microprocessor, a microcontroller, a digital signal processor (DSP), a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), or another type of processing component. In one or more exemplary embodiments, the processor 120 includes one or more processors that are programmable to perform functions. The memory 130 includes a random access memory (RAM), a read only memory (ROM), and / or another type of dynamic or static storage device (e.g., flash memory, magnetic memory, and / or optical memory) that stores information and / or instructions for use by the processor 220.

[0019] The storage unit 140 stores information and / or software related to the operation and use of the vehicle test device 100. For example, the storage unit 140 may include a hard disk (e.g., a magnetic disk, an optical disk, a magneto-optical disk, and / or a solid state disk), a compact disk (CD), a digital versatile disk (DVD), a floppy disk, a cartridge, a magnetic tape, and / or another type of non-transitory computer-readable medium along with a corresponding drive. The input unit 150 includes components (e.g., a touch screen display, a keyboard, a keypad, a mouse, buttons, switches, and / or a microphone) that enable the vehicle test device 100 to receive information, such as via user input. Additionally or alternatively, the input unit 150 may include sensors (e.g., a global positioning system (GPS) component, an accelerometer, a gyroscope, and / or an actuator) that detect information. The output unit 160 includes components (e.g., a display, a speaker, and / or one or more light emitting diodes (LEDs)) that provide output information from the vehicle test device 100.

[0020] The communication interface 170 includes transceiver-like components (e.g., a transceiver and / or a separate receiver and transmitter) that enable the vehicle test device 100 to communicate with other devices, such as via a wired connection, a wireless connection, or a combination of wired and wireless connections. The communication interface 170 may enable the vehicle test device 100 to receive information from another device and / or provide information to another device. For example, the communication interface 170 may include, but is not limited to, an Ethernet interface, an optical interface, a coaxial interface, an infrared interface, a radio frequency (RF) interface, a universal serial bus (USB) interface, a Wi-Fi interface, a cellular network interface, or the like.

[0021] Vehicle test device 100 may perform one or more of the example processes described herein. According to one or more example embodiments, vehicle test device 100 may perform such processes in response to processor 120 executing software instructions stored by a non-transitory computer readable medium, such as memory 130 and / or storage unit 140. Computer readable media is defined herein as a non-transitory memory device. A memory device includes memory space within a single physical storage device or memory space spanning multiple physical storage devices.

[0022] The software instructions may be read into the memory 130 and / or storage unit 140 from another computer-readable medium or from another device via the communications interface 170. The software instructions stored in the memory 130 and / or storage unit 140, when executed, may cause the processor 120 to perform one or more of the processes described herein.

[0023] Additionally or alternatively, hardwired circuitry may be used in place of, or in combination with, software instructions to perform one or more of the processes described herein. Thus, one or more exemplary embodiments described herein are not limited to any specific combination of hardware circuitry and software.

[0024] The number and arrangement of components shown in Figure 1 are provided as an example. In fact, vehicle test device 100 may include additional, fewer, different, or differently arranged components than those shown in Figure 1. Additionally or alternatively, a set of components (e.g., one or more components) of vehicle test device 100 may perform one or more functions that are described as being performed by another set of components of vehicle test device 100.

[0025] 2 is a system architecture diagram according to one or more exemplary embodiments. In particular, FIG. 2 illustrates an embodiment in which a vehicle application implemented using a machine learning (ML) model is tested.

[0026] According to an embodiment, a vehicle 13 may be provided that communicates with a server 14. It should be understood that the server 14 may be implemented as a single server, multiple servers, or using a cloud application, depending on the particular implementation. An operator 15 and a product owner / developer / user 16 may be able to interact with the server 14.

[0027] The vehicle 13 may include at least one sensor 17 and a vehicle application 20. In particular, the at least one sensor 17 may include, but is not limited to, a camera, an accelerometer, a gyroscope, an IR sensor, etc. The at least one sensor 17 may be responsible for collecting data related to testing of the vehicle. The at least one sensor 17 may be capable of transmitting collected data to a database 3 of vehicle data located on the server 14. The vehicle application 20 may be any application related to the operation of the vehicle 13 (e.g., related to steering or acceleration of the vehicle 13). The vehicle application 20 may be implemented using machine learning (ML) models.

[0028] The server 14 may include an incident recording server 11 and a requirements management dashboard 12. The incident recording server 11 may include a collection of incident scenarios that may be input by an operator 15 in natural language. In particular, the natural language may be in any known human language, and according to some embodiments, it may be written in a structure that may be easily understood by a person who may not necessarily have knowledge on how to interpret the code. In other words, the natural language may encompass a human language that, when interpreted (e.g., when speaking or reading), would be understood by a normal person. An incident scenario may specify a situation in which the vehicle application 20 needs to make a decision on how to operate the vehicle 13, for example, specifying that there is an approaching vehicle within any horizontal and vertical distance from the two lanes on the left. Based on the collection of incident scenarios written to be human readable, the incident recording server may be capable of generating / storing / outputting an incident record (IR) ticket 4 that is also in natural language. It is to be understood that the incident scenario may include, but is not necessarily limited to, a real-life scenario or a test scenario.

[0029] The requirements management dashboard 12 (which in some embodiments may be implemented as its own server) may contain a set of test requirements that may be entered in natural language by a product owner / developer / user 16. The test requirements may specify test targets, test conditions, or test criteria for the operation of the vehicle 13. Based on the set of test requirements, the requirements management dashboard may be able to generate / store / output requirements description (RD) tickets 5 written in natural language. The incident scenarios may contain information about accidents / problems / failures involving the vehicle 13 for example.

[0030] Based on the IR ticket 4 and the RD ticket 5, a ticket 6 may be generated (e.g., by combining the IR ticket 4 and the RD ticket 5) and stored accordingly in a ticket database 7 in the server 14. Each ticket 6 may have its own unique identifier (ID). It should be understood that although a database is specified, other storage means (e.g., cloud storage, content delivery networks) may also be implemented for storing the tickets.

[0031] A sensor data classifier 1 may also be provided to the server 14. According to some embodiments where the collected sensor data is images, the sensor data classifier 1 may be an image classifier implemented utilizing a neural network 2. The sensor data classifier 1 may be used to compare the collected sensor data obtained from the database of vehicle data 3 to the ticket 6 to determine whether the particular collected sensor data matches the incident scenario description and / or test requirement description in the ticket 6. While an image classifier and / or a neural network may be used to perform this operation, it should be understood that other means of comparing the collected sensor data to the incident scenario description in the ticket 6 may be used.

[0032] Upon determining a match between the description of the ticket 6 and the collected sensor data from the vehicle data database 3, a link (i.e., a URL) to the matched data and the ticket 6 may be sent to a requirements file (RaC file) generator 8 located on the server 14, which may generate a RaC file 9 based on the link to the collected sensor data and the ticket 6.

[0033] The RaC file 9 may include a ticket ID (e.g., associated with ticket 6), a unique file ID (which may be used to uniquely identify the RaC file 9), test requirements (test targets, test conditions, test criteria, etc.) converted from the natural language written in the ticket 6 into code, and a link (URL) to the collected sensor data stored in the vehicle data database 3.

[0034] The RaC file 9 may be responsible for storing the expected behavior of the ML model. The expected behavior may comprise test requirements. In particular, the test requirements may specify the performance metrics of the ML model that need to be tested, for example, based on the numerical value of a particular performance metric that needs to be achieved, the criteria that need to be achieved, the type of test, etc. Accordingly, the RaC file 9 may include the following as test requirements: test parameters, test targets, URLs or file paths of test data, requirements, pass criteria, test conditions, acceptance requirements, and acceptance criteria that are used during the ML evaluation process. Such requirements, test targets, file paths of test data, pass criteria, and test conditions may be easily interpreted (e.g., by the ML evaluation pipeline 10) to determine how the ML testing and evaluation of the ML model should be performed by the ML evaluation pipeline and thereafter by the ML evaluation pipeline 10. For example, the RaC file 9 may include criteria in the form of code, which the ML evaluation pipeline may easily interpret into instructions on how to perform the ML evaluation of the ML model. The RaC file 9 may be in a format such as a YAML format or a domain-specific language (DSL) format. The RaC file 9 may also be in the format of a programming language such as PYTHON®. The RaC file 9 may be separate from the code used to actually perform the ML evaluation, thus eliminating the need to "hard-code" requirements into the ML evaluation process.

[0035] An ML evaluation pipeline 10 may be provided for testing the ML model based on the RaC file 9. The ML evaluation pipeline may include an interface for interpreting the requirements specified in the RaC file 9 and an interface for evaluating the ML model based on the interpreted requirements. The ML evaluation pipeline 10 may use a link (URL), file path, or unique key to the collected sensor data in the RaC file 9 to retrieve the collected sensor data from the database 3 of vehicle data during testing. Based on the results of the ML evaluation, the tested ML model may be used to generate a pre-deployment application 20-1, which may then be subsequently deployed in the vehicle 13 as a vehicle application 20. The ML evaluation pipeline 10 may also update the status of the ticket 6 in the ticket database 7 based on the results of the ML evaluation. Although the ML evaluation pipeline 10 is specified, it should be understood that any suitable means of interpreting the requirements from the RaC file and evaluating the ML model based on the interpreted requirements may be used according to an embodiment.

[0036] 3 is an alternative system architecture diagram according to one or more exemplary embodiments. In particular, FIG. 3 illustrates an exemplary embodiment in which vehicle data is collected and used to test vehicle applications accordingly. Similar components to FIG. 2 are included, and therefore redundant descriptions may be omitted for ease of reading.

[0037] In particular, with respect to the embodiment shown in Figure 2, the embodiment shown in Figure 3 provides a vehicle 13 further including a ticket receiver 31, a data transmitter 34, and a sensor data classifier 1 and / or a neural network 2 located in the vehicle 13 rather than in the server 14. The ticket receiver 31 may be capable of receiving a ticket 6 from a ticket database 7. It should be appreciated that according to some embodiments, the ticket 6 may be received in a compressed format, an encrypted format, or a format partially embedded in a feature map.

[0038] The sensor data classifier 1 and / or the neural network 2 may also be configured to determine whether the collected sensor data obtained from the at least one sensor 17 matches the description of the ticket 6. If so, the data transmitter 34 may be enabled to transmit the collected sensor data to the database 3 of vehicle data. When the server 14 receives the collected sensor data from the data transmitter 34, the URL of the received data may be transmitted to the RaC file generator 8. Thus, in the embodiment shown in FIG. 3, as the number of collected test data for testing the requirements or incident scenarios written in the ticket 6 increases, the number of tests corresponding to the ticket 6 also increases.

[0039] 4 is a flow chart diagram illustrating a method 400 for testing vehicle applications based on tickets according to one or more exemplary embodiments. It should be appreciated that, according to some embodiments, the system architectures illustrated in FIGS. 2 and 3 may be used to implement the method 400.

[0040] 4 , in operation S410, a ticket 6 may be generated that includes at least one test requirement written in a human readable form and at least one incident scenario description written in a human readable form. According to some embodiments, this may be done by combining an incident record (IR) ticket 4 written in a natural language and obtained from an incident record server 11, and a requirements description (RD) ticket 5 obtained from a requirements management dashboard 12. According to some embodiments, the ticket 6 may be stored in a ticket database 7.

[0041] According to some embodiments, the ticket 6 may be received by a ticket receiver 31 in the vehicle 13 .

[0042] At operation S420, a determination may be made as to whether collected sensor data from a vehicle (i.e., vehicle 13) matches at least one human readable incident scenario description and / or test requirement description in ticket 6. According to some embodiments, this may be done using a sensor data classifier 1 and / or a neural network 2. According to embodiments, the collected sensor data may be initially stored in a database 3 of vehicle data. According to some embodiments, the sensor data classifier 1 and / or the neural network 2 may be implemented in the vehicle.

[0043] In operation S430, a RaC file 9 may be generated based on the ticket 6 and the collected sensor data. According to an embodiment, operation S430 may be performed only if operation S420 determines that a match exists between the collected sensor data and the incident description scenario from the ticket 6. The RaC file 9 may include a ticket identifier (which may be the same as in the ticket 6), a file identifier, coded test requirements, and a link to the collected sensor data. The coded test requirements may be generated based on a translation of the human readable test requirements from the ticket 6. According to an embodiment, operation S430 may be performed by the RaC file generator 8.

[0044] According to some embodiments, based on a determination that the collected sensor data from the vehicle 13 matches at least one natural language incident scenario description and / or test requirements description, the collected sensor data may be transmitted by a data transmitter 34 in the vehicle 13 to a database 3 of vehicle data.

[0045] At operation S440, the ML model (which may be used to implement the vehicle application 20) may be evaluated based on the RaC file 9 as generated at operation S430. In particular, this may evaluate with respect to whether the ML model achieves the test requirements (as interpreted from the RaC file 9) originally specified in the ticket 6. Operation S440 may be implemented using the ML evaluation pipeline 10, which may interpret the requirements from the RaC file 9 and perform the evaluation based on the interpreted requirements. According to some embodiments, once operation S440 is completed, the vehicle application may be implemented and deployed within the vehicle 13. In some embodiments, once operation S440 is completed, the status of the ticket 6 may be updated as a result of the evaluation.

[0046] Based on the above embodiment, the conversion of human-readable test requirements and human-readable incident scenario descriptions into coded files allows operators / developers to easily and broadly specify the conditions and requirements under which data may be collected since it is in natural language before processing. Therefore, the conditions for collecting data may be less strict and false positives / false negatives may be avoided.

[0047] The foregoing disclosure provides illustration and description, but is not intended to be exhaustive or to limit the exemplary embodiment or embodiments to the precise forms disclosed. Modifications and variations are possible in light of the disclosure or may be acquired from practice of the exemplary embodiment or embodiments.

[0048] One or more exemplary embodiments may relate to a system, method, and / or computer-readable medium at any possible level of technical detail of integration. Furthermore, one or more of the above-mentioned components may be implemented as instructions stored in a computer-readable medium and executable by at least one processor (and / or may include at least one processor). The computer-readable medium may include a computer-readable non-transitory storage medium (or media) having computer-readable program instructions that cause a processor to perform operations.

[0049] A computer-readable storage medium may be a tangible device that can hold and store instructions used by an instruction execution device. A computer-readable storage medium may be, for example, but not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the above. A non-exhaustive list of more specific examples of computer-readable storage media includes the following: portable computer diskettes, hard disks, random access memories (RAMs), read-only memories (ROMs), erasable programmable read-only memories (EPROMs or flash memories), static random access memories (SRAMs), portable compact disk read-only memories (CD-ROMs), digital versatile disks (DVDs), memory sticks, floppy disks, mechanically encoded devices such as punch cards or ridge structures in grooves with instructions recorded on them, and any suitable combination of the above. As used herein, computer-readable storage media should not be construed as being transitory signals per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through a waveguide or other transmission medium (e.g., light pulses passing through a fiber optic cable), or electrical signals transmitted over wires.

[0050] The computer-readable program instructions described herein may be downloaded from a computer-readable storage medium into each computing / processing device, or may be downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network may comprise copper transmission cables, optical transmission fiber, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in a computer-readable storage medium in the respective computing / processing device.

[0051] The computer readable program code / instructions for performing the operations may be assembler instructions, instruction set architecture (ISA) instructions, machine instructions, machine dependent instructions, microcode, firmware instructions, state setting data, configuration data for an integrated circuit, or source or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, or the like, and procedural programming languages ​​such as the "C" programming language or similar programming languages. The computer readable program instructions may be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or wide area network (WAN), or a connection to an external computer may be made (e.g., through the Internet using an Internet Service Provider). In one or more exemplary embodiments, an electronic circuit including, for example, a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA) may execute computer-readable program instructions by utilizing state information of the computer-readable program instructions to personalize the electronic circuit to perform an aspect or operation.

[0052] The computer-readable program instructions may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions executed by the processor of the computer or other programmable data processing apparatus produce means for implementing the function / act specified in the block or blocks of the flowcharts and / or block diagrams. The computer-readable program instructions may also be stored in a computer-readable storage medium that may direct a computer, programmable data processing apparatus, and / or other device to function in a particular manner, such that the computer-readable storage medium having instructions stored therein comprises an article of manufacture including instructions that implement an aspect of the function / act specified in the block or blocks of the flowcharts and / or block diagrams.

[0053] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause the computer, other programmable apparatus, or other device to perform a series of operational steps to generate a computer-implemented process, such that the instructions executing on the computer, other programmable apparatus, or other device implement the function / act specified in the block or blocks of the flowcharts and / or block diagrams.

[0054] The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible example embodiments of the systems, methods, and computer-readable media according to one or more exemplary embodiments. In this regard, each block in the flowcharts or block diagrams may represent a portion of a microservice, a module, a segment, or an instruction, comprising one or more executable instructions that implement the specified logical function. The methods, computer systems, and computer-readable media may include additional, fewer, different, or differently arranged blocks compared to those depicted in the figures. In one or more alternative exemplary embodiments, the functions noted in the blocks may occur out of the order noted in the figures. For example, two blocks shown in succession may in fact be executed simultaneously or substantially simultaneously, or the blocks may be executed in the reverse order depending on the functionality involved. It should also be noted that each block of the block diagrams and / or flowchart diagrams, as well as combinations of blocks in the block diagrams and / or flowchart diagrams, may be implemented by a dedicated hardware-based system that performs the specified functions or acts or executes a combination of dedicated hardware and computer instructions.

[0055] It will be apparent that the systems and / or methods described herein may be implemented in various forms of hardware, firmware, or a combination of hardware and software. The actual dedicated control hardware or software code used to implement the systems and / or methods is not a limitation of one or more exemplary embodiments. Thus, the operation and behavior of the systems and / or methods are described herein without reference to specific software code, and it will be understood that software and hardware can be designed to implement the systems and / or methods based on the description herein.

Claims

1. 1. A method for testing a vehicle application, the method comprising: generating a ticket, the ticket comprising at least one natural language test requirement description and / or at least one natural language incident scenario description; determining whether sensor data collected from the vehicle matches at least one human readable incident scenario description or natural language test requirement description in the ticket; based on a determination that the collected sensor data from the vehicle matches the at least one human readable incident scenario description or natural language test requirements description; generating a requirements file (RaC file) based on the ticket and sensor data collected from the vehicle; and evaluating an ML model based on the RaC file to determine whether the ML model achieves the test requirements, wherein the ML model is used to implement a vehicle application; and A method comprising:

2. The method of claim 1 , wherein the ticket is generated based on an incident record (IR) ticket written in a natural language and a requirements description (RD) ticket written in a natural language.

3. The method of claim 1 or 2, wherein the ticket is stored in a ticket database and the collected sensor data is obtained from a database of vehicle data.

4. The method of claim 1 or 2, wherein determining whether sensor data collected from a vehicle matches a description in the ticket is performed using a sensor data classifier and / or a neural network.

5. The method of claim 1 or 2, wherein the RaC file comprises a ticket identifier, a file identifier, coded test requirements, and a link to the collected sensor data.

6. The method of claim 5 , wherein generating the RaC file comprises converting the at least one natural language test requirements description into coded test requirements.

7. The method of claim 1 or 2, wherein upon completion of evaluation of the ML model based on the RaC file, the vehicle application is deployed in the vehicle.

8. The method of claim 1 or 2, wherein upon completion of an evaluation of the ML model based on the RaC file, the status of the ticket is updated based on a result of the evaluation.

9. The method of claim 4 , wherein once generation of the ticket is complete, the ticket is received by a ticket receiver in the vehicle, and the sensor data classifier and / or the neural network are implemented in the vehicle.

10. 10. The method of claim 9, wherein based on a determination that the collected sensor data from the vehicle matches the at least one natural language incident scenario description, the collected sensor data is transmitted by a data transmitter in the vehicle to a database of vehicle data.

11. 1. An apparatus for testing a vehicle application, the apparatus comprising: at least one memory storing computer executable instructions; At least one processor; wherein the at least one processor executes the computer-executable instructions to generating a ticket, the ticket comprising at least one natural language test requirement description and / or at least one natural language incident scenario description; determining whether sensor data collected from the vehicle matches at least one human readable incident scenario description or natural language test requirement description in the ticket; based on a determination that the collected sensor data from the vehicle matches the at least one human readable incident scenario description or natural language test requirements description; generating a requirements file (RaC file) based on the ticket and sensor data collected from the vehicle; and evaluating an ML model based on the RaC file to determine whether the ML model achieves the test requirements, wherein the ML model is used to implement a vehicle application; and The apparatus is configured to:

12. The apparatus of claim 11 , wherein the ticket is generated based on an incident record (IR) ticket written in a natural language and a requirements description (RD) ticket written in a natural language.

13. 13. Apparatus according to claim 11 or 12, wherein the tickets are stored in a ticket database and the collected sensor data is obtained from a database of vehicle data.

14. 13. The apparatus of claim 11 or 12, wherein determining whether sensor data collected from a vehicle matches a description in the ticket is performed using a sensor data classifier and / or a neural network.

15. 13. The apparatus of claim 11 or 12, wherein the RaC file comprises a ticket identifier, a file identifier, coded test requirements, and a link to the collected sensor data.

16. 16. The apparatus of claim 15, wherein the at least one processor is further configured to execute the computer-executable instructions to generate the RaC file by converting the at least one natural language test requirements description into coded test requirements.

17. The apparatus of claim 11 or 12, wherein upon completion of evaluation of the ML model based on the RaC file, the vehicle application is deployed in the vehicle.

18. The apparatus of claim 11 or 12, wherein upon completion of an evaluation of the ML model based on the RaC file, a status of the ticket is updated based on a result of the evaluation.

19. 15. The apparatus of claim 14, wherein once generation of the ticket is complete, the ticket is received by a ticket receiver in the vehicle, and the sensor data classifier and / or the neural network are implemented in the vehicle.

20. 20. The apparatus of claim 19, wherein based on a determination that the collected sensor data from the vehicle matches the at least one natural language incident scenario description, the collected sensor data is transmitted by a data transmitter in the vehicle to a database of vehicle data.

Images