Automated vehicle test system based on requirements written in natural language

Abstract

To provide a method.SOLUTION: A method, a system, and a device for automated vehicle tests are provided. The method comprises: generating a ticket, wherein the ticket comprises at least one natural language test requirement and at least one natural language incident scenario description; determining whether the sensor data collected from a vehicle matches the at least one natural language incident scenario description in the ticket; generating a requirement file (RaC file) based on the ticket and the sensor data collected from the vehicle based on a determination that the sensor data collected from the vehicle matches the at least one natural language incident scenario description; and evaluating an ML model based on the RaC file to determine whether the ML model meets the test requirements, wherein the ML model is used to implement a vehicle application.SELECTED DRAWING: Figure 4

Description

[Technical Field]

[0001] Systems and methods consistent with exemplary embodiments of the present disclosure relate to automated vehicle testing based on requirements written in natural language. [Background technology]

[0002] In the related art, testing of automated vehicle systems can be implemented using machine learning (ML) models that can be used to automate various tasks related to the operation of the automated vehicle. For this purpose, it is necessary to collect test data to test the vehicle application.

[0003] Typically, the vehicle may be equipped with sensors (e.g., cameras, accelerometers, etc.) to collect data, which the operator / manufacturer may then process for testing purposes. To this end, the data collected by the sensors may be input into a vehicle application (which may, for example, be implemented using an ML model).

[0004] In related art, data is collected from a vehicle and input into a test vehicle application based on rule-based trigger conditions. In particular, the rule-based trigger conditions require explicit, definitive, and clear definitions of the conditions under which sensor data is collected from the vehicle to a database / server. The rule-based trigger conditions must be clearly specified with numerical criteria and strict conditional branching, and therefore cannot contain any ambiguity. Such strict conditions for collecting data may result in the lack of collection of data necessary for testing, leading to false positives / false negatives during collection.

[0005] Therefore, there is a need for a system for collecting data that can allow test requirements to be specified in a less rigid manner. Summary of the Invention

[0006] According to one or more exemplary embodiments, an apparatus and method for natural language-based automated vehicle testing are provided. In particular, a ticket for an issue management system may be generated based on test requirements and incident scenario descriptions written in natural language. Based on a match of the ticket to sensor data collected from the vehicle, a coded file (e.g., a Requirements as Code (RaC) file) may be generated, and a machine learning (ML) model (which may implement a vehicle application) may be evaluated based on the coded file to determine whether the ML model conforms to the test requirements. Therefore, because the test requirements and incident scenarios are written in a human-readable format and then converted into the coded file, an operator / developer may easily and broadly specify the conditions and requirements under which data may be collected by specifying the test requirements and incident scenarios in natural language that allows for ambiguity. Therefore, the conditions for collecting data may not be too strict, and false positives / false negatives may be avoided.

[0007] According to an embodiment, a method for testing a vehicle application may be provided. The method may include generating a ticket, the ticket comprising at least one human-readable test requirement and at least one natural-language incident scenario description; determining whether sensor data collected from a vehicle matches the at least one human-readable incident scenario description in the ticket; and, based on a determination that the sensor data collected from the vehicle matches the at least one human-readable incident scenario description, generating a requirements file (RaC file) based on the ticket and the sensor data collected from the vehicle, and evaluating an ML model based on the RaC file to determine whether the ML model achieves the test requirement, wherein the ML model is used to implement the vehicle application. According to an embodiment, the ticket may be generated based on an incident record (IR) ticket written in natural language and a requirements description (RD) ticket written in natural language. The ticket may be stored in a ticket database, such as an issue management system (e.g., a system that can open and manage tickets when new issues need to be resolved), and the collected sensor data is retrieved from a database of vehicle data. Determining whether the sensor data collected from the vehicle matches the description on the ticket may be done using an image classifier and / or a neural network.

[0008] According to an embodiment, the RaC file may include a ticket identifier, a file identifier, coded test requirements, and a link to the collected sensor data. Generating the RaC file may include converting at least one human-readable test requirement into coded test requirements.

[0009] According to an embodiment, once evaluation of the ML model based on the RaC file is complete, the vehicle application is deployed in the vehicle. Once evaluation of the ML model based on the RaC file is complete, the status of the ticket may be updated based on the results of the evaluation.

[0010] According to an embodiment, once ticket generation is complete, the ticket may be received by a ticket receiver in the vehicle, and an image classifier and / or neural network may be implemented in the vehicle. Based on a determination that the sensor data collected from the vehicle matches at least one human-readable incident scenario description, the collected sensor data may be transmitted by a data transmitter in the vehicle to a database of vehicle data.

[0011] Additional aspects will be set forth in part in the description that follows, and in part will be apparent from the description, or may be realized by practice of presented embodiments of the present disclosure. [Brief explanation of the drawings]

[0012] Features, aspects, and advantages of certain preferred embodiments of the present disclosure are described below with reference to the accompanying drawings, in which like reference numerals refer to like elements. [Figure 1] FIG. 1 is a diagram of exemplary components of a device according to an exemplary embodiment. [Figure 2] FIG. 2 is a system architecture diagram according to one or more exemplary embodiments. [Figure 3] FIG. 3 is an alternative system architecture diagram in accordance with one or more exemplary embodiments. [Figure 4] FIG. 4 is a flowchart illustrating a method for testing vehicle applications based on tickets according to one or more exemplary embodiments. DETAILED DESCRIPTION OF THE INVENTION

[0013] The following detailed description of exemplary embodiments refers to the accompanying drawings. While the present disclosure provides illustration and description, it is not intended to be exhaustive or to limit one or more exemplary embodiments to the precise form disclosed. Modifications and variations are possible in light of the disclosure or may be learned from the practice of one or more exemplary embodiments. Moreover, one or more features or components of one exemplary embodiment may be incorporated into or combined with another exemplary embodiment (or one or more features of another exemplary embodiment). Furthermore, in the flowcharts and descriptions of operations provided herein, it is understood that one or more operations may be omitted, one or more operations may be added, one or more operations may occur (at least partially) concurrently, or the order of one or more operations may be switched.

[0014] It will be apparent that exemplary embodiments of the systems and / or methods and / or non-transitory computer-readable storage media described herein may be implemented in various forms of hardware, firmware, or a combination of hardware and software. The actual specialized control hardware or software code used to implement the systems and / or methods is not a limitation of one or more exemplary embodiments. Thus, the operation and behavior of the systems and / or methods and / or non-transitory computer-readable storage media will be described herein without reference to specific software code. It will be understood that software and hardware can be designed to implement the systems and / or methods based on the description herein.

[0015] Although particular combinations of features are recited in the claims and / or disclosed herein, such combinations are not intended to limit the disclosure of possible example embodiments. Indeed, many of the features may be combined in ways not specifically recited in the claims and / or disclosed herein. Although each dependent claim listed below may depend directly on only one claim, the disclosure of possible example embodiments includes each dependent claim in combination with all other claims in the claim set.

[0016] No element, act, or instruction used herein should be construed as critical or required unless expressly stated otherwise. Also, as used herein, the articles "a" and "an" are intended to include one or more items and may be used interchangeably with "one or more." Where only one item is intended, the term "one" or similar terms are used. Also, as used herein, the terms "has," "have," "having," "include," "including," and the like are intended to be open-ended terms. Furthermore, the phrase "based on" is intended to mean "based at least in part on," unless expressly stated otherwise. Furthermore, phrases such as "at least one of [A] and [B]" or "at least one of [A] or [B]" should be understood to include A only, B only, or both A and B.

[0017] 1 is a diagram of example components of a vehicle test device 100. As shown in FIG. 1, the vehicle test device 100 may include a bus 110, a processor 120, a memory 130, a storage unit 140, an input unit 150, an output unit 160, and a communication interface 170.

[0018] Bus 110 includes components that enable communication between components of vehicle test device 100. Processor 120 may be implemented in hardware, firmware, or a combination of hardware and software. Processor 120 may be a central processing unit (CPU), a graphics processing unit (GPU), an accelerated processing unit (APU), a microprocessor, a microcontroller, a digital signal processor (DSP), a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), or another type of processing component. In one or more exemplary embodiments, processor 120 includes one or more processors that are programmable to perform functions. Memory 130 includes random access memory (RAM), read-only memory (ROM), and / or another type of dynamic or static storage device (e.g., flash memory, magnetic memory, and / or optical memory) that stores information and / or instructions for use by processor 220.

[0019] The storage unit 140 stores information and / or software related to the operation and use of the vehicle test device 100. For example, the storage unit 140 may include a hard disk (e.g., a magnetic disk, optical disk, magneto-optical disk, and / or solid-state disk), a compact disk (CD), a digital versatile disk (DVD), a floppy disk, a cartridge, magnetic tape, and / or another type of non-transitory computer-readable medium, along with a corresponding drive. The input unit 150 includes components (e.g., a touchscreen display, a keyboard, a keypad, a mouse, buttons, switches, and / or a microphone) that enable the vehicle test device 100 to receive information, such as via user input. Additionally or alternatively, the input unit 150 may include sensors (e.g., a global positioning system (GPS) component, an accelerometer, a gyroscope, and / or an actuator) that detect information. The output unit 160 includes components (e.g., a display, a speaker, and / or one or more light-emitting diodes (LEDs)) that provide output information from the vehicle test device 100.

[0020] The communication interface 170 includes transceiver-like components (e.g., a transceiver and / or a separate receiver and transmitter) that enable the vehicle test device 100 to communicate with other devices, such as via a wired connection, a wireless connection, or a combination of wired and wireless connections. The communication interface 170 may enable the vehicle test device 100 to receive information from and / or provide information to another device. For example, the communication interface 170 may include, but is not limited to, an Ethernet interface, an optical interface, a coaxial interface, an infrared interface, a radio frequency (RF) interface, a universal serial bus (USB) interface, a Wi-Fi interface, a cellular network interface, or the like.

[0021] Vehicle test device 100 may perform one or more of the example processes described herein. According to one or more example embodiments, vehicle test device 100 may perform the processes in response to processor 120 executing software instructions stored by a non-transitory computer-readable medium, such as memory 130 and / or storage unit 140. Computer-readable media are defined herein as non-transitory memory devices. Memory devices include memory space within a single physical storage device or memory space spanning multiple physical storage devices.

[0022] Software instructions may be loaded into memory 130 and / or storage unit 140 from another computer-readable medium or from another device via communications interface 170. The software instructions stored in memory 130 and / or storage unit 140, when executed, may cause processor 120 to perform one or more of the processes described herein.

[0023] Additionally or alternatively, hardwired circuitry may be used in place of or in combination with software instructions to perform one or more of the processes described herein. Thus, one or more exemplary embodiments described herein are not limited to any specific combination of hardware circuitry and software.

[0024] The number and arrangement of components shown in Figure 1 are provided as an example. In fact, vehicle test device 100 may include additional, fewer, different, or differently arranged components than those shown in Figure 1. Additionally or alternatively, a set of components (e.g., one or more components) of vehicle test device 100 may perform one or more functions that are described as being performed by another set of components of vehicle test device 100.

[0025] 2 is a system architecture diagram according to one or more exemplary embodiments. In particular, FIG. 2 illustrates an embodiment in which a vehicle application implemented using a machine learning (ML) model is tested.

[0026] According to an embodiment, a vehicle 13 may be provided that communicates with a server 14. It should be understood that the server 14 may be implemented as a single server, multiple servers, or using a cloud application, depending on the particular implementation. An operator 15 and a product owner / developer / user 16 may be able to interact with the server 14.

[0027] The vehicle 13 may include at least one sensor 17 and a vehicle application 20. In particular, the at least one sensor 17 may include, but is not limited to, a camera, an accelerometer, a gyroscope, an IR sensor, etc. The at least one sensor 17 may be responsible for collecting data related to vehicle testing. The at least one sensor 17 may be capable of transmitting the collected data to a database 3 of vehicle data located on the server 14. The vehicle application 20 may be any application related to the operation of the vehicle 13 (e.g., related to steering or acceleration of the vehicle 13). The vehicle application 20 may be implemented using a machine learning (ML) model.

[0028] The server 14 may include an incident recording server 11 and a requirements management dashboard 12. The incident recording server 11 may include a collection of incident scenarios that may be input by an operator 15 in natural language. In particular, the natural language may be in any known human language, and according to some embodiments, it may be written in a structure that can be easily understood by a person who may not necessarily have knowledge of how to interpret code. In other words, the natural language may encompass human language that, when interpreted (e.g., when speaking or reading), would be understood by a normal person. An incident scenario may specify a situation in which the vehicle application 20 needs to make a decision about how to operate the vehicle 13, for example, specifying that there is an approaching vehicle within any horizontal and vertical distance from the two left lanes. Based on the collection of incident scenarios written in a human-readable format, the incident recording server may be capable of generating / storing / outputting an incident record (IR) ticket 4, which may also be in natural language. It should be understood that an incident scenario may include, but is not necessarily limited to, a real-world scenario or a test scenario.

[0029] The requirements management dashboard 12 (which in some embodiments may be implemented as its own server) may contain a set of test requirements that may be entered in natural language by a product owner / developer / user 16. The test requirements may specify test targets, test conditions, or test criteria for the operation of the vehicle 13. Based on the set of test requirements, the requirements management dashboard may be able to generate / store / output requirements description (RD) tickets 5 written in natural language. Incident scenarios may include, for example, information about accidents, problems, or malfunctions involving the vehicle 13.

[0030] Based on the IR ticket 4 and the RD ticket 5, a ticket 6 may be generated (e.g., by combining the IR ticket 4 and the RD ticket 5) and stored accordingly in a ticket database 7 in the server 14. Each ticket 6 may have its own unique identifier (ID). Although a database is specified, it should be understood that other storage means (e.g., cloud storage, content delivery network) may also be implemented for storing tickets.

[0031] A sensor data classifier 1 may also be provided on server 14. According to some embodiments where the collected sensor data is images, sensor data classifier 1 may be an image classifier implemented using a neural network 2. Sensor data classifier 1 may be used to compare collected sensor data obtained from database of vehicle data 3 with ticket 6 to determine whether particular collected sensor data matches the incident scenario description and / or test requirement description in ticket 6. While an image classifier and / or a neural network may be used to perform this operation, it should be understood that other means of comparing collected sensor data to the incident scenario description in ticket 6 may be used.

[0032] Upon determining a match between the description of the ticket 6 and the collected sensor data from the vehicle data database 3, the link (i.e., URL) to the matched data and the ticket 6 may be sent to a requirements file (RaC file) generator 8 located on the server 14, which may generate a RaC file 9 based on the link to the collected sensor data and the ticket 6.

[0033] The RaC file 9 may include a ticket ID (e.g., associated with ticket 6), a unique file ID (which can be used to uniquely identify the RaC file 9), test requirements (test targets, test conditions, test criteria, etc.) converted from the natural language written in ticket 6 into code, and a link (URL) to the collected sensor data stored in the vehicle data database 3.

[0034] The RaC file 9 may be responsible for storing the expected behavior of the ML model. The expected behavior may comprise test requirements. In particular, the test requirements may specify the performance metrics of the ML model that need to be tested, for example, based on specific performance metric values ​​that need to be achieved, criteria that need to be achieved, the type of test, etc. Accordingly, the RaC file 9 may include test requirements such as test parameters, test targets, URLs or file paths of test data, requirements, pass criteria, test conditions, acceptance requirements, and acceptance criteria used during the ML evaluation process. Such requirements, test targets, test data file paths, pass criteria, and test conditions can be easily interpreted (e.g., by the ML evaluation pipeline 10) to determine how ML testing and evaluation of the ML model should be performed by the ML evaluation pipeline and subsequently by the ML evaluation pipeline 10. For example, the RaC file 9 may include standards in the form of code, which the ML evaluation pipeline can easily interpret into instructions on how to perform ML evaluation of the ML model. The RaC file 9 may be in a format such as YAML or a domain-specific language (DSL) format. The RaC file 9 may also be in the format of a programming language such as PYTHON®. Because the RaC file 9 is separate from the code used to actually perform the ML evaluation, it may eliminate the need to "hard-code" requirements into the ML evaluation process.

[0035] An ML evaluation pipeline 10 may be provided for testing ML models based on the RaC file 9. The ML evaluation pipeline may include an interface for interpreting requirements specified in the RaC file 9 and an interface for evaluating the ML model based on the interpreted requirements. The ML evaluation pipeline 10 may use a link (URL), file path, or unique key to the collected sensor data in the RaC file 9 to retrieve the sensor data collected during testing from the vehicle data database 3. Based on the results of the ML evaluation, the tested ML model may be used to generate a pre-deployment application 20-1, which may then be subsequently deployed in the vehicle 13 as a vehicle application 20. The ML evaluation pipeline 10 may also update the status of the ticket 6 in the ticket database 7 based on the results of the ML evaluation. While an ML evaluation pipeline 10 is specified, it should be understood that any suitable means for interpreting requirements from the RaC file and evaluating the ML model based on the interpreted requirements may be used, according to an embodiment.

[0036] 3 is an alternative system architecture diagram according to one or more exemplary embodiments. In particular, FIG. 3 illustrates an exemplary embodiment in which vehicle data is collected and used to test vehicle applications accordingly. Similar components to FIG. 2 are included, and therefore redundant descriptions may be omitted for ease of reading.

[0037] In particular, in contrast to the embodiment shown in Figure 2, the embodiment shown in Figure 3 provides a vehicle 13 that further includes a ticket receiver 31, a data transmitter 34, and a sensor data classifier 1 and / or a neural network 2 that are located in the vehicle 13 rather than in the server 14. The ticket receiver 31 may be capable of receiving a ticket 6 from a ticket database 7. It should be understood that, according to some embodiments, the ticket 6 may be received in a compressed format, an encrypted format, or a format that is partially embedded in a feature map.

[0038] The sensor data classifier 1 and / or the neural network 2 may similarly be configured to determine whether the collected sensor data obtained from the at least one sensor 17 matches the description in the ticket 6. If there is a match, the data transmitter 34 may be able to transmit the collected sensor data to the vehicle data database 3. When the server 14 receives the collected sensor data from the data transmitter 34, the URL of the received data may be sent to the RaC file generator 8. Thus, in the embodiment shown in FIG. 3 , as the number of test data collected to test the requirements or incident scenarios written in the ticket 6 increases, the number of tests corresponding to the ticket 6 also increases.

[0039] 4 is a flow chart diagram illustrating a method 400 for testing vehicle applications based on tickets according to one or more exemplary embodiments. It should be understood that, according to some embodiments, the system architectures illustrated in FIGS. 2 and 3 may be used to implement method 400.

[0040] 4 , in operation S410, a ticket 6 may be generated that includes at least one human-readable test requirement and at least one human-readable incident scenario description. According to some embodiments, this may be done by combining an incident record (IR) ticket 4 written in natural language and obtained from an incident record server 11, and a requirements description (RD) ticket 5 obtained from a requirements management dashboard 12. According to some embodiments, the ticket 6 may be stored in a ticket database 7.

[0041] According to some embodiments, the ticket 6 may be received by a ticket receiver 31 in the vehicle 13 .

[0042] In operation S420, a determination may be made as to whether collected sensor data from a vehicle (i.e., vehicle 13) matches at least one human-readable incident scenario description and / or test requirement description in ticket 6. According to some embodiments, this may be done using a sensor data classifier 1 and / or a neural network 2. According to embodiments, the collected sensor data may initially be stored in a database of vehicle data 3. According to some embodiments, the sensor data classifier 1 and / or the neural network 2 may be implemented in the vehicle.

[0043] At operation S430, a RaC file 9 may be generated based on the ticket 6 and the collected sensor data. According to an embodiment, operation S430 may be performed only if operation S420 determines that a match exists between the collected sensor data and the incident description scenario from ticket 6. The RaC file 9 may include a ticket identifier (which may be the same as in ticket 6), a file identifier, coded test requirements, and a link to the collected sensor data. The coded test requirements may be generated based on a translation of the human-readable test requirements from ticket 6. According to an embodiment, operation S430 may be performed by the RaC file generator 8.

[0044] According to some embodiments, based on a determination that the collected sensor data from the vehicle 13 matches at least one natural language incident scenario description and / or test requirement description, the collected sensor data may be transmitted by a data transmitter 34 in the vehicle 13 to a database 3 of vehicle data.

[0045] At operation S440, the ML model (which may be used to implement vehicle application 20) may be evaluated based on RaC file 9 as generated at operation S430. In particular, this may evaluate whether the ML model achieves the test requirements (as interpreted from RaC file 9) originally specified in ticket 6. Operation S440 may be implemented using an ML evaluation pipeline 10, which may interpret the requirements from RaC file 9 and perform the evaluation based on the interpreted requirements. According to some embodiments, once operation S440 is completed, the vehicle application may be implemented and deployed within vehicle 13. In some embodiments, once operation S440 is completed, the status of ticket 6 may be updated as a result of the evaluation.

[0046] Based on the above embodiment, since the human-readable test requirements and the human-readable incident scenario descriptions are converted into coded files, the operator / developer can easily and broadly specify the conditions and requirements under which data can be collected since it is in natural language before processing. Therefore, the conditions for collecting data can be less strict and false positives / false negatives can be avoided.

[0047] The foregoing disclosure provides illustration and description, but is not intended to be exhaustive or to limit the exemplary embodiment or embodiments to the precise forms disclosed. Modifications and variations are possible in light of the disclosure or may be learned from practice of the exemplary embodiment or embodiments.

[0048] One or more exemplary embodiments may relate to a system, method, and / or computer-readable medium at any possible level of technical detail of integration. Furthermore, one or more of the above-described components may be implemented as instructions stored on a computer-readable medium and executable by at least one processor (and / or may include at least one processor). The computer-readable medium may include a computer-readable non-transitory storage medium (or media) having computer-readable program instructions that cause a processor to perform operations.

[0049] A computer-readable storage medium may be a tangible device that can hold and store instructions for use by an instruction execution device. A computer-readable storage medium may be, for example, but is not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof. A non-exhaustive list of more specific examples of computer-readable storage media includes the following: portable computer diskettes, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disk read-only memory (CD-ROM), digital versatile disk (DVD), memory sticks, floppy disks, mechanically encoded devices such as punch cards or ridge-in-groove structures with instructions recorded thereon, and any suitable combination thereof. As used herein, computer-readable storage media should not be construed as being transitory signals per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through a waveguide or other transmission medium (e.g., light pulses passing through a fiber optic cable), or electrical signals transmitted over wires.

[0050] The computer-readable program instructions described herein may be downloaded to each computing / processing device from a computer-readable storage medium, or may be downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network may include copper transmission cables, optical fiber transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in a computer-readable storage medium in the respective computing / processing device.

[0051] The computer-readable program code / instructions for performing operations may be assembler instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, configuration data for integrated circuits, or source or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, or the like, and procedural programming languages ​​such as the "C" programming language or similar programming languages. The computer-readable program instructions may be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection to the external computer may be made (e.g., through the Internet using an Internet Service Provider). In one or more exemplary embodiments, an electronic circuit, including, for example, a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), may execute computer-readable program instructions by utilizing state information of the computer-readable program instructions to personalize the electronic circuit to perform an aspect or operation.

[0052] The computer-readable program instructions may be provided to a processor of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus to create a machine, such that the instructions, executed by the processor of the computer or other programmable data processing apparatus, create means for implementing the function(s) / act(s) specified in the block(s) of the flowcharts and / or block diagrams. The computer-readable program instructions may also be stored on a computer-readable storage medium that can direct a computer, programmable data processing apparatus, and / or other device to function in a particular manner, such that the computer-readable storage medium having instructions stored therein comprises an article of manufacture including instructions that implement aspects of the function(s) / act(s) specified in the block(s) of the flowcharts and / or block diagrams.

[0053] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause the computer, other programmable apparatus, or other device to perform a series of operational steps to create a computer-implemented process, such that the instructions executing on the computer, other programmable apparatus, or other device implement the function / act specified in the block or blocks of the flowcharts and / or block diagrams.

[0054] The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible example implementations of systems, methods, and computer-readable media according to one or more exemplary embodiments. In this regard, each block in a flowchart or block diagram may represent a microservice, module, segment, or portion of an instruction set, comprising one or more executable instructions that implement the specified logical function(s). The methods, computer systems, and computer-readable media may include additional, fewer, different, or differently arranged blocks compared to those depicted in the figures. In one or more alternative exemplary embodiments, the functions noted in the blocks may occur out of the order noted in the figures. For example, two blocks shown in succession may in fact be executed concurrently or substantially concurrently, or the blocks may even be executed in the reverse order, depending on the functionality involved. It should also be noted that each block of the block diagrams and / or flowchart diagrams, and combinations of blocks in the block diagrams and / or flowchart diagrams, may be implemented by a dedicated hardware-based system that performs the specified functions or acts or executes a combination of dedicated hardware and computer instructions.

[0055] It will be apparent that the systems and / or methods described herein may be implemented in various forms of hardware, firmware, or combinations of hardware and software. The actual specialized control hardware or software code used to implement the systems and / or methods is not a limitation of one or more exemplary embodiments. Thus, the operation and behavior of the systems and / or methods are described herein without reference to specific software code, and it will be understood that software and hardware can be designed to implement the systems and / or methods based on the description herein.

Claims

1. A method for testing a vehicle application, executed by a processor, the method comprising: generating a ticket, the ticket comprising at least one natural language test requirement description and / or at least one natural language incident scenario description; determining whether sensor data collected from the vehicle matches at least one human-readable incident scenario description or natural language test requirement description in the ticket; based on a determination that the collected sensor data from the vehicle matches the at least one human-readable incident scenario description or natural language test requirements description; generating a requirements file (RaC file) based on the ticket and sensor data collected from the vehicle; and evaluating an ML model based on the RaC file to determine whether the ML model achieves the test requirements, wherein the ML model is used to implement a vehicle application; and A method comprising:

2. The method of claim 1 , wherein the ticket is generated based on an incident record (IR) ticket written in natural language and a requirements description (RD) ticket written in natural language.

3. The method of claim 1 or 2, wherein the tickets are stored in a ticket database and the collected sensor data is obtained from a database of vehicle data.

4. The method of claim 1 or 2, wherein determining whether sensor data collected from a vehicle matches a description in the ticket is performed using a sensor data classifier and / or a neural network.

5. The method of claim 1 or 2, wherein the RaC file comprises a ticket identifier, a file identifier, coded test requirements, and a link to the collected sensor data.

6. The method of claim 5 , wherein generating the RaC file comprises converting the at least one natural language test requirements description into coded test requirements.

7. The method of claim 1 or 2, wherein upon completion of evaluation of the ML model based on the RaC file, the vehicle application is deployed in the vehicle.

8. The method of claim 1 or 2, wherein upon completion of evaluation of the ML model based on the RaC file, the status of the ticket is updated based on the results of the evaluation.

9. 5. The method of claim 4, wherein once generation of the ticket is complete, the ticket is received by a ticket receiver in the vehicle, and the sensor data classifier and / or the neural network are implemented in the vehicle.

10. 10. The method of claim 9, wherein, based on a determination that the collected sensor data from the vehicle matches the at least one natural language incident scenario description, the collected sensor data is transmitted by a data transmitter in the vehicle to a database of vehicle data.

11. 1. An apparatus for testing a vehicle application, the apparatus comprising: at least one memory storing computer-executable instructions; at least one processor; wherein the at least one processor executes the computer-executable instructions to generating a ticket, the ticket comprising at least one natural language test requirement description and / or at least one natural language incident scenario description; determining whether sensor data collected from the vehicle matches at least one human-readable incident scenario description or natural language test requirement description in the ticket; based on a determination that the collected sensor data from the vehicle matches the at least one human-readable incident scenario description or natural language test requirements description; generating a requirements file (RaC file) based on the ticket and sensor data collected from the vehicle; and evaluating an ML model based on the RaC file to determine whether the ML model achieves the test requirements, wherein the ML model is used to implement a vehicle application; and The apparatus is configured to:

12. The apparatus of claim 11 , wherein the ticket is generated based on an incident record (IR) ticket written in natural language and a requirements description (RD) ticket written in natural language.

13. 13. The apparatus of claim 11 or 12, wherein the tickets are stored in a ticket database and the collected sensor data is obtained from a database of vehicle data.

14. 13. The apparatus of claim 11 or 12, wherein determining whether sensor data collected from a vehicle matches a description in the ticket is performed using a sensor data classifier and / or a neural network.

15. 13. The apparatus of claim 11 or 12, wherein the RaC file comprises a ticket identifier, a file identifier, coded test requirements, and a link to the collected sensor data.

16. 16. The apparatus of claim 15, wherein the at least one processor is further configured to execute the computer-executable instructions to generate the RaC file by converting the at least one natural language test requirements description into coded test requirements.

17. 13. The apparatus of claim 11 or 12, wherein upon completion of evaluation of the ML model based on the RaC file, the vehicle application is deployed in the vehicle.

18. 13. The apparatus of claim 11 or 12, wherein upon completion of evaluation of the ML model based on the RaC file, the status of the ticket is updated based on the results of the evaluation.

19. 15. The apparatus of claim 14, wherein once generation of the ticket is complete, the ticket is received by a ticket receiver in the vehicle, and the sensor data classifier and / or the neural network are implemented in the vehicle.

20. 20. The apparatus of claim 19, wherein based on a determination that the collected sensor data from the vehicle matches the at least one natural language incident scenario description, the collected sensor data is transmitted by a data transmitter in the vehicle to a database of vehicle data.

Images