Vehicle application program testing method and device

Abstract

The invention provides a vehicle application program testing method and device. The method comprises the steps that a bill is generated, and the bill is provided with at least one natural language test element and at least one natural language event scene description; determining whether the sensor data collected from the vehicle is consistent with the event scene description of at least one natural language in the ticket; and based on a determination that the sensor data collected from the vehicle coincides with the at least one natural language event scenario description, generating a key file, RaC file, based on the ticket and the sensor data collected from the vehicle; and evaluating the ML model based on the RaC file, and determining whether the ML model achieves test requirements, where the ML model is used to implement a vehicle application.

Description

Technical Field

[0001] Methods and apparatus integrated with exemplary embodiments of the present disclosure relate to automated vehicle testing based on requirements written in natural language. Background Art

[0002] In a related art, testing of automated vehicle systems can be implemented using machine learning (ML) models that can be used to automate various tasks associated with the actions of automated vehicles. For this purpose, in order to test vehicle applications, test data needs to be collected.

[0003] Typically, the vehicle may be implemented using sensors (e.g., cameras, accelerometers, etc.) for collecting data, which may then be processed by the operator / manufacturer for testing. For this purpose, the data collected by the sensors may be input to a vehicle application (e.g., which may be implemented using an ML model).

[0004] In the related art, data is collected from the vehicle according to a rule-based trigger condition and input to the test vehicle application. In particular, the rule-based trigger condition requires an obvious, deterministic and clear definition of the conditions under which sensor data is collected from the vehicle to the database / server. The rule-based trigger condition needs to be clearly specified with a numerical reference and strict conditional branches, and therefore, cannot include any ambiguity. Depending on such strict conditions for collecting data, sometimes it results in a lack of collection of data required for testing, and sometimes causes false positives / false negatives in the collection.

[0005] Therefore, there is a need for a system for collecting data that can specify test requirements in a less rigid manner. Summary of the invention

[0006] According to one or more exemplary embodiments, an apparatus and method for natural language-based automated vehicle testing are provided. In particular, a ticket for a subject management system can be generated based on a test requirement description and an event scenario description written in a natural language. An encoded file (e.g., a file in which the requirement is encoded (Requirement as Code (RaC) file)) can be generated based on the consistency of the ticket with the sensor data collected from the vehicle, and a machine learning (ML) model (which can implement a vehicle application) can evaluate whether the ML model is suitable for the test requirement based on the encoded file. Therefore, the test requirements and event scenarios are converted into encoded files after being written in a human-readable manner, so that the operator / developer can specify the test requirements and event scenarios in a natural language that allows ambiguity, thereby easily and broadly specifying the conditions and requirements for collecting data. Therefore, the conditions for collecting data can be less strict, and false positives / false negatives can be avoided.

[0007] According to an embodiment, a method for testing a vehicle application may be provided. The method may include: generating a ticket, wherein the ticket has at least one test requirement description written in a human-readable manner and at least one event scenario description in a natural language; determining whether the sensor data collected from the vehicle is consistent with at least one event scenario description written in a human-readable manner in the ticket; and generating a requirement file (RaC file) based on the ticket and the sensor data collected from the vehicle based on the determination that the sensor data collected from the vehicle is consistent with at least one event scenario description written in a human-readable manner; and evaluating an ML model based on the RaC file to determine whether the ML model meets the test requirement, wherein the ML model is used to implement a vehicle application. According to an embodiment, the ticket may be generated based on an incident record (IR: Incident Record) ticket written in a natural language and a requirement description (RD: Requirement Description) ticket written in a natural language. The ticket may be stored in a ticket database such as a problem management system (for example, a system that can open a ticket for management when a new problem needs to be solved), and the collected sensor data may be obtained from a vehicle data database. Determining whether sensor data collected from the vehicle is consistent with the description in the ticket may be performed using an image classifier and / or a neural network.

[0008] According to an embodiment, the RaC file may include a ticket identifier, a file identifier, an encoded test element, and a link to the collected sensor data. The generation of the RaC file may include: converting at least one test element description written in a human-readable manner into an encoded test element.

[0009] According to an embodiment, when the evaluation of the ML model based on the RaC file is completed, the vehicle application is deployed in the vehicle. When the evaluation of the ML model based on the RaC file is completed, the status of the ticket may be updated based on the result of the evaluation.

[0010] According to an embodiment, when the generation of the ticket is completed, the ticket is received by a ticket receiver in the vehicle, and an image classifier and / or a neural network is implemented in the vehicle. Based on the determination that the sensor data collected from the vehicle is consistent with at least one event scenario description written in a human-readable manner, the collected sensor data can be sent to a database of vehicle data through a data transmitter in the vehicle.

[0011] The additional aspects will be described in part in the following description and in part will be obvious from the description, or may be achieved by practice of the embodiments presented in the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Hereinafter, features, aspects, and advantages of specific preferred embodiments of the present disclosure will be described with reference to the accompanying drawings, in which like reference numerals represent like elements.

[0013] Figure 1 is a diagram of exemplary components of an apparatus according to an exemplary embodiment.

[0014] Figure 2 is a system architecture diagram of more than one exemplary implementation.

[0015] Figure 3 is an alternative system architecture diagram of one or more exemplary embodiments.

[0016] Figure 4 is a flow chart illustrating a method for ticket-based testing of a vehicle application according to one or more exemplary embodiments. DETAILED DESCRIPTION

[0017] The following detailed description of exemplary embodiments refers to the accompanying drawings. The present disclosure provides examples and descriptions, but is not intended to be exhaustive or to limit one or more exemplary embodiments to the exact form disclosed. Modifications and variations may be made in light of the present disclosure, or may be learned from the implementation of one or more exemplary embodiments. Moreover, one or more features or constituent elements of an exemplary embodiment may be combined with another exemplary embodiment (or one or more features of another exemplary embodiment), or may be combined therewith. Moreover, it is understood that in the flowcharts and descriptions of the actions provided in this specification, one or more actions may be omitted, one or more actions may be added, one or more actions may be performed (at least partially) simultaneously, and the order of one or more actions may be switched.

[0018] It is apparent that the exemplary embodiments of the systems and / or methods and / or non-transitory computer-readable storage media described in this specification may be implemented in various forms of hardware, firmware, or a combination of hardware and software. The actual dedicated control hardware or software code used to implement the system and / or method is not limited to one or more exemplary embodiments. Therefore, the actions and behaviors of the systems and / or methods and / or non-transitory computer-readable storage media are described in this specification without reference to specific software codes. It is understood that the software and hardware are designed to implement the system and / or method based on the description of this specification.

[0019] A particular combination of features is listed in the claims and / or disclosed in the specification, but this combination is not intended to limit the disclosure of possible exemplary embodiments. In fact, many of the features may be combined in ways not specifically listed in the claims and / or disclosed in the specification. Each dependent claim listed below is directly dependent on only one claim, but the disclosure of possible exemplary embodiments includes each dependent claim in combination with all other claims in the claim set.

[0020] As long as it is not particularly clearly recorded, the elements, behaviors or instructions used in this specification should not be interpreted as important or necessary. In addition, the articles "a" and "an" used in this specification are intended to include more than one matter and can be used interchangeably with "more than one". In the case of only one matter, a term such as "one" or the same term is used. In addition, the terms "has", "have", "having", "include", "including" or similar terms used in this specification are meant to be open-ended terms. Moreover, as long as it is not particularly clearly recorded, phrases such as "based on..." are intended to mean "at least partially based on...". Moreover, expressions such as "at least one of [A] and [B]" or "at least one of [A] or [B]" should be understood to include only A, only B or both A and B.

[0021] Figure 1 FIG. 1 is a diagram showing exemplary components of the vehicle testing device 100. Figure 1 As shown, the vehicle test device 100 may include a bus 110 , a processor 120 , a memory 130 , a storage unit 140 , an input unit 150 , an output unit 160 , and a communication interface 170 .

[0022] The bus 110 includes components that enable communication between components of the vehicle test device 100. The processor 120 can be implemented by hardware, firmware, or a combination of hardware and software. The processor 120 can be a central processing unit (CPU), a graphics processing unit (GPU), an accelerated processing unit (APU), a microprocessor, a microcontroller, a digital signal processor (DSP), a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), or other types of processing components. In one or more exemplary embodiments, the processor 120 includes one or more processors that can be programmed to perform functions. The memory 130 includes a random access memory (RAM), a read-only memory (ROM), and / or other types of dynamic or static storage devices (e.g., flash memory, magnetic memory, and / or optical memory) for storing information and / or instructions for use by the processor 120.

[0023] The storage unit 140 stores information and / or software related to the operation and use of the vehicle test device 100. For example, the storage unit 140 may include a corresponding drive and include a hard disk (e.g., a magnetic disk, an optical disk, a magneto-optical disk, and / or a solid state hard disk), a compact disc (CD: Compact Disc), a digital versatile disc (DVD: Digital Versatile Disc), a floppy disk, a cartridge, a magnetic tape, and / or other types of non-transitory computer-readable media. The input unit 150 includes components that enable the vehicle test device 100 to receive information via user input, etc. (e.g., a touch screen display, a keyboard, a keypad, a mouse, a button, a switch, and / or a microphone). In addition or alternatively, the input unit 150 may include a sensor that senses information (e.g., a global positioning system (GPS: Global Positioning System) component, an accelerometer, a gyroscope, and / or an actuator). The output unit 160 includes components that provide output information from the vehicle test device 100 (e.g., a display, a speaker, and / or one or more light emitting diodes (LED: Light Emitting Diode)).

[0024] The communication interface 170 includes transceiver-like components (e.g., a transceiver and / or a separate receiver and transmitter) that enable the vehicle test device 100 to communicate with other devices via a wired connection, a wireless connection, or a combination of a wired connection and a wireless connection. The communication interface 170 enables the vehicle test device 100 to receive information from another device and / or provide information to another device. For example, the communication interface 170 may include an Ethernet interface, an optical interface, a coaxial interface, an infrared interface, a radio frequency (RF) interface, a universal serial bus (USB) interface, a Wi-Fi (Wireless Fidelity) interface, a cellular network interface, or the like, but is not limited to these.

[0025] The vehicle test device 100 can perform one or more exemplary processes described in this specification. According to one or more exemplary embodiments, the vehicle test device 100 can perform the process in response to the processor 120 executing software instructions stored by a non-transitory computer-readable medium such as the memory 130 and / or the storage unit 140. The computer-readable medium is defined as a non-transitory memory device in this specification. The memory device includes a memory space within a single physical storage device or a memory space across multiple physical storage devices.

[0026] The software instructions may be read into the memory 130 and / or storage 140 from another computer-readable medium or from another device via the communication interface 170. When executed, the software instructions stored in the memory 130 and / or storage 140 may cause the processor 120 to perform one or more processes described in this specification.

[0027] Additionally or alternatively, to perform one or more processes described in this specification, hardwired circuits may be used in place of software instructions, or in combination with software instructions. Therefore, the one or more exemplary embodiments described in this specification are not limited to any specific combination of hardware circuits and software.

[0028] supply Figure 1 The number and arrangement of the components shown are provided as an example. Figure 1 The vehicle test apparatus 100 may include additional components, fewer components, different components, or components in a different configuration than shown. Additionally or alternatively, a set of components (e.g., more than one component) of the vehicle test apparatus 100 may perform one or more functions described as being performed by another set of components of the vehicle test apparatus 100.

[0029] Figure 2 is a system architecture diagram of more than one exemplary embodiment. In particular, Figure 2 An embodiment in which a vehicle application implemented using a machine learning (ML) model is tested is shown.

[0030] According to an embodiment, a vehicle 13 may be provided that communicates with a server 14. It should be understood that the server 14 may be implemented by a single server, may be implemented as multiple servers, or may be implemented using a cloud application, depending on the specific implementation. It may be that an operator 15 and a product owner / developer / user 16 can interact with the server 14.

[0031] The vehicle 13 may include at least one sensor 17 and a vehicle application 20. In particular, the at least one sensor 17 may include a camera, an accelerometer, a gyroscope, an IR (infrared) sensor, etc., but is not limited thereto. The at least one sensor 17 may be responsible for collecting data related to the test of the vehicle. It may be that the at least one sensor 17 can send the collected data to a database 3 of vehicle data configured on the server 14. The vehicle application 20 may be any application related to the action of the vehicle 13 (for example, related to the steering or acceleration of the vehicle 13). The vehicle application 20 may be implemented using a machine learning (ML) model.

[0032] The server 14 may include an event recording server 11 and a requirement management dashboard 12. The event recording server 11 may include a collection of event scenarios that may be input by an operator 15 in natural language. In particular, the natural language may be any known human language, and according to some embodiments, the natural language may be written in a structure that can be easily understood by people who may not necessarily have knowledge related to the method of interpreting the code. In other words, the natural language may include human language that can be understood by ordinary people when interpreted (for example, in a conversation or interpretation). The event scenario may specify a situation in which the vehicle application 20 needs to make a decision related to the method of causing the vehicle 13 to act, for example, it may specify that there is an approaching vehicle within an arbitrary horizontal and vertical distance from the two lanes on the left. It may be that the event recording server can generate / store / output an event recording (IR) ticket 4 also related to the natural language based on the collection of event scenarios written in a human-readable manner. It should be understood that the event scenario may include a realistic scenario or a test scenario, but is not necessarily limited to these.

[0033] The requirement management dashboard 12 (which may be implemented as its own server in some embodiments) may include a set of test requirements that may be input in natural language by the product owner / developer / user 16. The test requirements may specify a test goal, a test condition, or a test benchmark related to the operation of the vehicle 13. It may be that the requirement management dashboard can generate / store / output a requirement description (RD) ticket 5 described in natural language based on the set of test requirements. The event scenario may include, for example, information related to an accident / problem / bad condition of the vehicle 13.

[0034] The ticket 6 may be generated based on the IR ticket 4 and the RD ticket 5 (e.g., by combining the IR ticket 4 with the RD ticket 5), and thus the ticket 6 may be stored in the ticket database 7 in the server 14. Each ticket 6 may have its own inherent identifier (ID). It should be understood that although a database is specified, other storage components (e.g., cloud storage, content distribution network) may also be implemented for storing tickets.

[0035] The sensor data classifier 1 may also be provided in the server 14. According to the embodiment in which the collected sensor data is part of an image, the sensor data classifier 1 may be an image classifier implemented using a neural network 2. The sensor data classifier 1 may be used to compare the collected sensor data obtained from the database 3 of vehicle data with the ticket 6 to determine whether the specific collected sensor data is consistent with the event scene description and / or the test requirement description in the ticket 6. It should be understood that in order to perform such an action, an image classifier and / or a neural network may be used, but other components related to the comparison of the collected sensor data with the event scene description in the ticket 6 may be used.

[0036] When the description of ticket 6 is determined to be consistent with the sensor data collected from the vehicle data database 3, a link to the consistent data (i.e., URL (Uniform Resource Locator)) and ticket 6 can be sent to a requirement file (RaC file) generator 8 configured on the server 14. The RaC file generator 8 can generate a RaC file 9 based on the link to the collected sensor data and ticket 6.

[0037] The RaC file 9 may include a ticket ID (for example, associated with the ticket 6), an inherent file ID (which can be used to uniquely identify the RaC file 9), test requirements (test objectives, test conditions, test benchmarks, etc.) converted from the natural language written in the ticket 6 into code, and a link (URL) to the collected sensor data stored in the database 3 of vehicle data.

[0038] The RaC file 9 may be responsible for storing the expected behavior of the ML model. The expected behavior may have test requirements. In particular, the test requirements may specify the performance indicators of the ML model to be tested, for example, based on the values ​​of the specific performance indicators to be achieved, the benchmarks to be achieved, the types of tests, etc. Thus, the RaC file 9 may include test parameters, test targets, URLs or file paths of test data, requirements, qualified benchmarks, test conditions, acceptance conditions, and acceptance benchmarks used in the ML evaluation process as test requirements. Such requirements, test targets, file paths of test data, qualified benchmarks, and test conditions can be easily interpreted (for example, by the ML evaluation pipeline 10) to determine how the ML test and evaluation of the ML model is performed by the ML evaluation pipeline, and whether it should be performed by the ML evaluation pipeline 10 thereafter. For example, the RaC file 9 may include benchmarks in the form of code, which the ML evaluation pipeline can easily interpret as instructions related to the method of performing the ML evaluation of the ML model. The RaC file 9 may be in a format such as a YAML format or a domain-specific language (DSL) format. Furthermore, the RaC file 9 may be in the format of a programming language such as PYTHON, etc. The RaC file 9 is separate from the code for actually executing the ML evaluation, thus eliminating the need to "hard code" the elements into the ML evaluation process.

[0039] An ML evaluation pipeline 10 may be provided to test the ML model based on the RaC file 9. The ML evaluation pipeline may include an interface for interpreting the requirements specified in the RaC file 9 and an interface for evaluating the ML model based on the interpreted requirements. The ML evaluation pipeline 10 may use a link (URL), a file path, or an inherent key for the collected sensor data in the RaC file 9 to obtain the collected sensor data from the database 3 of vehicle data during testing. Based on the results of the ML evaluation, the tested ML model may be used to generate a pre-deployment application 20-1, and then, thereafter, deployed in the vehicle 13 as a vehicle application 20. In addition, the ML evaluation pipeline 10 may update the status of the ticket 6 in the ticket database 7 based on the results of the ML evaluation. It should be understood that although the ML evaluation pipeline 10 is specified, any suitable component that interprets the requirements based on the RaC file and evaluates the ML model based on the interpreted requirements may be used according to the embodiment.

[0040] Figure 3 is an alternative system architecture diagram of one or more exemplary embodiments. In particular, Figure 3 An exemplary embodiment for collecting vehicle data and testing a vehicle application based on the vehicle data is shown. Figure 2For the same constituent elements, therefore, redundant descriptions may be excluded for ease of reading.

[0041] In particular, relative to Figure 2 The embodiment shown, Figure 3 The illustrated embodiment provides a vehicle 13 that also includes a ticket receiver 31, a data transmitter 34, and a sensor data classifier 1 and / or a neural network 2 configured on the vehicle 13 instead of the server 14. It can be that the ticket receiver 31 can receive the ticket 6 from the ticket database 7. It should be understood that according to some embodiments, the ticket 6 can be received in a compressed format, an encrypted format, or a format partially embedded in a feature map.

[0042] Likewise, the sensor data classifier 1 and / or the neural network 2 may be configured to determine whether the collected sensor data obtained from the at least one sensor 17 is consistent with the description of the ticket 6. It may be that, in the event of consistency, the data transmitter 34 can send the collected sensor data to the database 3 of vehicle data. When the server 14 receives the collected sensor data from the data transmitter 34, the URL of the received data may be sent to the RaC file generator 8. Thus, in Figure 3 In the illustrated embodiment, if the amount of collected test data for testing the requirements or event scenarios written in the ticket 6 increases, the number of tests corresponding to the ticket 6 will also increase.

[0043] Figure 4 4 is a flowchart of a method 400 for testing a vehicle application based on a ticket according to one or more exemplary embodiments. It should be understood that according to some embodiments, in order to implement the method 400, Figure 2 and Figure 3 The system architecture is shown.

[0044] Reference Figure 4 In action S410, a ticket 6 including at least one test requirement description written in a human-readable manner and at least one event scenario description written in a human-readable manner may be generated. According to some embodiments, this may be performed by combining an event record (IR) ticket 4 written in a natural language and obtained from an event record server 11 with a requirement description (RD) ticket 5 obtained from a requirement management dashboard 12. According to some embodiments, the ticket 6 may be stored in a ticket database 7.

[0045] According to some embodiments, the ticket 6 may be received by a ticket receiver 31 in the vehicle 13 .

[0046] In action S420, it may be determined whether the sensor data collected from the vehicle (i.e., vehicle 13) is consistent with at least one event scenario description and / or test requirement description written in a human-readable manner in the ticket 6. According to some embodiments, this may be performed using a sensor data classifier 1 and / or a neural network 2. According to an embodiment, the collected sensor data may be initially stored in a database 3 of vehicle data. According to some embodiments, the sensor data classifier 1 and / or the neural network 2 may be implemented in the vehicle.

[0047] In action S430, a RaC file 9 may be generated based on the ticket 6 and the collected sensor data. According to an embodiment, action S430 may be performed only if it is determined in action S420 that there is a consistency between the collected sensor data and the event scenario description from the ticket 6. The RaC file 9 may include a ticket identifier (which may be the same as in the ticket 6), a file identifier, encoded test requirements, and a link to the collected sensor data. The encoded test requirements may be generated based on a conversion of the test requirement description written in a human-readable manner from the ticket 6. According to an embodiment, action S430 may be performed by a RaC file generator 8.

[0048] According to some embodiments, based on a determination that sensor data collected from vehicle 13 is consistent with at least one natural language event scenario description and / or test requirement description, the collected sensor data may be sent to vehicle data database 3 via data transmitter 34 in vehicle 13 .

[0049] In act S440, the ML model (which may be used to implement the vehicle application 20) may be evaluated based on the RaC file 9 such as that generated in act S430. In particular, this may evaluate whether the ML model meets the test requirements originally specified in the ticket 6 (as interpreted from the RaC file 9). Act S440 may be implemented using an ML evaluation pipeline 10, which may interpret the requirements from the RaC file 9 and perform the evaluation based on the interpreted requirements. According to some embodiments, when act S440 is completed, the vehicle application may be implemented and deployed in the vehicle 13. In some embodiments, when act S440 is completed, the status of the ticket 6 may be updated as a result of the evaluation.

[0050] Based on the above embodiment, the test requirement description written in a human-readable manner and the event scenario description written in a human-readable manner are converted into an encoded file, so that the operator / developer can easily and broadly specify the conditions and requirements for collecting data because it is related to the natural language before processing. Therefore, the conditions for collecting data can be less strict, and false positives / false negatives can be avoided.

[0051] The above disclosure provides examples and descriptions, but is not intended to be exhaustive or to limit the one or more exemplary embodiments to the precise forms disclosed. Modifications and variations may be made in light of this disclosure or may be learned from practice of one or more exemplary embodiments.

[0052] One or more exemplary embodiments may involve systems, methods and / or computer-readable media at any possible level of technical detail of the integration. Moreover, one or more of the above-mentioned constituent elements may be implemented as instructions stored in a computer-readable medium and executable by at least one processor (and / or include at least one processor). The computer-readable medium may include a computer-readable non-transitory storage medium (or medium(s)) having computer-readable program instructions that cause the processor to perform actions.

[0053] A computer-readable storage medium may be a tangible device that can hold and store instructions used by an instruction execution device. A computer-readable storage medium may be, for example, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the above devices, but is not limited to these. A more specific exemplary non-exhaustive list of computer-readable storage media includes the following, namely, portable computer disks, hard disks, random access memories (RAM), read-only memories (ROM), erasable programmable read-only memories (EPROM or flash memory), static random access memories (SRAM), portable compact disc read-only memories (CD-ROM), digital versatile discs (DVD), memory sticks, floppy disks, punched cards with instructions recorded or mechanically encoded devices such as raised structures in grooves, and any suitable combination of the above. The computer-readable storage medium used in this specification should not be interpreted as a temporary signal itself such as an electric wave or other freely propagating electromagnetic wave, an electromagnetic wave propagating through a waveguide or other transmission medium (for example, a light pulse through an optical cable), or an electrical signal sent through a wire.

[0054] The computer-readable program instructions recorded in this specification can be downloaded from a computer-readable storage medium to a respective computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can have copper transmission cables, optical transmission fibers, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and transmits the computer-readable program instructions to be stored in a computer-readable storage medium in the respective computing / processing device.

[0055] The computer readable program code / instructions for performing the actions may be source code or object code written in assembler instructions, instruction set architecture (ISA) instructions, machine instructions, machine dependent instructions, microcode, firmware instructions, state setting data, configuration data for integrated circuits, or any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++ or similar languages, and procedural programming languages ​​such as the "C" programming language or similar programming languages. The computer readable program instructions may be executed entirely on the user's computer, may be executed partially on the user's computer, may be executed as a stand-alone software package, may be executed partially on the user's computer and partially on a remote computer, or may be executed entirely on a remote computer or server. In the latter case, the remote computer may be connected to the user's computer via any type of network including a local area network (LAN) or a wide area network (WAN), or a connection to an external computer may be made (e.g., using an Internet service provider and thereby via the Internet). In one or more exemplary embodiments, for example, an electronic circuit including a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA) may utilize state information of computer-readable program instructions to personalize the electronic circuit, thereby executing the computer-readable program instructions to perform a scheme or action.

[0056] The computer-readable program instructions may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device in order to generate a machine, as a result, the instructions executed by the processor of the computer or other programmable data processing device generate a component that implements the function / behavior specified in the functional block or functional blocks (multiple) of the flowchart and / or block diagram. The computer-readable program instructions may also be stored in a computer-readable storage medium, wherein the computer-readable storage medium may instruct a computer, a programmable data processing device, and / or other equipment to function in a specific manner, as a result, the computer-readable storage medium having the instructions stored therein has an article of manufacture, which includes instructions for implementing the scheme of the function / behavior specified in the functional block or functional blocks (multiple) of the flowchart and / or block diagram.

[0057] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device, and perform a series of action steps on the computer, other programmable apparatus, or other device to generate a computer-implemented process, as a result of which the instructions executed on the computer, other programmable apparatus, or other device implement the functions / behaviors specified in the functional box or functional boxes (multiple) in the flowchart and / or block diagram.

[0058] The flowchart and block diagram in the figure show the architecture, functions and actions of possible exemplary embodiments of the system, method and computer-readable medium of more than one exemplary embodiment. In this regard, each functional block in the flowchart or block diagram may represent a microservice, module, segment or part of an instruction having more than one executable instruction to implement the specified logical function. The method, computer system and computer-readable medium may include additional functional blocks, fewer functional blocks, different functional blocks or functional blocks of different configurations compared to those depicted in the accompanying drawings. In more than one alternative exemplary embodiment, the functions recorded in the functional blocks may be generated independently of the order recorded in the figure. For example, two functional blocks shown in succession may be executed in fact or substantially at the same time, or the functional blocks may be executed in reverse order according to the associated functions. It should be noted that each functional block of the block diagram and / or flowchart, and the combination of functional blocks of the block diagram and / or flowchart can be implemented by a dedicated hardware-based system, wherein the system performs the specified function or behavior, or executes a combination of dedicated hardware and computer instructions.

[0059] It is apparent that the systems and / or methods described in this specification may be implemented in various forms of hardware, firmware, or a combination of hardware and software. The actual dedicated control hardware or software code used to implement the system and / or method is not limited to one or more exemplary embodiments. Therefore, it is understood that the actions and behaviors of the system and / or method are described in this specification without reference to specific software codes, and the software and hardware may be designed to implement the system and / or method based on the description of this specification.

Claims

1. A vehicle application testing method is a method for testing a vehicle application, comprising: Generate a ticket, wherein the ticket has at least one test requirement description in a natural language and / or at least one event scenario description in a natural language; Determining whether the sensor data collected from the vehicle is consistent with at least one of the event scenario description written in a human-readable manner or the test element description in a natural language in the ticket; and Based on a determination that the sensor data collected from the vehicle is consistent with the at least one event scenario description written in a human-readable manner or a test element description in a natural language, Generate a requirement file, namely a RaC file, based on the ticket and sensor data collected from the vehicle; and The machine learning ML model is evaluated based on the RaC file to determine whether the ML model meets the test requirements, wherein the ML model is used to implement a vehicle application.

2. The method according to claim 1, wherein: The ticket is generated based on an event record IR ticket written in a natural language and a requirement description RD ticket written in a natural language.

3. The method according to claim 1 or 2, wherein: The tickets are stored in a ticket database, and the collected sensor data are obtained from a vehicle data database.

4. The method according to any one of claims 1 to 3, wherein: Determining whether sensor data collected from the vehicle is consistent with the description in the ticket is performed using a sensor data classifier and / or a neural network.

5. The method according to any one of claims 1 to 4, wherein: The RaC file includes a ticket identifier, a file identifier, an encoded test requirement, and a link to the collected sensor data.

6. The method according to claim 5, wherein: Generating the RaC file includes: converting the test requirement description in the at least one natural language into an encoded test requirement.

7. The method according to any one of claims 1 to 6, wherein: When the evaluation of the ML model based on the RaC file is completed, the vehicle application is deployed in the vehicle.

8. The method according to any one of claims 1 to 7, wherein: When the evaluation of the ML model based on the RaC file is completed, the status of the ticket is updated based on the result of the evaluation.

9. The method according to claim 4, wherein: When the generation of the ticket is completed, the ticket is received by a ticket receiver in the vehicle, and the sensor data classifier and / or the neural network are implemented in the vehicle.

10. The method according to claim 9, wherein: Based on the determination that the sensor data collected from the vehicle is consistent with the at least one natural language event scenario description, the collected sensor data is sent to a database of vehicle data via a data transmitter in the vehicle.

11. A vehicle application testing device is a device for testing a vehicle application, wherein: The device comprises: at least one memory storing computer executable instructions; and at least one processor, The at least one processor is configured to execute the computer executable instructions to perform the following processing: Generate a ticket, wherein the ticket has at least one test requirement description in a natural language and / or at least one event scenario description in a natural language; Determining whether the sensor data collected from the vehicle is consistent with at least one of the event scenario description written in a human-readable manner or the test element description in a natural language in the ticket; and Based on a determination that the sensor data collected from the vehicle is consistent with the at least one event scenario description written in a human-readable manner or a test element description in a natural language, Generate a requirement file, namely a RaC file, based on the ticket and sensor data collected from the vehicle; and The machine learning ML model is evaluated based on the RaC file to determine whether the ML model meets the test requirements, wherein the ML model is used to implement a vehicle application.

12. The device according to claim 11, wherein The ticket is generated based on an event record IR ticket written in a natural language and a requirement description RD ticket written in a natural language.

13. The device according to claim 11 or 12, wherein: The tickets are stored in a ticket database, and the collected sensor data are obtained from a vehicle data database.

14. The device according to any one of claims 11 to 13, wherein: Determining whether sensor data collected from the vehicle is consistent with the description in the ticket is performed using a sensor data classifier and / or a neural network.

15. The device according to any one of claims 11 to 14, wherein: The RaC file includes a ticket identifier, a file identifier, an encoded test requirement, and a link to the collected sensor data.

16. The device according to claim 15, wherein: The at least one processor is further configured to execute the computer-executable instructions to perform the following processing: converting the at least one natural language test element description into an encoded test element, thereby generating the RaC file.

17. The device according to any one of claims 11 to 16, wherein: When the evaluation of the ML model based on the RaC file is completed, the vehicle application is deployed in the vehicle.

18. The device according to any one of claims 11 to 17, wherein: When the evaluation of the ML model based on the RaC file is completed, the status of the ticket is updated based on the result of the evaluation.

19. The device according to claim 14, wherein: When the generation of the ticket is completed, the ticket is received by a ticket receiver in the vehicle, and the sensor data classifier and / or the neural network are implemented in the vehicle.

20. The device according to claim 19, wherein Based on the determination that the sensor data collected from the vehicle is consistent with the at least one natural language event scenario description, the collected sensor data is sent to a database of vehicle data via a data transmitter in the vehicle.

Images