General simulation test for a generalized system under test (SuT)

Abstract

To provide a general-purpose simulation test for a generalized system under test (SuT).SOLUTION: A simulation system based on a vehicle system includes: a simulation module configured to generate simulated sensor data; at least one mock component being in a state connected to the simulation module and configured to simulate operation of a module of the vehicle system; and at least one test hardware component being in a state connected to the simulation module and the at least one mock component, the operation of the at least one test hardware component being tested during the operation of the simulation system based on the simulated sensor data having been generated.SELECTED DRAWING: Figure 4

Description

Technical Field

[0001] The present disclosure generally relates to systems and methods for providing a general-purpose test strategy.

Background Art

[0002] In a system under test (SuT), a developer utilizes a simulator to test components of the system. In a typical SuT scenario, the developer determines which components to test and which components to simulate via a shim. Specifically, for vehicle instrumentation, the developer can decide to test the planning module, then simulate the perception module and the localization module by an input shim, and simulate the controller using an output shim.

[0003] However, this SuT configuration has several problems. First, the shim is a software component that is permanently built without customization options, which means that once the SuT is completed, the developer cannot change the fidelity of the shim for different test purposes (i.e., a new shim has to be built), which causes serious inefficiencies. Second, since the teams building the perception module, the localization module, the planning module, the controller, and other components are usually separate entities, the construction of the SuT configuration is slow. Moreover, the shim components are built on a case-by-case basis, making the SuT process costly. Further, typically, there is no general-purpose interface language or communication between the components of the system, which means that the built shims are not easily interchangeable or cannot be integrated into a new SuT configuration.

Summary of the Invention

[0004] According to one aspect of an exemplary embodiment, a simulation system based on a vehicle system may include a simulation module configured to generate simulated sensor data, at least one mock component connected to the simulation module and configured to simulate the operation of a module of the vehicle system, and at least one test hardware component connected to the simulation module and the at least one mock component, the operation of which is tested during the operation of the simulation system based on the generated simulated sensor data.

[0005] According to one aspect of an exemplary embodiment, a method of a simulation system may include generating, by a simulation module, simulated sensor data; simulating, by at least one mock component connected to the simulation module, the operation of a module of the vehicle system; and testing, during the operation of the simulation system, at least one test hardware component connected to the simulation module and the at least one mock component based on the generated simulated sensor data.

[0006] According to one aspect of an exemplary embodiment, a non-transitory computer-readable storage medium may store instructions that, when executed by at least one processor, cause the at least one processor to generate, by a simulation module, simulated sensor data; simulate, by at least one mock component connected to the simulation module, the operation of a module of the vehicle system; and test, during the operation of the simulation system, at least one test hardware component connected to the simulation module and the at least one mock component based on the generated simulated sensor data.

[0007] Further aspects are in part described in the following specification, and in part will become apparent from the specification or can be learned by practicing the presented embodiments of the disclosure.

[0008] The above and other aspects, features and embodiments of the present disclosure will become more apparent by considering the following description in conjunction with the accompanying drawings.

Brief Description of the Drawings

[0009]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Modes for Carrying Out the Invention

[0010] The following detailed description of the exemplary embodiments refers to the accompanying drawings. The same reference numbers in different drawings may identify the same or similar elements.

[0011] FIG. 1 is a schematic diagram of a system according to an embodiment. FIG. 1 includes a client device 110, a server device 120, and a network 130. The client device 110 and the server device 120 can be interconnected via a wired connection, a wireless connection, or a combination of wired and wireless connections.

[0012] The client device 110 may include a computing device (such as a desktop computer, laptop computer, tablet computer, handheld computer, smart speaker, server device, etc.), a mobile phone (such as a smartphone, wireless telephone, etc.), a camera device, a wearable device (such as smart glasses or a smartwatch), or a similar device.

[0013] The server device 120 includes one or more devices. For example, the server device 120 may be a server device, a computing device, etc.

[0014] The network 130 includes one or more wired and / or wireless networks. For example, the network 130 may include a cellular network (such as a fifth-generation (5G) network, a long-term evolution (LTE) network, a third-generation (3G) network, a code division multiple access (CDMA) network, etc.), a public land mobile network (PLMN), a local area network (LAN), a wide area network (WAN), a metropolitan area network (MAN), a telephone network (such as a public switched telephone network (PSTN)), a private network, an ad hoc network, an intranet, the Internet, an optical fiber network, etc. and / or a combination of these or other types of networks.

[0015] The number and arrangement of the devices and networks shown in FIG. 1 are provided as an example. In reality, there may be additional devices and / or networks, fewer devices and / or networks, different devices and / or networks, or differently arranged devices and / or networks compared to those shown in FIG. 1. Further, two or more devices shown in FIG. 1 can be implemented within a single device, or a single device shown in FIG. 1 can be implemented as multiple distributed devices. Additionally or alternatively, a device set (e.g., one or more devices) can perform one or more functions described as being performed by another device set.

[0016] FIG. 2 is a schematic diagram of components of one or more devices of FIG. 1 according to an embodiment. Device 200 can correspond to client device 110 and / or server device 120.

[0017] As shown in FIG. 2, device 200 can include bus 210, processor 220, memory 230, storage component 240, input component 250, output component 260, and communication interface 270.

[0018] Bus 210 includes components that enable communication between components of device 200. Processor 220 is implemented in the form of hardware, firmware, or a combination of hardware and software. Processor 220 is a central processing unit (CPU), a graphics processing unit (GPU), an accelerated processing unit (APU), a microprocessor, a microcontroller, a digital signal processor (DSP), a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), or another type of processing component. Processor 220 can include one or more processors programmed to perform one function.

[0019] Memory 230 includes a random access memory (RAM), a read-only memory (ROM), and / or another type of dynamic or static storage device (such as flash memory, magnetic memory, and / or optical memory) for storing information and / or instructions used by processor 220.

[0020] Storage component 240 stores information and / or software related to the operation and use of device 200. For example, storage component 240 can include a hard disk (such as a magnetic disk, optical disk, magneto-optical disk, and / or solid state disk), a compact disc (CD), a digital versatile disc (DVD), a floppy disk, a cartridge, a magnetic tape, and / or another type of non-transitory computer-readable medium, as well as a corresponding drive.

[0021] Input component 250 includes components that enable device 200 to receive information via user input (such as a touch screen display, keyboard, keypad, mouse, button, switch, and / or microphone). Input component 250 can include sensors for detecting information (such as a global positioning system (GPS) component, an accelerometer, a gyroscope, and / or an actuator).

[0022] Output component 260 includes components that provide output information from device 200 (such as a display, a speaker, and / or one or more light emitting diodes (LEDs)).

[0023] The communication interface 270 includes transceiver-like components (such as a transceiver and / or separate receivers and transmitters) that enable the device 200 to communicate with other devices via, for example, a wired connection, a wireless connection, or a combination of wired and wireless connections. The communication interface 270 enables the device 200 to receive information from and / or provide information to another device. For example, the communication interface 270 may include an Ethernet interface, an optical interface, a coaxial interface, an infrared interface, a radio frequency (RF) interface, a universal serial bus (USB) interface, a Wi-Fi interface, a cellular network interface, etc.

[0024] The device 200 can perform one or more of the processes described herein. The device 200 can operate based on a processor 220 that executes software instructions stored by a non-transitory computer-readable medium such as the memory 230 and / or the storage component 240. The computer-readable medium is defined herein as a non-transitory memory device. The memory device includes a memory space within a single physical memory device or a memory space that spans across multiple physical memory devices.

[0025] The software instructions can be loaded into the memory 230 and / or the storage component 240 from another computer-readable medium or from another device via the communication interface 270. When executed, the software instructions stored in the memory 230 and / or the storage component 240 can cause the processor 220 to perform one or more of the processes described herein.

[0026] Additionally or alternatively, a hardwired circuit can be used instead of or in combination with software instructions to perform one or more of the processes described herein. Accordingly, the embodiments described herein are not limited to any particular combination of hardware circuitry and software.

[0027] FIG. 3 is a schematic diagram of a vehicle system 300 according to one embodiment. Although FIG. 3 depicts a vehicle system, embodiments of the present disclosure are not limited to a vehicle environment and can be implemented in other test environments as would be understood by one of ordinary skill in the art. The vehicle system 300 can include a perception module 302 that fuses information received from sensors into a coherent representation of the environment, and a localization module 304 that determines location information of the vehicle based on information received from the sensors. The vehicle system 300 can similarly include a planning module 306 that determines a strategic approach for achieving a particular goal (e.g., finding an appropriate route) based on data from the perception module 302 and the localization module 304. The vehicle system 300 can similarly include a controller 308 that receives strategy information from the planning module 306, determines appropriate commands for executing the approach determined by the planning module 306, and outputs the commands to an actuator to execute the determined approach. The vehicle system 300 can similarly include additional modules that mediate signals between the modules described above. Accordingly, the vehicle system can be considered to optionally have many modules that participate in a path for converting information received from sensors into output commands to an actuator.

[0028] To test at least one component of a system, a simulation system can be implemented both in hardware and software to efficiently and effectively test one or more components (i.e., the system under test (SuT)). The one or more components to be tested remain within the system as software or hardware, and the remaining components are replaced by mocks, which are software simulations of untested hardware components.

[0029] Figure 4 is a schematic diagram of a simulation system 400 according to an embodiment. In the example shown in Figure 4, the planning module 408 is in the process of testing (e.g., the planning module 306 in Figure 3). However, it is possible to test additional or alternative components. The simulation system 400 may include a simulation module 402 configured to generate information corresponding to the information generated by the sensor or other information that is considered to be generated within a physical vehicle system (e.g., the vehicle system 300 in Figure 3). Instead of a simulation, in one example, the simulation system 400 can include a perception mock 404 and a localization mock 406 connected to the simulation module. The perception mock 404 and the localization mock 406 are software simulations of a hardware and / or software perception module (e.g., the perception module 302 in Figure 3) and a hardware and / or software localization module (e.g., the localization module 304 in Figure 3), respectively. In this example, the planning module 408 is in the process of testing within the simulation system 400, and thus, the simulation system 400 may include a software and / or hardware planning module 408. The simulation system may similarly include a controller mock 410 that is a software simulation of a hardware and / or software controller (e.g., the controller 308 in Figure 3). The controller mock 410 outputs the generated command to the simulation module 402, and the simulation module 402 operates according to the generated command to complete the test.

[0030] The components of the simulation system 400 are connected based on a general-purpose interface language. Since each component of the simulation system 400 can be built by different groups / teams using different interface communication or programming languages, the basis of the general-purpose interface language is provided to each team that constructs the hardware components and mocks under test. The general-purpose interface language enables efficient high-speed simulation testing by stabilizing the codebase, forcing the organization to make decisions centrally, and preemptively planning the architecture. This can be adapted to agile workflows.

[0031] In an example where all components with mutual interfaces are written in the same programming language, a general-purpose interface language or interface description language (IDL) may not be required in that case. All that is required is that the interface itself is defined (i.e., exactly which objects, such as data types, are exchanged at the interface). It is beneficial for the interface to be essentially fixed during development. The more fixed the interface is across generations of the system as a whole, the more preferable it is for engineering development because teams can reuse libraries of mocks and real components. In an example where different components are written in different programming languages, then it is possible to utilize IDL or a general-purpose interface language. The language can be a type of language-independent description such that engineers working on components sharing one interface can know that they use different programming languages within each component and that there is a shared language for that interface (i.e., IDL or a general-purpose interface language).

[0032] Furthermore, by having a general-purpose interface language, mocks can be customized, and thus the characteristics of the simulation system can be changed without the need to construct new mocks. For example, mocks can be constructed with built-in fidelity adjustments, or otherwise, a large number of mocks with varying fidelity can be constructed to enable changing the fidelity of the mocks in specific tests according to the desired complexity of the mock's operation (e.g., the fidelity adjustment device 412 of the perception mock 404). The general-purpose interface language reduces the production cost of a large number of compatible mocks of the same component.

[0033] Furthermore, in the case of a combination of software and hardware implementations, some components can be implemented as mocks while some components can be implemented as the original hardware and / or software components. For example, construct a simulation system to test both a planning module (e.g., the planning module 306 in FIG. 3) and a controller (e.g., the controller 308 in FIG. 3), and thus the simulation system can include a perception mock, a localization mock, a hardware and / or software planning module, and a hardware and / or software controller. In another example, construct a simulation system to test a perception module (e.g., the perception module 302 in FIG. 3), a localization module (e.g., the localization module 304 in FIG. 3), and a controller (e.g., the controller 308 in FIG. 3), and thus the simulation system can include a hardware and / or software perception module, a hardware and / or software localization module, a planning mock, and a hardware and / or software controller. This can be referred to as k-subset testing. Since all components utilize the general-purpose interface language, it is possible to easily implement software and hardware replacements and combinations while reducing the engineering cost of testing the system.

[0034] FIG. 5 is a flowchart of a method of a simulation system according to an embodiment. In operation 502, the system generates simulated sensor data by a simulation module. In operation 504, the system simulates the operation of a module of a vehicle system by at least one mock component in connection with the simulation module. In operation 506, during the operation of the simulation system, the system tests at least one test hardware component in connection with the simulation module and at least one mock component based on the generated simulated sensor data.

[0035] The above disclosure provides illustration and description, but is not intended to be exhaustive or to limit implementation to the precise forms disclosed. Modifications and variations are possible in light of the above disclosure, or can be obtained from practice of the implementation.

[0036] Some embodiments relate to systems, methods, and / or computer-readable media at any incorporated level of technical detail considered. The computer-readable media can have computer-readable non-transitory storage media having computer-readable program instructions thereon for causing a processor to perform operations.

[0037] A computer-readable storage medium can be a tangible device that holds and stores instructions for use by an instruction execution device. The computer-readable storage medium can be, for example, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof, but is not limited thereto. A non-exhaustive list of more specific examples of computer-readable storage media includes the following: portable computer diskettes, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disk read-only memory (CD-ROM), digital versatile disks (DVDs), memory sticks, floppy disks, mechanically encoded devices such as punch cards or raised structures in grooves having instructions recorded thereon, and any suitable combination of the foregoing. A computer-readable storage medium as used herein should not be construed to be a signal per se that is transient, such as, for example, radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through a waveguide or other transmission medium (e.g., optical pulses passing through an optical fiber cable), or electrical signals transmitted through a wire.

[0038] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to respective computing / processing devices, to an external computer, or to an external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, optical transmission fibers, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and transfers the computer-readable program instructions for storage in a computer-readable storage medium within each respective computing / processing device.

[0039] The computer-readable program code / instructions for performing the operations may be source code or object code written in any combination of one or more programming languages, including assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state-setting data, configuration data for integrated circuits, or any combination of object-oriented programming languages such as Smalltalk, C++, and procedural programming languages such as the "C" programming language or similar programming languages. The computer-readable program instructions may be executed entirely on the user's computer, partially on the user's computer as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider). In some embodiments, an electronic circuit, including, for example, a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), can execute the computer-readable program instructions by utilizing the state information of the computer-readable program instructions to personalize the electronic circuit for the purpose of performing a plurality of aspects or operations.

[0040] These computer-readable program instructions can be provided to a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus for producing machines, such that the instructions executed via the computer's processor or other programmable data processing apparatus create means for implementing the functions / acts specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions may also be stored in a computer-readable storage medium that, in an article of manufacture including instructions for implementing the functions / acts specified in one or more blocks of the flowchart and / or block diagram, causes a computer, programmable data processing apparatus, and / or other device to function in a particular manner.

[0041] The computer-readable program instructions may also be loaded onto a computer, other programmable apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus, or other device so as to produce a computer-implemented process in such a manner that the instructions executed on the computer, other programmable apparatus, or other device implement the functions / acts specified in one or more blocks of the flowchart and / or block diagram.

[0042] The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer-readable media according to various embodiments. In this regard, each block in the flowchart or block diagram may represent a module, segment, or portion of one or more executable instructions for implementing the specified logical function. The methods, computer systems, and computer-readable media may include additional blocks, fewer blocks, different blocks, or blocks arranged in a different form compared to those depicted in the figures. In some alternative implementations, the functions noted within the blocks may occur out of the order noted in the figures. For example, two blocks shown in succession may in fact be executed concurrently or substantially concurrently, or, depending upon the functionality involved, may sometimes be executed in the reverse order. Similarly, it is noted that each block of the illustrations of the block diagrams and / or flowcharts and combinations of blocks in the illustrations of the block diagrams and / or flowcharts can be implemented by a dedicated hardware-based system that performs the specified functions or acts or by a combination of dedicated hardware and computer instructions.

[0043] It is evident that the systems and / or methods described herein can be implemented in different forms of hardware, firmware, or a combination of hardware and software. The actual dedicated control hardware or software code used to implement these systems and / or methods is not limiting. Thus, the operation and behavior of the systems and / or methods are described herein without reference to specific software code, and it is understood that software and hardware can be designed based on the description herein to implement the systems and / or methods.

[0044] No element, act, or instruction used in this specification should be construed as having a decisive meaning or being essential unless it is explicitly described as such. Similarly, the articles "a" and "an" used in this specification are intended to include one or more items and can be used interchangeably with "one or more". Further, the term "set" as used herein is intended to include one or more items (e.g., combinations of related items, unrelated items, items related and not related to other items), and can be used interchangeably with "one or more". When only one item is intended, "one" or similar language is used. Similarly, terms such as "has", "have", and "having" as used herein are intended as open-ended terms. Further, the expression "based on" is intended to mean "at least partially based on" unless otherwise explicitly stated.

[0045] The descriptions of the various aspects and embodiments have been presented for purposes of illustration, but are not intended to be exhaustive or limited to the disclosed embodiments. Whether a combination of features is recited in the claims and / or disclosed in the specification, such combinations are not intended to limit the disclosure of possible implementations. In fact, many of these features can be combined in ways not specifically recited in the claims and / or disclosed in the specification. Each of the dependent claims listed below can depend directly on only one claim, but the disclosure of possible implementations includes each dependent claim in combination with all the other claims in the claim set. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope of the described embodiments. The terminology used herein is chosen to best explain the principles of the embodiments, practical applications, and technical improvements over technologies found in the marketplace, or to enable others of ordinary skill in the art to understand the embodiments disclosed herein.

Claims

1. In a simulation system based on a vehicle system, a simulation module configured to generate simulated sensor data; at least one mock component in connection with the simulation module and configured to simulate the operation of a module of the vehicle system; at least one test hardware component in connection with the simulation module and the at least one mock component, and whose operation is tested during the operation of the simulation system based on the generated simulated sensor data; comprising the simulation module, the at least one mock component, and the at least one test hardware component are in a connected state based on a general-purpose interface language, the at least one mock component includes a fidelity adjustment device that adjusts the complexity of the operation of the mock component incorporated into the mock component by customizing the mock component using the general-purpose interface language, a simulation system.

2. The at least one test hardware component includes a planning module configured to generate strategic information based on perception information and positioning information, The simulation system according to claim 1.

3. The at least one mock component includes a perception mock configured to generate the perception information based on the generated simulated sensor data, The simulation system according to claim 2.

4. The at least one mock component includes a positioning mock configured to generate positioning information based on the generated simulated sensor data, The simulation system according to claim 2.

5. further comprising a controller mock configured to output at least one command generated by the at least one test hardware component to the simulation module, The simulation system according to claim 1.

6. In a method of a simulation system, generating simulated sensor data by a simulation module; simulating the operation of a module of a vehicle system by at least one mock component in connection with the simulation module; testing at least one test hardware component in connection with the simulation module and the at least one mock component based on the generated simulated sensor data during operation of the simulation system; comprising the simulation module, the at least one mock component, and the at least one test hardware component are in connection based on a general purpose interface language; the at least one mock component includes a fidelity adjustment device that adjusts the complexity of the operation of the mock component incorporated into the mock component by customizing the mock component using the general purpose interface language; a method.

7. the at least one test hardware component includes a planning module configured to generate strategic information based on perception information and localization information; The method according to claim 6.

8. the at least one mock component includes a perception mock configured to generate the perception information based on the generated simulated sensor data; The method according to claim 7.

9. the at least one mock component includes a localization mock configured to generate localization information based on the generated simulated sensor data; The method according to claim 7.

10. further comprising outputting, by a controller mock, at least one command generated by the at least one test hardware component to the simulation module; The method according to claim 6.

11. when executed by at least one processor, causing the at least one processor to: generate simulated sensor data by a simulation module; At least one mock component in a connection state with the simulation module simulates the operation of the modules of the vehicle system; During the operation of the simulation system, at least one test hardware component in a connection state with the simulation module and the at least one mock component is tested based on the generated simulated sensor data; A non-transitory computer storage medium for storing instructions, wherein the simulation module, at least one mock component, and at least one test hardware component are in a connection state based on a general-purpose interface language, the at least one mock component includes a fidelity adjustment device that adjusts the complexity of the operation of the mock component incorporated into the mock component by customizing the mock component using the general-purpose interface language; a non-transitory computer storage medium.

12. The storage medium according to claim 11, wherein the at least one test hardware component includes a planning module configured to generate strategic information based on perceptual information and positioning information. The storage medium according to claim 11.

13. The storage medium according to claim 12, wherein the at least one mock component includes a perception mock configured to generate the perceptual information based on the generated simulated sensor data. The storage medium according to claim 12.

14. The storage medium according to claim 12, wherein the at least one mock component includes a positioning mock configured to generate positioning information based on the generated simulated sensor data. The storage medium according to claim 12.

Images