System and method for intelligent testing environment allocation - Patents.com

Abstract

To properly allocate a task to a test environment.SOLUTION: Provided are a system, method, and device for intelligent test environment allocation. According to embodiments, the method for intelligently determining one or more test environments for testing software of an embedded system, may include: obtaining, by a task allocator, capability information of a plurality of test environments; obtaining, by the task allocator, policy information of a task to be executed for testing the software of the embedded system; determining, by the task allocator, a test environment, from among the plurality of test environments, that satisfies the policy information; and allocating, by the task allocator, the task to the determined test environment, wherein the embedded system may be an in-vehicle electronic control unit (ECU), and wherein the plurality of tests environments may include at least one software-in-the-loop (SIL) test environment, at least one hardware-in-the-loop (HIL) test environment, and at least one virtual ECU (V-ECU) test environment.SELECTED DRAWING: Figure 7A

Description

[Technical field]

[0001] Systems and methods consistent with embodiments of the present disclosure relate to test environments, and more particularly, to systems and methods for intelligently allocating or determining test environments for testing a system. [Background technology]

[0002] In the prior art, various test environments have been introduced to test the functions and software performance of a system. For example, a vehicle embedded system such as an electronic control unit (ECU) can be tested by utilizing at least one Software-in-the-Loop (SIL) test environment, at least one Hardware-in-the-Loop (HIL) test environment, and at least one Virtual-ECU (V-ECU) test environment, among others.

[0003] For example, a SIL test environment may utilize a SIL simulator (SiLS) to evaluate or simulate the implementation of a system on generic or general-purpose hardware instead of the actual ECU on which the system is intended to run. A SIL test environment allows multiple tests to be run in parallel, enabling testing of a partially developed product (e.g., partially virtualized hardware) before the product is fully developed. As a result, a SIL test environment generally provides faster test simulation with high scalability and low cost. Nevertheless, the reliability and accuracy of a SIL test environment is lower because the underlying hardware is generic and / or may differ from the actual tested system.

[0004] On the other hand, HIL test environments utilize a HIL simulator (HiLS) communicatively connected to real elements (e.g., physical ECUs) for evaluating or testing the actual system (e.g., physical ECUs) on which the system is intended to operate, thereby providing highly reliable test results since the true operating environment is faithfully reproduced. Nevertheless, performing tests using HIL test environments can be expensive and time-consuming (e.g., due to low scalability), and cannot provide for multiple parallel tests / simulations given the numerical limitations of available hardware.

[0005] In between, a V-ECU test environment may reproduce or emulate certain ECU environments and functions in software such that the test results are more reliable and accurate than a SIL test environment. Moreover, a V-ECU test environment may be less expensive (e.g., cheaper, etc.) than a HIL test environment, and therefore more scalable and more amenable to a large number of simultaneous test scenarios. Moreover, a V-ECU test environment also enables parallel testing. However, a V-ECU test environment may not be able to fully emulate the ECU being tested (e.g., it may not be able to emulate a system-on-chip (SoC), etc.) and is limited in terms of the actual hardware / product functions that can be reproduced. Furthermore, it may be difficult to reproduce an ECU exactly when the relevant information is not fully known (e.g., in cases where the hardware model is kept secret by its vendor). In addition, a V-ECU test environment requires the conversion of the test object (i.e., one or more ECUs) into a software form (e.g., virtual ECUs, etc.) and may therefore not be suitable for providing instant testing or simulation.

[0006] Considering the above, each test environment (e.g., SIL, HIL, V-ECU, etc.) or methodology offers different trade-offs of costs and benefits. Therefore, appropriate test environment selection and allocation is important to optimize testing and meet diverse testing requirements. Nevertheless, the test environment selection and allocation of tasks (e.g., tests to be performed, etc.) to test environments in the prior art are not optimal and have several limitations and deficiencies, as follows:

[0007] First, prior art systems and methods only involve determining a test environment between multiple test environments of the same type or a generic test environment, such as a HIL or SIL test environment, and do not have the functionality or algorithms to utilize all available test environments and methodologies (including a V-ECU test environment), i.e., prior art systems and methods are unable to intelligently determine or allocate a particular test environment that best suits a particular user's needs and / or testing requirements.

[0008] Furthermore, in the development of embedded systems (e.g., software for vehicles), it is required to test the software at various stages of development. Being limited to only one type of test environment may not be optimal in the development cycle. For example, while performing HIL tests is the most accurate, they may not be appropriate in the early stages of software development, especially when the development of the associated hardware (e.g., ECU, SoC, etc.) is not yet complete. Also, when using HIL test environments, software development may be delayed because it takes a long time to execute a large number of scenarios due to the small number of hardware environments available for testing. Summary of the Invention

[0009] According to embodiments, methods, systems, and apparatuses are provided for intelligently determining one or more test environments for testing software of an embedded system. In particular, embodiments of the present disclosure can utilize multiple test environments to which one or more tasks (e.g., test execution, etc.) are intelligently assigned, thereby allowing multiple tasks to be performed or executed simultaneously in multiple test environments under various test requirements or conditions. Furthermore, by utilizing performance information of the test environments, one or more tasks can be intelligently assigned to optimize testing or simulation while satisfying various requirements (speed, cost, etc.) of the one or more tasks.

[0010] According to an embodiment, there is provided a method for intelligently determining one or more test environments for testing software of an embedded system, the method including: acquiring, by a task allocator, performance information of a plurality of test environments; acquiring, by the task allocator, policy information of tasks to be executed for testing software of the embedded system; determining, by the task allocator, a test environment that satisfies the policy information from among the plurality of test environments; and allocating tasks to the determined test environment by the task allocator, wherein the embedded system is an on-board electronic control unit (ECU); and the plurality of test environments include at least one Software-in-the-Loop (SIL) test environment, at least one Hardware-in-the-Loop (HIL) test environment, and at least one Virtual ECU (V-ECU) test environment.

[0011] According to an embodiment, the performance information includes at least one of processor speed, supported ECU features, supported instruction sets, and cost of use.

[0012] According to an embodiment, the method further includes receiving, by the task allocator, test results of the assigned tasks from the determined test environment, and updating, by the task allocator, performance information of the determined test environment based on the received test results.

[0013] According to an embodiment, the method further includes determining, by the task allocator, another test environment from among the plurality of test environments based on the received test results, and assigning, by the task allocator, a task to the determined another test environment.

[0014] According to an embodiment, obtaining policy information includes obtaining policy information for a plurality of tasks, determining the test environments includes determining a plurality of test environments from among the plurality of test environments to which the plurality of tasks are respectively assigned based on the policy information, and allocating the tasks includes allocating the plurality of tasks for concurrent execution in the plurality of test environments.

[0015] According to an embodiment, the method further comprises storing, by the task allocator, the performance information in a record, the record comprising a table.

[0016] According to an embodiment, the method further includes receiving, by the task allocator, test results of the assigned tasks from the determined test environment, and updating, by the task allocator, the table of records based on the received test results.

[0017] According to an embodiment, there is provided a system for intelligently determining one or more test environments for testing software of an embedded system, the system including a memory for storing instructions and a task allocator, the task allocator being configured to execute the instructions to obtain performance information of a plurality of test environments, obtain policy information of tasks to be executed to test software of the embedded system, determine a test environment from the plurality of test environments that satisfies the policy information, and allocate tasks to the determined test environment, the embedded system being an on-board electronic control unit (ECU), and the plurality of test environments including at least one Software-in-the-Loop (SIL) test environment, at least one Hardware-in-the-Loop (HIL) test environment, and at least one Virtual ECU (V-ECU) test environment.

[0018] According to an embodiment, the performance information includes at least one of processor speed, supported ECU features, supported instruction sets, and cost of use.

[0019] According to an embodiment, the task allocator is further configured to execute instructions to receive test results of the assigned tasks from the determined test environment and to update performance information of the determined test environment based on the received test results.

[0020] According to an embodiment, the task allocator is further configured to execute instructions to determine another test environment from among the plurality of test environments based on the received test results and to assign the task to the determined another test environment.

[0021] According to an embodiment, the task allocator is configured to obtain policy information by executing instructions to obtain policy information for a plurality of tasks. The task allocator is further configured to determine test environments by executing instructions to determine a plurality of test environments among the plurality of test environments to which to respectively allocate the plurality of tasks based on the policy information. The task allocator is further configured to assign tasks by executing instructions to allocate the plurality of tasks for concurrent execution in the plurality of test environments.

[0022] According to an embodiment, the task allocator is further configured to execute the instructions to store the performance information in a record, the record comprising a table.

[0023] According to an embodiment, the task allocator is further configured to execute instructions to receive test results of the assigned tasks from the determined test environment and to update the table of records based on the received test results.

[0024] According to an embodiment, a non-transitory computer-readable recording medium is provided, the non-transitory computer-readable recording medium having instructions executable by a task allocator recorded thereon, the instructions causing the task allocator to perform a method for intelligently determining one or more test environments for testing software of an embedded system, the method including: acquiring, by the task allocator, performance information of a plurality of test environments; acquiring, by the task allocator, policy information of tasks executed to test software of the embedded system; determining, by the task allocator, a test environment that satisfies the policy information from among the plurality of test environments; and allocating, by the task allocator, a task to the determined test environment, wherein the embedded system is an on-vehicle electronic control unit (ECU), and the plurality of test environments include at least one software-in-the-loop (SIL) test environment, at least one hardware-in-the-loop (HIL) test environment, and at least one virtual ECU (V-ECU) test environment.

[0025] According to an embodiment, the performance information includes at least one of processor speed, supported ECU features, supported instruction sets, and cost of use.

[0026] According to an embodiment, a non-transitory computer-readable recording medium may have instructions executable by a task allocator recorded thereon to cause the task allocator to perform the method, and the method may further include receiving, by the task allocator, test results of the assigned tasks from the determined test environment, and updating, by the task allocator, performance information of the determined test environment based on the received test results.

[0027] According to an embodiment, a non-transitory computer-readable recording medium has recorded thereon instructions executable by the task allocator, the instructions causing the task allocator to perform a method further including: determining, by the task allocator, another test environment from among a plurality of test environments based on the received test results; and assigning, by the task allocator, a task to the determined another test environment.

[0028] According to an embodiment, a non-transitory computer-readable recording medium has instructions executable by a task allocator recorded thereon, the instructions causing the task allocator to perform a method, wherein obtaining policy information includes obtaining policy information for a plurality of tasks, determining test environments includes determining a plurality of test environments from among the plurality of test environments to which the plurality of tasks are respectively assigned based on the policy information, and allocating tasks includes allocating the plurality of tasks for concurrent execution in the plurality of test environments.

[0029] According to an embodiment, a non-transitory computer-readable recording medium has instructions recorded thereon that are executable by a task allocator, the instructions causing the task allocator to perform a method that further includes storing, by the task allocator, performance information in the record, the record including a table.

[0030] Additional aspects will be set forth in part in the description that follows, and in part will be apparent from the description, or may be realized by practice of illustrative embodiments of the present disclosure. [Brief description of the drawings]

[0031] [Figure 1] FIG. 1 illustrates a block diagram of a system architecture for intelligently allocating one or more tasks to one or more test environments according to one or more embodiments. [Diagram 2] FIG. 2 illustrates a block diagram of an example hardware-in-the-loop (HIL) testing environment configuration according to one or more embodiments. [Figure 3A]FIG. 3A illustrates a block diagram of an example configuration of a software-in-the-loop (SIL) testing environment in accordance with one or more embodiments. [Figure 3B] FIG. 3B illustrates a block diagram of multiple application software configurations according to one or more embodiments. [Figure 4A] FIG. 4A illustrates a block diagram of an example configuration of a virtual ECU (V-ECU) testing environment according to one or more embodiments. [Figure 4B] FIG. 4B illustrates a block diagram of a V-ECU and multiple application software configurations according to one or more embodiments. [Diagram 5] FIG. 5 illustrates a block diagram of example components of a task assignment system according to one or more embodiments. [Figure 6] FIG. 6 illustrates a block diagram of exemplary communications between a task assignment system and multiple servers according to one or more embodiments. [Figure 7A] FIG. 7A illustrates a block diagram of an exemplary method for intelligently determining a test environment for testing software of a system according to one or more embodiments. [Figure 7B] FIG. 7B illustrates a call flow of a non-exhaustive example use case in which the example method of FIG. 7A may be implemented, according to one or more embodiments. [Figure 8A] FIG. 8A illustrates a block diagram of an example method for managing information according to one or more embodiments. [Figure 8B] FIG. 8B illustrates a call flow of a non-exhaustive example use case in which the example method of FIG. 8A may be implemented, according to one or more embodiments. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0032] The features, advantages and importance of preferred embodiments of the present disclosure will now be described with reference to the accompanying drawings, in which like reference numbers refer to like elements, and in which:

[0033] The following detailed description of the preferred embodiments refers to the accompanying drawings. The foregoing disclosure provides illustration and description, but is not intended to be exhaustive or to limit the implementation to the precise form disclosed. Modifications and variations are possible in light of the above disclosure or may be acquired from practice of the implementations. Furthermore, one or more features or components of one embodiment may be incorporated or combined with another embodiment (or one or more features of the other embodiments). Furthermore, in the flowcharts and descriptions of operations provided below, it will be understood that one or more operations may be omitted, one or more operations may be added, one or more operations may be performed (at least partially) simultaneously, and the order of one or more operations may be rearranged.

[0034] Although particular combinations of features are recited in the claims and / or disclosed in the specification, these combinations are not intended to limit the disclosure of possible implementations. Indeed, many of these features may be combined in ways not specifically recited in the claims and / or disclosed in the specification. Although each dependent claim listed below may depend directly on only one claim, the disclosure of possible embodiments includes each dependent claim in combination with all other claims in the claim set.

[0035] No element, act, or instruction used herein should be construed as critical or required unless expressly recited as such. Additionally, the articles "a" and "an" as used herein are intended to include one or more items and may be used interchangeably with "one or more." When only one item is intended, the term "one" or similar terms are used. Additionally, as used herein, "has," "having," "includes," "including," or the like are intended to be open-ended terms. Additionally, the phrase "based on" is intended to mean "based at least in part on," unless expressly stated otherwise. Additionally, phrases such as "at least one of 'A' and 'B'" or "at least one of 'A' or 'B'" should be understood as including only A, only B, or both A and B.

[0036] Throughout this specification, references to "one embodiment," "an embodiment," "a non-limiting preferred embodiment," or similar terms mean that a particular feature, structure, or characteristic described in connection with the illustrated embodiment is included in at least one embodiment of the current solution. Thus, throughout this specification, the phrases "in one embodiment," "in an embodiment," "a non-limiting preferred embodiment," and similar expressions may, but do not necessarily, all refer to the same embodiment.

[0037] Moreover, the described features, advantages, and characteristics of the present disclosure may be combined in any suitable manner in one or more embodiments. Those skilled in the art will recognize, upon consideration of the description herein, that the present disclosure can be practiced without one or more of the particular features or advantages of a particular embodiment. In other instances, certain embodiments may recognize additional features and advantages that may not be present in all embodiments of the present disclosure.

[0038] In one embodiment of the disclosure described herein, the display page may include information residing in the memory of the computing device, which may be transmitted from the computing device to the database center and vice versa over a network. The information may be stored in the memory of each computing device, in a data storage device at the edge of the network, or in a server at the database center. The computing device or mobile device may accept non-transitory computer-readable media that may include instructions, logic, data, or code that may be stored in a persistent or temporary memory of the mobile device, or that may affect or initiate an operation by the mobile device in some way. Similarly, one or more servers may communicate with one or more mobile devices over a network to transmit computer files residing in the memory. The network may include, for example, the Internet, a wireless communication network, or any other network for connecting one or more mobile devices to one or more servers.

[0039] Additionally, although the present disclosure is described with respect to embedded systems in a vehicle (e.g., electronic control units (ECUs), etc.), it will be understood that the embodiments of the present disclosure described herein may also be applied to any other suitable type of system, such as radar systems, robotic systems, power systems, and the like, without departing from the scope of the present disclosure.

[0040] Embodiments of the present disclosure provide methods, systems, and apparatus for intelligently determining one or more test environments for testing software of a system. In particular, methods, systems, and apparatus consistent with embodiments of the present disclosure provide for intelligent determination of optimal test environments for allocating one or more tasks (e.g., test execution, etc.) based on the requirements of the tasks and the capabilities of the available test environments.

[0041] Finally, embodiments of the present disclosure can utilize multiple different types of test environments, thereby allowing multiple tests to be performed or run simultaneously in multiple test environments under various test requirements or conditions. Furthermore, by utilizing performance information of the test environments, one or more tasks can be intelligently assigned to optimize testing and meet the various requirements (speed, cost, etc.) of the one or more tasks.

[0042] FIG. 1 illustrates a block diagram of a system architecture 100 for intelligently allocating one or more tasks to one or more testing environments according to one or more embodiments.

[0043] 1, system architecture 100 may include a task assignment system 110 and a plurality of test environments 120. Task assignment system 110 may be communicatively coupled to each of the plurality of test environments 120 and configured to perform operations for intelligently determining one or more test environments from among the plurality of test environments 120 and for assigning one or more tasks (e.g., running / simulating a test, performing a performance check, scheduling a test run, etc.) to the determined test environment. Further description regarding components included in task assignment system 110 is provided below with reference to FIG. 5, and further description of example operations that may be performed by task assignment system 110 is provided below with reference to FIG. 6-8B.

[0044] In addition, as shown in FIG. 1, the multiple test environments may include at least one hardware-in-the-loop (HIL) test environment 120-1, at least one software-in-the-loop (SIL) test environment 120-2, and at least one virtual electronic control unit (V-ECU) test environment 120-3. Each of the test environments 120-1 to 120-3 may be communicatively coupled to the task assignment system 110 and may provide to and receive from the task assignment system 110 one or more signals, information, data, or the like, related to one or more tasks. Further description of the operation of the HIL test environment 120-1 and the components included therein is provided below with reference to FIG. 2, further description of the operation of the SIL test environment 120-2 and the components included therein is provided below with reference to FIG. 3A and FIG. 3B, and further description of the operation of the V-ECU test environment 120-3 and the components included therein is provided below with reference to FIG. 4A and FIG. 4B.

[0045] It will be appreciated that the above test environments 120-1 through 120-3 are merely examples of possible test environments, and the scope of the present disclosure should not be limited thereto. In particular, more or fewer test environments may be included in test environment 120, and / or any other suitable type of test environment (e.g., a model-in-the-loop (MIL) test environment, a processor-in-the-loop (PIL) test environment, etc.) may also be included in test environment 120 without departing from the scope of the present disclosure.

[0046] 2, which illustrates a block diagram of an example hardware-in-the-loop (HIL) testing environment 200 according to one or more embodiments. HIL testing environment 200 may correspond to HIL testing environment 120-1 of FIG. 1, and thus it will be understood that features described herein with reference to HIL testing environments 120-1 and 200 are mutually applicable unless expressly stated otherwise.

[0047] In general, HIL test environment 200 may be configured to manage one or more tasks related to testing of one or more test objects, such as (but not limited to) running or simulating a test, scheduling a test run, or the like. In particular, HIL test environment 200 may be communicatively connected to one or more developed (or partially developed) hardware / physical components and may simulate a real-world environment or actual use case therewith to test or evaluate the one or more hardware / physical components. For example, an engine ECU may be developed and tested before being integrated into a vehicle. In such an example, instead of testing the engine ECU with an actual engine, HIL test environment 200 may perform a simulation including engine hardware and software interacting with the engine ECU.

[0048] 2, the HIL testing environment 200 may include at least one communication interface 210, at least one processor 220, at least one storage device 230, and at least one HIL simulator (HiLS) 240. The at least one HiLS 240 may be communicatively coupled to at least one electronic control unit (ECU) 250.

[0049] The communication interface 210 may include transceiver-like components (e.g., a transceiver, separate receivers and transmitters, etc.) that enable the components of the HIL testing environment 200 to communicate with each other and with one or more components external to the HIL testing environment 200, such as via wired connections, wireless connections, or a combination of wired and wireless connections. For example, the communication interface 210 may connect the processor 220 to the HiLS 240, thereby enabling them to communicate with each other and interoperate when performing one or more tests / simulations. As another example, the communication interface 210 may couple the HIL testing environment 200 (or one or more components included therein) to a task assignment system (e.g., such as the task assignment system 110 of FIG. 1), thereby enabling them to communicate with each other and interoperate.

[0050] According to an embodiment, communication interface 210 may include a bus, an Ethernet interface, an optical interface, a coaxial interface, an infrared interface, a radio frequency (RF) interface, a universal serial bus (USB) interface, a WiFi interface, a cellular network interface, a software interface, or the like. According to an embodiment, communication interface 210 may include at least one Controller Area Network (CAN) bus configurable to communicatively couple HIL test environment 200 (or one or more components included therein) to multiple ECUs. In some implementations, the CAN bus (or one or more associated functions) may be defined in software form, for example, via virtualization (e.g., virtualized network functions, etc.).

[0051] At least one processor 220 may include one or more processors that can be programmed to perform the functions or operations described herein. For example, processor 220 may be configured to execute computer-readable instructions stored on a storage medium (e.g., storage device 230, etc.) to thereby perform one or more acts or operations described herein. According to an embodiment, processor 220 may be configured to receive one or more signals (e.g., via communication interface 210, etc.) defining one or more instructions for performing one or more operations. Furthermore, processor 220 may be implemented in hardware, firmware, or a combination of hardware and software. Processor 220 may be a central processing unit (CPU), a graphics processing unit (GPU), an accelerated processing unit (APU), a microprocessor, a microcontroller, a digital signal processor (DSP), a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), and / or another type of processing or computing element.

[0052] According to an embodiment, the processor 220 may be configured to receive one or more information / signals from a task assignment system (e.g., the task assignment system 110 of FIG. 1 ), process or analyze the received information / signals, and perform one or more actions based on the processing or analysis. According to an embodiment, the processor 220 may retrieve one or more information or data stored in at least one storage device 230 based on the processing or analysis, and provide it to the task assignment system. Further, the processor 220 may generate a schedule for performing tests based on the processing or analysis, and store the schedule in the storage device 230. Further, the processor 220 may initiate, direct, and / or configure the HiLS 240 to perform one or more tests / simulations based on the processing or analysis.

[0053] At least one storage device 230 may include one or more storage media suitable for storing data, information, and / or computer-readable instructions therein. According to an embodiment, storage device 230 may include random access memory (RAM), read-only memory (ROM), and / or another type of dynamic or static storage device (e.g., flash memory, magnetic memory, and / or optical memory) that stores information and / or instructions for use by processor 220. Additionally or alternatively, storage device 230 may include hard disks (e.g., magnetic disks, optical disks, magneto-optical disks, and / or semiconductor disks), compact disks (CDs), digital versatile disks (DVDs), floppy disks, cartridges, magnetic tapes, and / or another type of non-transitory computer-readable medium with corresponding drives.

[0054] According to an embodiment, the storage device 230 may be configured to store information related to the capabilities of the HIL testing environment 200, such as, but not limited to, capacity information of a test server on which the HIL testing environment 200 is deployed / provided, processing performance of the HiLS 240, past test results, usage costs, or the like. According to an embodiment, the performance information may include processing / test speed (or a parameter from which the processing / test speed can be determined). Additionally, the storage device 230 may store computer readable instructions that, when executed by one or more processors (e.g., the processor 220), cause the one or more processors to perform one or more operations described herein.

[0055] At least one HiLS 240 may be communicatively coupled to at least one system and configured to perform one or more real-time environmental simulations or tests on the at least one system. According to an embodiment, the HiLS 240 may be communicatively coupled to the at least one system via the communication interface 210.

[0056] 2, HiLS 240 is communicatively coupled to an ECU (i.e., ECU 250), which may be an embedded system of the vehicle, although it will be appreciated that any other suitable physical components or systems (e.g., developed hardware, etc.) may be applied without departing from the scope of the present disclosure. HiLS 240 may be deployed in the form of hardware, software, or a combination thereof. In this regard, software components (e.g., software programming, functional algorithms, etc.) associated with HiLS 240 may be provided or stored on one or more storage media (e.g., at least one storage device 230, etc.) communicatively coupled to HiLS 240.

[0057] HiLS 240 may be configured to perform one or more simulations or tests. According to an embodiment, HiLS may perform electrical emulation of at least one sensor and at least one actuator, which acts as an interface between the simulation and the system / hardware element being tested. Parameters of each electrically emulated sensor and actuator are provided by HiLS 240 to ECU 250, and the response and / or performance of ECU 250 is monitored by HiLS 240. According to an embodiment, HiLS 240 may retrieve one or more predefined configurations or conditions from one or more storage media (e.g., storage device 230, etc.) and perform the simulation or emulation based on the predefined configurations or conditions. It will be understood that HiLS 240 may be configured to perform any other suitable operations to simulate or emulate a real environment or use case for testing ECU 250 without departing from the scope of the present disclosure.

[0058] ECU 250 may include any suitable type of embedded system or electronic control unit. According to an embodiment, ECU 250 may include one or more electronic control units utilized to perform one or more operations of a vehicle, such as, but not limited to, an engine ECU, a steering ECU, an infotainment ECU, an ECU for a passive safety system (e.g., airbags, seat belts, etc.), or the like. In this regard, it will be understood that any other suitable system, element, or product to be tested (e.g., developed hardware, etc.) may be substituted for ECU 250 without departing from the scope of the present disclosure.

[0059] Additionally, it is understood that the HIL testing environment 200 described above is merely an example of a HIL testing environment, and the scope of the present disclosure should not be limited thereto. For example, a HIL testing environment may include more or fewer components and / or may be configured in a different manner than described herein without departing from the scope of the present disclosure.

[0060] 3A, in accordance with one or more embodiments, a block diagram of an example software-in-the-loop (SIL) testing environment 300. SIL testing environment 300 may correspond to SIL testing environment 120-2 of FIG. 1, and thus it will be understood that features described herein with reference to SIL testing environments 120-2 and 300 are mutually applicable unless expressly stated otherwise.

[0061] Similar to the HIL test environment 200 described above, the SIL test environment 300 may be configured to manage one or more tasks associated with testing one or more test objects, such as (but not limited to) running or simulating a test, scheduling a test run, or the like. Nevertheless, unlike the HIL test environment 200, which is communicatively connected to one or more physical / hardware components and performs tests / simulations thereon, the SIL test environment 300 may simply acquire or receive one or more software components (e.g., application software, virtual functions, programming code, functional algorithms, etc.) and perform software-based tests / simulations thereon. In short, the SIL test environment 300 may be generated, deployed, and executed within any suitable computing device or environment, without the need for a connection to the physical / hardware components being tested as the HIL test environment 200 does.

[0062] Referring to FIG. 3A, the SIL test environment 300 may include at least one communication interface 310, at least one processor 320, and at least one memory device 330, each of which may have roles and functions similar to those of the communication interface 210, the processor 220, and the memory device 230 of the HIL test environment 200 of FIG. 2, and therefore, duplicate descriptions thereof may be omitted below for the sake of brevity.

[0063] In this regard, unlike the HIL testing environment 200, which includes a simulator (e.g., HiLS 240) communicatively coupled to physical / hardware components, the SIL testing environment 300 may utilize at least one software-based SIL simulator (SiLS) to perform one or more tests / simulations on one or more test objects. That is, the SiLS may be disposed in the form of a software application or computer program executable by the processor 320 and stored or provided in the storage device 330.

[0064] 3A, at least one application software 340 may be provided to the SIL test environment 300 (via the communication interface 310) for testing. At least one application software 340 may be stored in the storage device 330 and retrieved therefrom for testing when needed. According to an embodiment, at least one application software 340 may be provided or stored in one or more storage media (e.g., external servers, etc.) external to the SIL test environment 300 and provided or provided to the SIL test environment 300 on the fly when a test / simulation is performed. For example, in response to a test / simulation being performed, at least one SiLS executed by the processor 320 (or any other suitable component of the SIL test environment 300) may retrieve or receive the application software 340 (e.g., programming code, algorithms, etc.) in real time or near real time from one or more external storage media (via the communication interface 310) and perform a test / simulation thereon.

[0065] To this end, the SiLS and at least one application software may be created or deployed in any suitable programming language or computing technology, such as, but not limited to, Java, C++, or the like.

[0066] According to an embodiment, SIL testing environment 300 may run multiple tests / simulations simultaneously. For example, SIL testing environment 300 may run multiple tests / simulations of a single application software in parallel, run one test / simulation of multiple application software in parallel, or run multiple tests / simulations of multiple application software in parallel.

[0067] 3B illustrates a block diagram of a configuration of multiple application software, such as application software 340-1 associated with an advanced driver assistance system (ADAS), application software 340-2 associated with an in-vehicle infotainment (IVI) system, and a simulated ECU 340-3, in accordance with one or more embodiments.

[0068] The multiple application software may be communicatively connected to each other and to the SIL test environment 300 via the virtualized network 310-1. In this regard, the virtualized network 310-1 described herein may refer to any suitable physical network (e.g., Ethernet, WiFi, etc.) having one or more virtualized network functions (e.g., CAN bus, etc.) implemented therein and configured to connect the multiple application software to the SIL test environment 300. It will be appreciated that the virtualized network 310-1 is merely an example of a communication channel, and any suitable components (e.g., software interfaces, etc.) may be used instead or in addition to connect the multiple application software to the SIL test environment 300. In this manner, the SIL test environment 300 may receive or use multiple application software in parallel to perform multiple tests / simulations thereof.

[0069] It will be appreciated that the application software 340-1, application software 340-2, and simulated ECU 340-3 shown in FIG. 3B are merely examples of application software, and any other suitable application software (e.g., application software associated with an integrated circuit (IC), central application software, etc.) may be applied in a similar manner. It will further be appreciated that the configuration of FIG. 3B may be applied to multiple programming codes or functional algorithms. For example, an ADAS first algorithm and an ADAS second algorithm may be provided to and simulated / tested by a SIL test environment in a similar manner as described herein.

[0070] 4A, which illustrates a block diagram of an example configuration of a virtual ECU (V-ECU) testing environment 400 according to one or more embodiments. V-ECU testing environment 400 corresponds to V-ECU testing environment 120-3 of FIG. 1, and thus it will be understood that features described herein with reference to V-ECU testing environments 120-3 and 400 are mutually applicable unless expressly stated otherwise.

[0071] V-ECU test environment 400 may be similar to SIL test environment 300 described above with reference to Figures 3A and 3B in that the tests / simulations performed by V-ECU test environment 400 are also software-based. Referring to Figure 4A, SIL test environment 400 may include at least one communication interface 410, at least one processor 420, and at least one storage device 430, each of which may have similar roles and functions to communication interface 310, processor 320, and storage device 330 of SIL test environment 300 in Figure 3A, and therefore, duplicated descriptions thereof may be omitted below for brevity.

[0072] As shown in Figure 4A, at least one application software 440 may be provided to the V-ECU testing environment 400 for testing / simulation. The application software 440 may be similar to the application software 340 described above with reference to Figures 3A and 3B, and therefore, a duplicate description thereof may be omitted below for brevity. Furthermore, in addition to the application software (e.g., application software 340 / 440), at least one V-ECU may be provided to the V-ECU testing environment 400 for testing.

[0073] According to an embodiment, V-ECU 450 may differ from a simulated ECU (e.g., examples of application software 340 / 440) in that V-ECU 450 may include application software, programming code, functional algorithms, or the like that defines a final ECU (e.g., in terms of design, development, manufacturing, etc.), whereas a simulated ECU may include application software, programming code, functional algorithms, or the like that defines a general or non-final ECU.

[0074] According to another embodiment, V-ECU 450 may include application software, programming code, functional algorithms, or the like that defines one or more essential operations or functions of an ECU (e.g., a central ECU), and application software tested / simulated with V-ECU 450 may include application software, programming code, functional algorithms, or the like that defines general or non-essential operations or functions of the ECU.

[0075] According to an embodiment, V-ECU testing environment 400 may run multiple tests / simulations simultaneously. For example, V-ECU testing environment 400 may run multiple tests / simulations of one V-ECU in parallel, run one test / simulation of multiple V-ECUs in parallel, and / or run multiple tests / simulations of at least one V-ECU and at least one application software in parallel.

[0076] FIG. 4B illustrates a block diagram of a V-ECU and a plurality of application software configurations according to one or more embodiments. Specifically, FIG. 4B illustrates an example of communication between a plurality of simulated ECUs 440-1 and 440-2 and a V-ECU 450-1. According to an embodiment, the V-ECU 450-1 may include a virtual center ECU. Furthermore, it is understood that one or more configurations or operations associated with FIG. 4B may be similar to configurations or operations described above with reference to FIG. 3B, and therefore, the overlapping description thereof may be omitted below for brevity.

[0077] 4B , multiple simulated ECUs 440-1 and 440-2 and V-ECU 450-1 may be communicatively connected to each other and to V-ECU testing environment 400 via virtualization network 410-1. It will be appreciated that virtualization network 410-1 is merely an example of a communication channel, and any suitable components (e.g., software interfaces, etc.) may be used instead or in addition to connect multiple simulated ECUs 440-1 and 440-2 and V-ECU 450-1 to V-ECU testing environment 400. In this manner, the V-ECU testing environment may use one or more application software in parallel with the V-ECUs to perform multiple tests / simulations thereof.

[0078]

[0036] Referring now to Figure 5, Figure 5 illustrates a block diagram of example components of a task assignment system 500, in accordance with one or more embodiments. Task assignment system 500 may correspond to task assignment system 110 of Figure 1, and thus it will be understood that features described herein with reference to task assignment systems 110 and 500 are mutually applicable unless expressly stated otherwise.

[0079] Referring to FIG. 5, the task allocation system 500 may include at least one communication interface 510, at least one processor 520, and at least one memory device 530, each of which may have at least some of the same functions, roles, and / or examples as the communication interfaces 210 / 310 / 410, processors 220 / 320 / 420, and memory devices 230 / 330 / 430 described above, and therefore duplicate descriptions related thereto may be omitted below for brevity.

[0080] Additionally, the task assignment system 500 may include at least one input component 540 and at least one output component 550. The at least one input component 540 may include one or more components (e.g., a touch screen display, a button, a switch, a microphone, a sensor, etc.) that allow the task assignment system 500 to receive information. The at least one output component 550 may include one or more components (e.g., a display, a speaker, one or more light emitting diodes (LEDs), etc.) that provide output information from the task assignment system 500.

[0081] To this end, the components of the task assignment system 500 may interoperate with each other and / or with one or more components external to the task assignment system 500 to perform one or more actions associated with one or more tasks.

[0082] In particular, the communications interface 510 can communicatively connect the task assignment system 500 (or one or more components thereof) to one or more test environments (or components that provide or deploy the test environments, such as a test server), thereby enabling the task assignment system 500 to interoperate with the one or more test environments to manage one or more tasks.

[0083] Additionally, the communication interface 510 may communicatively connect the task assignment system 500 (or one or more components therein) to one or more terminals / devices of a user (e.g., a user wishing to specify a policy or test parameters, etc.) via a wireless network connection and / or via a wired connection, and may receive one or more user inputs from the one or more terminals / devices. According to an embodiment, the one or more user inputs may include information defining a test policy, test requirements, test parameters, test conditions, or the like. In this regard, the communication interface 510 may be communicatively connected to the storage device 530 and may provide the received one or more user inputs to the storage device 530 for storing the received one or more user inputs therein, may be communicatively connected to the processor 520 and may provide the received one or more user inputs to the processor 520 in real-time or near real-time for processing, and may be communicatively connected to the output component 550 and may provide the received one or more user inputs to the output component 550 so that the one or more user inputs can be output (e.g., rendered, displayed, etc.) via the output component 550, or the like.

[0084] In addition to providing one or more user inputs via a terminal / device, a user may provide one or more user inputs directly to the task assignment system 500, for example, via the input component 540. In this regard, the input component 540 may be communicatively coupled to the storage device 530 to store the received one or more user inputs in the storage device 530, may be communicatively coupled to the processor 520 to provide the received one or more user inputs in real time or near real time to the processor 520 for processing, may be communicatively coupled to the output component 550 to provide the received one or more user inputs to the output component 550 so that the one or more user inputs can be output (e.g., rendered, displayed, etc.) via the output component 550, may be communicatively coupled to the communication interface 510 to provide the received one or more user inputs to one or more components external to the task assignment system 500 (e.g., for storage, for further processing, etc.), or the like.

[0085] The storage device 530 may be configured to receive information or data from an external component (via the communication interface 510), from the input component 540, from the processor 520, or the like, and may be configured to store information or data therein. For example, the storage device 530 may store information or data related to a test / simulation strategy, store information or data related to past test / simulation results of one or more test environments communicatively connected to the task assignment system, store information or data related to performance of one or more test environments, or the like. According to an embodiment, the storage device 530 may be configured to store information or data related to performance of one or more test environments in a table (or any other suitable record format).

[0086] Additionally, storage device 530 may provide (e.g., continuously, periodically, in response to a request or event, etc.) one or more stored data or information to at least one processor 520. Additionally, storage device 530 may store computer-readable instructions that, when executed by at least one processor 520, cause at least one processor 520 to perform one or more of the operations discussed herein.

[0087] At least one processor 520 may be configured to intelligently determine one or more test environments for running one or more tests / simulations on one or more software of the system. Specifically, the processor 520 may determine at least one test environment that satisfies one or more requirements of a task from among a plurality of test environments communicatively connected to the task allocation system 500, and assign the task to the determined test environment. Thus, the processor 520 may be referred to herein as a "task allocator."

[0088] According to an embodiment, processor 520 may be configured to obtain one or more pieces of information or data (sometimes referred to herein as "performance information") associated with the performance of one or more test environments. For example, processor 520 may request, retrieve, or receive one or more pieces of performance information in real time, near real time, or the like, from multiple test environments (or components providing or deploying multiple test environments) via communications interface 510. Alternatively, or in addition, processor 520 may request, retrieve, or receive one or more stored performance information from one or more storage media (e.g., storage device 530, etc.).

[0089] As discussed above, the performance information may include (but is not limited to) capacity information, processor / test speed (or parameters that can determine the speed at which a processor can execute a test), features / algorithms emulated or supported, instruction sets supported, lease fees, cost of use, or the like. For example, at least a portion of the capacity information may be obtained (by the test environment or components associated therewith) via a CPUID instruction that allows one or more software to learn details (e.g., capacity, processing power, load, etc.) of an associated component (e.g., processor, etc.).

[0090] Additionally, the processor 520 may be configured to obtain one or more pieces of information or data (sometimes referred to herein as "policy information") of policy associated with a task. For example, the processor 520 may request, retrieve, or receive one or more pieces of policy information from a user (e.g., via the input component 540, a terminal / device communicatively connected to the communication interface 510, etc.) in real-time, near real-time, or the like. Alternatively, or in addition, the processor 520 may request, retrieve, or receive one or more stored policy information from the storage device 530. Additionally, the processor 520 may obtain policy information for a number of tasks (e.g., tests to be performed, tests scheduled, etc.).

[0091] The policy information may include one or more information or parameters that define one or more requirements / conditions for placing the tasks, such as, for example, but not limited to, a target test duration, a target cost, test functionality, and any other suitable parameters, constraints, or requirements of the test. For example, the policy information may include information that defines that a task (e.g., a test to be performed) requires short / long test time, requires costs to be within a budget, requires testing on specific functions / algorithms, and / or the like.

[0092] According to an embodiment, the processor 520 may be configured to determine a test environment (among the multiple test environments) that satisfies the policy information of the task, and then assign the task to the determined test environment. For example, the processor 520 may determine, based on the performance information of each of the multiple test environments, which of the multiple test environments is capable of executing a test / simulation that complies with the requirements / conditions defined in the policy information, and assign or place the task accordingly.

[0093] According to an embodiment, based on a determination that multiple test environments satisfy the requirements / conditions defined in the policy information, processor 520 may assign or place the task to the test environment that can satisfy the most requirements / conditions (e.g., test environment 1 that satisfies three requirements / conditions may be selected over test environment 2 that fulfills two requirements / conditions, etc.) and / or to the test environment that can provide optimal performance / results (e.g., the test environment that can complete the test / simulation in the shortest period of time, the test environment that is least costly, etc.).

[0094] According to an embodiment, based on a determination that none of the multiple test environments meets the requirements / conditions defined in the policy information, the processor 520 may generate a notification message to inform a user (e.g., a user that requested the test / simulation, a user that defined the policy information, etc.) regarding the determination.

[0095] Alternatively, before generating or providing a notification message to the user, the processor 520 may determine one or more combinations of a portion of the multiple test environments that may satisfy the requirements / conditions, and allocate / distribute tasks to the combination of test environments accordingly. For example, assuming that the multiple test environments include test environment 1 and test environment 2, and based on a determination that neither test environment 1 nor test environment 2 has the capability to satisfy the requirements / conditions defined in the policy information, the processor 520 may determine (based on the performance information of test environment 1 and test environment 2) whether test environment 1 and test environment 2 as a combination satisfy the requirements / conditions. Based on a determination that test environment 1 and test environment 2 as a combination may satisfy the requirements / conditions, the processor 520 may appropriately allocate / distribute tasks to test environment 1 and test environment 2 according to their respective available capabilities (e.g., allocate a portion of the tasks to test environment 1 and the remaining portion of the tasks to test environment 2, etc.). Otherwise, the processor 520 may generate and provide a notification message to inform the user that there is no available test environment that can perform the tests / simulations according to the policy information.

[0096] Given the above, it will be appreciated that processor 520 may be configured to allocate or arrange a plurality of tasks in a similar manner, for example, processor 520 may obtain policy information for a plurality of tasks, determine one or more test environments (among a plurality of test environments) to which the plurality of tasks are respectively allocated based on the policy information, and allocate the plurality of tasks for testing / simulation in the determined one or more test environments in a similar manner as described above.

[0097] Further, according to an embodiment, processor 520 may be configured to generate a record of the received or stored performance information (e.g., stored in storage device 530, etc.). For example, upon obtaining or receiving new performance information, processor 520 may generate a record of the performance information and store the record (e.g., storing the record in storage device 530, storing the record in an external storage medium communicatively connected to task assignment system 500 via communication interface 510, etc.). According to an embodiment, processor 520 may generate the record based on the performance information of the multiple test environments, and the record may include a table including the performance information for each of the multiple test environments.

[0098] Additionally, the processor 520 may be configured to update the records of performance information. For example, upon assigning or distributing one or more tasks to a test environment, the processor 520 may monitor the performance of the test environment, receive one or more test results of the assigned tasks, or the like, from the test environment, and update the relevant information in the records accordingly. For example, whenever one or more test results indicate a failure of the test execution (e.g., due to an exception or a timeout for not completing a task within a specified period of time), the processor 520 may update the records (or a table contained therein) to avoid reassigning that same task to the same environment. Similarly, whenever one or more test results indicate a particular parameter (e.g., time or speed) that differs from that expected or reflected in the records, the processor 520 may update the records accordingly to more accurately reflect the current capabilities of the corresponding test environment.

[0099] Additionally, upon receiving a task failure report or determining a task failure, processor 520 may be configured to relocate the failed task to another test environment. For example, whenever a task failure is reported due to a timeout, processor 520 may relocate the task to another test environment whose associated performance information indicates a higher test speed than the previous test environment to which the task was assigned.

[0100] Additionally, processor 520 may be configured to arrange one or more tasks based on test results of a previous test plan. For example, a first series of tests or simulations are performed to test software, and the source code of the software is subsequently updated. In this regard, processor 520 may optimize task allocation for testing (e.g., reduce test execution time, etc.) whenever testing the software with the updated source code in a second series of tests or simulations based on the results of the first series of tests or simulations.

[0101] It will be understood that the operation and functionality of the components of task assignment system 500 as described above are merely examples of possible embodiments, and the scope of the present disclosure should not be limited thereto. For example, task assignment system 500 may include more or fewer components than described, one or more components of task assignment system 500 may be arranged in a different configuration, one or more components of task assignment system 500 may include more or less functionality, or may be in a similar configuration.

[0102] In view of the above, embodiments of the present disclosure provide a task allocation system communicatively coupled to a plurality of test environments, where the task allocation system may intelligently manage one or more tasks. For example, the task allocation system may determine an optimal test environment for executing one or more tests / simulations, track and update performance information of the plurality of test environments, intelligently allocate / re-allocate one or more tasks based on the updated performance information, or the like. In this regard, it will be appreciated that the plurality of test environments may include one or more HIL test environments, one or more SIL test environments, and / or one or more V-ECU test environments, although other embodiments are not limited thereto. For example, in other embodiments, other test environments may additionally (or instead) be applied, such as one or more MIL test environments, one or more PIL test environments, or the like.

[0103] It is further contemplated that one or more of the components, configurations, operations, or the like described above with reference to Figures 1-5 may be located or implemented on one or more servers, such as cloud servers, server clusters, and the like. Thus, according to an embodiment, communications between the task assignment system and the multiple test environments and operations included therein may be presented in the form of communications between the servers.

[0104] 6, which illustrates a block diagram of an example communication between a task allocation system and a plurality of servers according to one or more embodiments. The task allocation system 610 may correspond to the task allocation system described above with reference to FIGS. 1 and 5, and each of the first test server 620-1 and the second test server 620-2 may provide or deploy one or more test environments, such as (but not limited to) one or more HIL test environments, one or more SIL test environments, and / or one or more V-ECU environments described above with reference to FIGS. 1 to 4B. It will therefore be understood that the first test server 620-1 and / or the second test server 620-2 may include one or more components and one or more configurations described above with reference to FIGS. 2 to 4B.

[0105] To this end, the task assignment system 610 may be communicatively connected to the first test server 620-1 and the second test server 602-2, respectively. For example, a communication interface of the task assignment system 610 may be communicatively connected to a communication interface of the first test server 620-1 and a communication interface of the second test server 620-2 via a wireless network connection and / or a wired network connection. It will be understood that the task assignment system 610 may be communicatively connected to less than two test servers, more than two test servers, or the like, in a manner similar to that described herein without departing from the scope of the present disclosure.

[0106] Next, an embodiment of the operation between the task assignment system 610 and the test servers (ie, the first test server 620-1 and the second test server 620-2) will be described below with reference to FIGS. 7A to 8B.

[0107] Referring first to FIG. 7A, FIG. 7A illustrates a block diagram of an example method 700 for intelligently determining one or more test environments for testing software of a system, according to one or more embodiments. One or more operations of the method 700 may be performed by at least one task allocator (e.g., at least one processor, etc.) of the task allocation system 610 of FIG. 6. Further, according to an embodiment, the system is an embedded system, such as (but not limited to) an in-vehicle ECU. Furthermore, the test environments associated with one or more operations of the method 700 may be located or provided in the first test server 620-1 and the second test server 620-2 of FIG. 6.

[0108] Additionally, Figure 7B illustrates a call flow of a non-exhaustive example use case in which method 700 may be implemented, according to one or more embodiments. Accordingly, below, one or more operations of method 700 of Figure 7A may be described with reference to Figure 7B.

[0109] 7A, at operation S710, the task assignment system may be configured to obtain one or more pieces of performance information. For example, a task allocator of the task assignment system may generate a request signal or query message requesting performance information of available test environments (e.g., test environments associated with the task assignment system) and subsequently transmit the request signal or query message to one or more test servers communicatively coupled to the task assignment system. In response, the one or more test servers may provide the latest or most updated performance information to the task assignment system.

[0110] Referring to the use case example of FIG. 7B, the task allocator of the task allocation system 610 may first obtain (in operation S710-1) from a first test server 620-1 performance information of one or more test environments provided or deployed to the first test server 620-1, and then obtain (in operation S710-2) from a second test server 620-2 performance information of one or more test environments provided or deployed to the second test server 620-2.

[0111] It will be appreciated that the task allocator may obtain performance information from multiple test servers in any suitable sequential manner, for example, the task allocator may perform operations S710-1 and S710-2 simultaneously to obtain performance information of multiple test environments in parallel, may perform operation S710-2 before operation S710-1 to first obtain performance information from the second test server 620-2, or the like.

[0112] 7A , at operation S720, the task assignment system may be configured to obtain one or more policy information. For example, the task allocator may obtain (e.g., from one or more storage media, from a user terminal / device communicatively connected to the task assignment system, from an input component, etc.) policy information for a task to be performed (e.g., running a test, scheduling a test, etc.) to test software of the system.

[0113] Further, in operation S730, the task allocation system may be configured to determine at least one test environment, for example, the task allocator may determine a test environment from among a plurality of test environments that satisfies the policy information.

[0114] Referring to the example use case of FIG. 7B, upon obtaining performance information of multiple test environments provided or deployed on the first test server 620-1 and the second test server 620-2, the task allocator of the task allocation system 610 may perform a task planning process (including operations S720 and S730) to generate a test plan to execute or conduct one or more tests / simulations across the test environments (e.g., in parallel, simultaneously, etc.).

[0115] In this regard, the task allocator may generate a test plan based on the performance information and the policy information. For example, the task allocator may place tasks according to the performance information of each test environment and the placement policy of the tasks. Thus, the task allocator may determine an optimal test environment that meets the requirements (defined in the policy information) of each task. For example, as described above, if the policy information for a test defines that the test must be completed within a certain time range, then based on the performance information of each test environment, it may determine which test environment can meet the time constraint and place the tasks in the test environment accordingly.

[0116] To this end, it will be appreciated that one or more of the operations described above with reference to Figure 5 may be applicable or performed by the task allocator in generating the test plan. For example, the performance information utilized by the task allocator in generating the test plan may include performance information obtained directly from the associated test servers and / or may include performance information stored in records described above with reference to Figure 5, and the task allocator may determine one or more optimal test environments in a manner similar to that described with respect to the operation of processor 520 in Figure 5, or perform similar operations.

[0117] 7A, at operation S740, the task assignment system may be configured to assign a task. For example, a task allocator of the task assignment system may generate a signal, message, instruction, or the like, including information of the task (e.g., a schedule for performing the test, a target time period for completing the test, etc.) based on the task's policy information. The task allocator may then provide the generated signal / message / instruction to a test server that provides or deploys the determined test environment.

[0118] Referring to the use case example of FIG. 7B, once the task allocator of the task allocation system 610 completes the task plan, it may first assign or place the tasks to a test environment provided or located on a first test server 620-1 (in operation S740-1), and then assign or place the tasks to a test environment provided or located on a second test server 620-2 (in operation S740-2).

[0119] It will be appreciated that the tasks assigned or placed in acts S740-1 and S740-2 may be the same, different, or partially coincident. For example, the task assigned or placed in act S740-2 may be a task in the same family as the task assigned or placed in act S740-1.

[0120] As another example, the task may include multiple tasks, and the policy information obtained by the task allocator may include policy information for the multiple tasks. In this regard, the task allocator may determine multiple test environments among the multiple test environments to which to allocate or place the multiple tasks, respectively, based on the associated policy information. Thus, the task allocator may allocate the multiple tasks for concurrent execution in the multiple test environments.

[0121] It will further be understood that the task allocator may assign or place tasks in any suitable order, for example, the task allocator may perform operations S740-1 and S740-2 simultaneously to assign or place tasks to multiple test environments in parallel, may perform operation S740-2 before operation S740-1 to first assign or place tasks to the test environment of the second test server 620-2, or the like.

[0122]

[0046] Referring now to Figure 8A, Figure 8A illustrates a block diagram of an example method 800 of managing information, according to one or more embodiments. Similar to method 700 of Figure 7A, one or more operations of method 800 may be performed by at least one task allocator of task assignment system 610 of Figure 6. Further, the test environment associated with one or more operations of method 800 may be located or provided in first test server 620-1 and second test server 620-2 of Figure 6. Additionally, it will be appreciated that one or more operations of method 800 may be performed in combination with one or more operations of method 700 of Figure 7A.

[0123] Additionally, Figure 8B illustrates a call flow of a non-exhaustive example use case in which method 800 may be implemented, according to one or more embodiments. Accordingly, below, one or more operations of method 800 of Figure 8A may be described with reference to Figure 8B.

[0124] As shown in FIG 8A, at act S810, the task assignment system may receive information. For example, a task allocator of the task assignment system may receive performance information of a test environment provided or hosted on one or more test servers from one or more test servers. In this regard, act S810 may be part of act S710 of FIG 7A. Alternatively, or in addition, the task allocator may receive one or more test results of the assigned tasks from one or more test servers (or test environments hosted or hosted therein).

[0125] 8B, in acts S810-1 and S810-2, a task allocator of the task allocation system 610 may receive performance information of one or more test environments from the first test server 620-1 and from the second test server 620-2, respectively. Additionally, in acts S810-3 and S810-4, the task allocator may receive one or more test results of one or more assigned tasks from the first test server 620-1 and from the second test server 620-2, respectively.

[0126] It will also be appreciated that the task allocator may receive information in any suitable order, for example, the task allocator may receive performance information from both the first test server 620-1 and the second test server 620-2 simultaneously, may receive test results from the second test server 620-2 first before receiving other information, etc., or the like.

[0127] 8A , at operation S820, the task assignment system may be configured to manage the received information. For example, a task allocator of the task assignment system may generate a record and store the received information in the record. According to an embodiment in which the record is created, the task allocator may update the information in the record based on the received information.

[0128] According to an embodiment, the record may include a table, and the task allocator may store performance information for each test environment in the table of the record. In this regard, upon receiving performance information from the test server, the task allocator may simply replace the performance information in the table with the newly received performance information, or may determine a difference between the performance information in the table and the newly received performance information and update the performance information in the table that differs from the newly received performance information accordingly. The task allocator may update the performance information in the table based on the received test results of one or more tests in a manner similar to that described above.

[0129] 8B, in operations S820-1 and S820-2, the task allocator of the task allocation system 610 may update records (e.g., update performance information stored in a table of records) based on performance information of one or more test environments received from the first test server 620-1 and the second test server 620-2, respectively. Additionally, in operations S820-3 and S820-4, the task allocator of the task allocation system 610 may update records based on one or more test results of one or more assigned tasks received from the first test server 620-1 and the second test server 620-2, respectively. It will also be appreciated that the task allocator may update records in any suitable order.

[0130] 7A-8B are merely examples of operations between the task allocation system 610 and the test servers (i.e., the first test server 620-1 and the second test server 620-2). It will be understood that any other suitable operations may be performed without departing from the scope of this disclosure. For example, as described above with reference to FIG. 5, a task allocator (i.e., processor 520) of the task allocation system may assign or place one or more tasks based on the test results, optimize the assignment or placement of one or more tasks based on the test results, or perform the same type of operations.

[0131] It is understood that the particular order or hierarchy of the process / flowchart blocks disclosed herein is an illustration of an example approach. Based on design preferences, it is understood that the particular order or hierarchy of the process / flowchart blocks may be rearranged. Additionally, some blocks may be combined or omitted. The accompanying method claims present elements of the various blocks in a sample order, and are not intended to be limited to the particular order or hierarchy presented.

[0132] Some embodiments may relate to systems, methods and / or computer-readable media at any possible level of technical detail of integration. Additionally, as noted above, one or more of the above-mentioned components described above may be implemented as instructions stored on a computer-readable medium and executable by at least one processor (and / or may include at least one processor). A computer-readable medium may include a computer-readable non-transitory storage medium (or medium) having computer-readable program instructions for causing a processor to perform operations.

[0133] A computer readable storage medium may be a tangible device capable of holding and storing instructions for use by an instruction execution device. A computer readable storage medium may be, for example, but not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof. A non-exhaustive list of more specific examples of computer readable storage media includes: portable computer diskettes, hard disks, random access memories (RAM), read-only memories (ROM), erasable programmable read-only memories (EPROM or flash memory), static random access memories (SRAM), portable compact disks-read-only memories (CD-ROM), digital versatile disks (DVD), memory sticks, floppy disks, mechanically encoded devices such as punch cards or grooved ridge structures having recorded instructions thereon, and any suitable combination thereof. As used herein, computer-readable storage media should not be interpreted as signals that are transitory in nature, such as electric waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through a waveguide or other transmission medium (e.g., light pulses passing through a fiber optic cable), or electrical signals transmitted over electrical wires.

[0134] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device or to an external computer or storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network may include copper transmission cables, optical transmission fiber, wireless communications, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface of each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in a computer-readable storage medium within the respective computing / processing device.

[0135] The computer readable program code / instructions for performing the operations may be assembler instructions, instruction set architecture (ISA) instructions, machine instructions, machine dependent instructions, microcode, firmware instructions, state setting data, configuration data for an integrated circuit, or may be source code or object code written in any combination of one or more programming languages, including object oriented programming languages ​​such as Smalltalk, C++ or the like, and procedural programming languages ​​such as the "C" programming language or similar programming languages. The computer readable program instructions may run entirely on the user's computer, partially on the user's computer, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server, as a standalone software package. In the latter scenario, the remote computer may be connected to the user's computer via any type of network, such as a local area network (LAN) or wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet Service Provider). In some embodiments, an electronic circuit including, for example, a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), may execute computer readable program instructions to perform aspects or operations using state information of the computer readable program instructions to personalize the electronic circuit.

[0136] These computer-readable program instructions may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions operating on the processor of the computer or other programmable data processing apparatus produce means for performing the functions / operations specified in the blocks or blocks of the flowcharts and / or block diagrams. These computer-readable program instructions may be stored in a computer-readable storage medium that can direct a computer, programmable data processing apparatus, and / or other apparatus to function in a particular manner, such that the computer-readable storage medium having instructions stored therein includes an article of manufacture including instructions implementing aspects of the functions / operations specified in the blocks or blocks of the flowcharts and / or block diagrams.

[0137] Computer-readable program instructions may be loaded into a computer, other programmable data processing apparatus, or other device and cause the computer, other programmable apparatus, or other device to perform a series of operational steps to produce a computer-implemented process, such that the instructions operating on the computer, other programmable apparatus, or other device perform the functions / operations specified in the blocks or sections of the flowcharts and / or block diagrams.

[0138] The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of the systems, methods, and computer-readable media according to various embodiments. In this regard, each block in the flowcharts or block diagrams may represent a module, segment, or part of an instruction, including one or more executable instructions for implementing the specified logical functions. The methods, computer systems, and computer-readable media may include additional, fewer, different, or differently arranged blocks than those depicted in the figures. In some alternative implementations, the functions shown in the blocks may occur out of the order depicted in the figures. For example, two blocks shown in succession may in fact be executed in parallel or substantially in parallel, or the blocks may sometimes be executed in reverse order, depending on the functionality involved. It should also be noted that each block of the block diagrams and / or flowchart diagrams, and combinations of blocks in the block diagrams and / or flowchart diagrams, may be implemented by a special-purpose hardware-based system that executes the specified functions or operations or executes a combination of special-purpose hardware and computer instructions.

[0139] It will be apparent that the systems and / or methods described herein may be implemented in different forms of hardware, firmware, or a combination of hardware and software. The actual specialized control hardware or software code used to implement these systems and / or methods is not intended to limit implementation. Thus, although the operation and behavior of the systems and / or methods are described herein without reference to specific software code, it will be understood that software and hardware may be designed to implement the systems and / or methods based on the description herein.

Claims

1. 1. A method for intelligently determining one or more test environments for testing software of an embedded system, comprising: acquiring performance information of a plurality of test environments by a task allocator; obtaining, by the task allocator, policy information for tasks to be executed to test software of the embedded system; determining, by the task allocator, a test environment that satisfies the policy information from among the plurality of test environments; assigning, by the task allocator, the tasks to the determined test environments; receiving, by the task allocator, test results of the assigned tasks from the determined test environment; updating, by the task allocator, performance information of the determined test environment based on the received test results; Including, the embedded system is an on-board electronic control unit (ECU); The method, wherein the plurality of test environments includes at least one Software-in-the-Loop (SIL) test environment, at least one Hardware-in-the-Loop (HIL) test environment, and at least one Virtual ECU (V-ECU) test environment.

2. determining, by the task allocator, another test environment from among the plurality of test environments based on the received test results; assigning, by the task allocator, the task to the determined different test environment; and The method of claim 1 further comprising:

3. The method of claim 1 or 2, wherein the performance information includes at least one of processor speed, supported ECU features, supported instruction sets, and cost of use.

4. obtaining the policy information includes obtaining the policy information for a plurality of tasks; determining the test environments includes determining a plurality of test environments from the plurality of test environments for allocating the plurality of tasks, respectively, based on the policy information; The method of claim 1 or 2, wherein allocating the tasks comprises allocating the tasks for concurrent execution in the multiple test environments.

5. The method of claim 1 or 2, further comprising storing the performance information in a record by the task allocator, the record comprising a table.

6. receiving, by the task allocator, test results of the assigned tasks from the determined test environment; updating, by the task allocator, the table of records based on the received test results; The method of claim 5 further comprising:

7. 1. A system for intelligently determining one or more test environments for testing software of an embedded system, comprising: A memory for storing instructions; Task allocator and Equipped with The task allocator executes the instructions to Acquire performance information from multiple test environments, Obtaining policy information for tasks to be executed to test software of the embedded system; determining a test environment that satisfies the policy information from among the plurality of test environments; assigning the tasks to the determined test environments; receiving test results of the assigned tasks from the determined test environment; configured to update the performance information of the determined test environment based on the received test results; the embedded system is an on-board electronic control unit (ECU); The plurality of test environments includes at least one software-in-the-loop (SIL) test environment, at least one hardware-in-the-loop (HIL) test environment, and at least one virtual ECU (V-ECU) test environment.

8. The task allocator, determining another test environment from among the plurality of test environments based on the received test results; The system of claim 7 , further configured to assign the task to the determined different testing environment.

9. 9. The system of claim 7 or 8, wherein the performance information includes at least one of processor speed, supported ECU features, supported instruction sets, and cost of use.

10. the task allocator is configured to obtain the policy information by executing the instructions to obtain the policy information for a plurality of tasks; the task allocator is configured to execute the instructions to determine the test environments by determining a plurality of test environments from among the plurality of test environments to allocate the plurality of tasks, respectively, based on the policy information; The system of claim 7 or 8, wherein the task allocator is configured to execute the instructions to allocate the tasks by allocating the tasks for concurrent execution in the multiple test environments.

11. The system of claim 7 or 8, wherein the task allocator is further configured to execute the instructions and store the performance information in a record, the record comprising a table.

12. The task allocator executes the instructions to receiving test results of the assigned tasks from the determined test environment; The system of claim 11 , further configured to update the table of records based on the received test results.

13. A non-transitory computer-readable recording medium having instructions executable by a task allocator, comprising: The instructions cause the task allocator to perform a method for intelligently determining one or more test environments for testing software of an embedded system; The method comprises: obtaining performance information of a plurality of test environments by the task allocator; obtaining, by the task allocator, policy information for tasks to be executed to test software of the embedded system; determining, by the task allocator, a test environment that satisfies the policy information from among the plurality of test environments; assigning, by the task allocator, the tasks to the determined test environments; receiving, by the task allocator, test results of the assigned tasks from the determined test environment; updating, by the task allocator, performance information of the determined test environment based on the received test results; Including, the embedded system is an on-board electronic control unit (ECU); The plurality of test environments include at least one software-in-the-loop (SIL) test environment, at least one hardware-in-the-loop (HIL) test environment, and at least one virtual ECU (V-ECU) test environment.

14. The method comprises: determining, by the task allocator, another test environment from among the plurality of test environments based on the received test results; assigning, by the task allocator, the task to the determined different test environment; and 14. The non-transitory computer-readable storage medium of claim 13, further comprising:

15. The non-transitory computer-readable storage medium of claim 13 or 14, wherein the performance information includes at least one of processor speed, supported ECU features, supported instruction sets, and cost of use.

16. obtaining the policy information includes obtaining the policy information for a plurality of tasks; determining the test environments includes determining a plurality of test environments from the plurality of test environments for allocating the plurality of tasks, respectively, based on the policy information; The method of claim 13 or 14, wherein allocating the tasks comprises allocating the tasks for concurrent execution in the multiple test environments.

17. The method comprises: The non-transitory computer-readable storage medium of claim 13 or 14, further comprising storing, by the task allocator, the performance information in a record, the record comprising a table.

Images