System and method for managing test for testing embedded system of vehicle

Abstract

To provide a system, method, and device for managing a test for testing software of an embedded system.SOLUTION: A method may be implemented by at least one processor and may include the steps of: receiving, from a user, at least one first user input associated with one or more test requirements; generating, based on the first user input, at least one test package; generating, based on the at least one test package, at least one test environment configuration file; validating the at least one test package and the at least one test environment configuration file; determining a change in the software; and based on determining the change in the software, executing a test according to the at least one validated test package and the at least one validated test environment configuration file. The software of the embedded system may include an on-vehicle electronic control unit (ECU).SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] Systems and methods consistent with exemplary embodiments of the present disclosure relate to test management, and more particularly, to managing the testing of software related to an embedded system of a vehicle.

Background Art

[0002] Testing of software is necessary to ensure that the software functions as intended, meets the specified requirements, and reliably functions in various scenarios. Software testing is an important part of the software development life cycle (SDLC) and is performed to identify software defects, errors, or bugs before the software is deployed to an actual system.

[0003] When the software includes complex functions and / or is required to interoperate with another software, the testing of the software becomes complex and may involve multiple procedures and users / stakeholders. For example, in the development of vehicle-related functions in vehicle systems such as lane change assistance and mobile smart keys, multiple electronic control units (ECUs) are developed to perform the intended functions and can interoperate with each other.

[0004] In this regard, each of the ECUs may be managed by different users and / or may be located at geographically different locations. For example, the first ECU may be developed by a first developer located at a first location (e.g., an in-house development engineer of an automobile manufacturer), and the first ECU may need to interoperate with a second ECU developed by a second developer located at a second location (e.g., a vendor). As another example, the first ECU may be tested by a third user located at a third location (e.g., a test engineer). Further, the first ECU may interoperate with hardware located at a fourth location (e.g., a physical ECU, etc.), and thus, testing of the first ECU requires the involvement of the hardware.

[0005] Therefore, in the prior art, when a user wants to test software (e.g., a software-based ECU), the user needs to visit a physical test facility where the software and related test components (e.g., software components such as software-based ECUs developed by other users, hardware components such as physical ECUs, etc.) are collectively deployed. Thus, testing the software may require the user to physically visit the test facility to set up and execute the test, which can be time-consuming and burdensome for the user.

[0006] Also, in the prior art, it may be difficult for a user to set up, edit, and adjust tests at a test facility. This is especially the case when it is necessary to test new functions or software that has not been tested in the past with complex test cases, as the test components available at the test facility are general-purpose and limited, and the related test packages are restricted and may not be able to meet the intended test requirements or test cases.

[0007] Furthermore, since test settings can be complex and time-consuming, tests are often executed when the setup procedure is completed without verifying the accuracy of the test settings. This can lead to inaccurate test results, especially when the user setting up the test is inexperienced and prone to making mistakes when setting up the test. Furthermore, even if an error in the test settings is found during the test, the user may not be able to quickly correct the error on the spot.

[0008] Also, in the prior art, a user may not be able to access information on test components available at a test facility before visiting the test facility. For example, a vendor may not be able to obtain information on available test packages and / or related restrictions / constraints at a test facility managed by an automobile manufacturer, and thus it is extremely difficult for the vendor to accurately construct test requirements or a test plan without physically visiting the test facility.

[0009] Furthermore, in the prior art, most (if not all) of the end-to-end process of testing, such as defining test requirements, determining appropriate test packages based on the test requirements, selecting and setting up an appropriate test environment, collecting related test components, deploying the software to be tested and related test components to the test environment, executing tests on the software based on the determined test packages, obtaining test results, and reproducing test results, are executed separately, manually managed on different users and / or different systems, inefficient, burdensome, and may introduce human errors.

[0010] In view of the above, conventional software development is time-consuming, burdensome to users, and may be delayed due to the uncertainty of test management. SUMMARY OF THE INVENTION

[0011] According to an embodiment, a method and a system are provided for efficiently and effectively managing one or more tests for testing one or more softwares of an in-vehicle system of a vehicle. For example, an exemplary embodiment of the present disclosure provides a method and a system for processing an end-to-end process of test management, such as collecting user input, providing a test package, providing a test environment configuration file, verifying the test package and / or the test environment configuration file, and executing one or more tests.

[0012] According to an embodiment, a method is provided for managing tests for testing software of an embedded system. The method can be implemented on at least one processor and includes receiving, from a user, at least one first user input related to one or more test requirements; generating at least one test package based on the first user input; generating at least one test environment configuration file based on the at least one test package; verifying the at least one test package and the at least one test environment configuration file; determining a change in the software; and executing a test according to the verified at least one test package and the verified at least one test environment configuration file based on determining the change in the software. The software of the embedded system can include an in-vehicle electronic control unit (ECU).

[0013] According to an embodiment, the method can further include publishing at least one verified test package and at least one verified test environment configuration file by storing them in one or more storage media accessible by other users. The method can also include collecting one or more test results related to the test, generating at least one graphical user interface (GUI) including the one or more test results, and presenting the at least one GUI to the user.

[0014] According to an embodiment, the first user input can include information related to at least one user-defined test case, and generating at least one test package can include generating at least one test scenario template based on the at least one user-defined test case, obtaining one or more test package artifacts based on the at least one test scenario template, and combining the at least one test scenario template with the one or more test package artifacts to generate at least one test package.

[0015] According to an embodiment, generating at least one test environment configuration file includes receiving from a user at least one second user input related to at least one user-defined test plan, generating at least one test plan template based on the second user input and at least one test package, generating at least one test plan by adding at least one test plan template to at least one user-defined test case, receiving from the user at least one third user input related to at least one user-defined test cycle, and generating at least one test environment configuration file based on the third user input and at least one test plan. The at least one test environment configuration file can include information defining settings of at least one test bench.

[0016] According to an embodiment, verifying at least one test package and at least one test environment configuration file includes performing a pre-test of software based on the at least one test package and the at least one test environment configuration file, presenting the results of the pre-test to the user, receiving from the user at least one fourth user input related to one of approval of the results of the pre-test and rejection of the results of the pre-test, determining that the at least one test package and the at least one test environment configuration file are valid based on determining that the fourth user input is associated with approval, and determining that the at least one test package and the at least one test environment configuration file are invalid based on determining that the fourth user input is associated with rejection.

[0017] According to an embodiment, determining a software change can include obtaining the current status of the software and comparing the current status with the last known status of the software to determine whether a change has occurred in the software. The software change can include a breaking change that satisfies one or more conditions defined in at least one test package.

[0018] According to an embodiment, executing a test can include generating at least one test bench related to at least one test environment for testing software based on at least one verified test environment configuration file, selecting at least one node related to at least one test environment defined in at least one test bench based on at least one test bench, deploying the software to the selected at least one node, and performing a test for testing the software at the selected at least one node based on at least one of the verified test packages.

[0019] According to an embodiment, a system for managing tests for testing software of an embedded system is provided. The system includes a storage device for storing instructions, and at least one processor configured to execute instructions for receiving from a user at least one first user input related to one or more test requirements, generating at least one test package based on the first user input, generating at least one test environment configuration file based on at least one test package, verifying at least one test package and at least one test environment configuration file, determining a software change, and executing a test according to at least one verified test package and at least one verified test environment configuration file based on the determination of the software change. The software of the embedded system can include an in-vehicle electronic control unit (ECU).

[0020] According to an embodiment, at least one processor may be further configured to execute instructions for publishing at least one verified test package and at least one verified test environment configuration file by storing the at least one verified test package and the at least one verified test environment configuration file in one or more storage media accessible by other users. Further, at least one processor may be further configured to execute instructions for collecting one or more test results related to a test, generating at least one graphical user interface (GUI) including the one or more test results, and presenting the at least one GUI to a user.

[0021] According to an embodiment, a first user input includes information related to at least one user-defined test case, and at least one processor may be further configured to execute instructions for generating at least one test scenario template based on the at least one user-defined test case, obtaining one or more test package artifacts based on the at least one test scenario template, and combining the at least one test scenario template with the one or more test package artifacts to generate at least one test package.

[0022] According to an embodiment, at least one processor receives from a user at least one second user input related to at least one user-defined test plan, generates at least one test plan template based on the second user input and at least one test package, generates at least one test plan by adding at least one test plan template to at least one user-defined test case, receives from the user at least one third user input related to at least one user-defined test cycle, and generates at least one test environment configuration file based on the third user input and at least one test plan, thereby being further configured to execute instructions for generating at least one test environment configuration file. The at least one test environment configuration file can include information defining settings of at least one test bench.

[0023] According to an embodiment, at least one processor performs a pre-test of software based on at least one test package and at least one test environment configuration file, presents the pre-test result to the user, receives from the user at least one fourth user input related to one of approval of the pre-test result and rejection of the pre-test result, determines that the at least one test package and the at least one test environment configuration file are valid based on a determination that the fourth user input is associated with approval, and determines that the at least one test package and the at least one test environment configuration file are invalid based on a determination that the fourth user input is associated with rejection, thereby being further configured to execute instructions for verifying at least one test package and at least one test environment configuration file.

[0024] According to an embodiment, at least one processor may be further configured to execute instructions for determining a change in software by obtaining a current status of the software and comparing the current status to a last known status of the software to determine whether a change has occurred in the software. The change in the software can include a breaking change that satisfies one or more conditions defined in at least one test package.

[0025] According to an embodiment, at least one processor may be further configured to execute instructions for performing a test by generating at least one test bench associated with at least one test environment for testing software based on at least one verified test environment configuration file, selecting at least one node associated with the at least one test environment defined in the at least one test bench based on the at least one test bench, deploying the selected at least one node software, and performing a test for testing the selected at least one node software based on at least one verified said stop package.

[0026] Additional aspects are described in part in the following description, are partially apparent from the description, or can be realized by practice of the presented embodiments of the disclosure.

Brief Description of the Drawings

[0027] The features, advantages, and significance of exemplary embodiments of the present disclosure are described below with reference to the accompanying drawings in which like reference numerals represent like elements.

[0028]

Figure 1

[0029]

Figure 2

[0030]

Figure 3A

[0031]

Figure 3B

[0032]

Figure 4

[0033] The following detailed description of exemplary embodiments refers to the accompanying drawings. The foregoing disclosure provides illustration and description, but is not intended to be exhaustive or to limit the embodiments to the exact forms disclosed. Modifications and variations are possible in light of the above disclosure, or may be acquired by practice of the embodiments. Additionally, one or more features or components of one or more embodiments can be incorporated into, or combined with, other embodiments (or one or more features of other embodiments). Further, it is understood that in the description of the operations provided below, one or more operations may be omitted, one or more operations may be added, one or more operations may be performed simultaneously (at least in part), and the order of one or more operations may be switched.

[0034] Even if a particular combination of features is recited in the claims and / or disclosed herein, that combination is not intended to limit the disclosure of possible implementations. In fact, many of these features may be combined in ways that are not specifically recited in the claims and / or disclosed herein. Each of the dependent claims listed below may depend directly on only one claim, but the disclosure of possible implementations includes each dependent claim in combination with all the other claims in the claim set.

[0035] Any element, act, or instruction used herein should not be construed as decisive or essential unless explicitly described as such. Also, as used herein, the articles "a" and "an" are intended to include one or more items and can be used interchangeably with "one or more." When only one item is intended, the term "one" or similar language is used. Also, as used herein, terms such as "has," "having," "includes," "including," etc. are intended to be open-ended terms that are not limiting. Further, the phrase "based on" is intended to mean "at least in part based on" unless explicitly described otherwise. Additionally, expressions such as "at least one of [A] and [B]" or "at least one of [A] or [B]" should be understood to include only A, only B, or both A and B.

[0036] References throughout this specification to "one embodiment," "an embodiment," "a non-limiting preferred embodiment," or similar terms mean that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment of the solution. Thus, the phrases "in one embodiment," "in an embodiment," "in a non-limiting preferred one embodiment," and similar terms throughout this specification may all refer to the same embodiment, but not necessarily so.

[0037] Furthermore, the features, advantages, and characteristics of the present disclosure described may be combined in any suitable manner in one or more embodiments. Those skilled in the art will recognize, in view of the description herein, that the present disclosure may be practiced without using one or more of the specific features or advantages of a particular embodiment. In other instances, additional features and advantages may be recognized in certain embodiments that may not be present in all embodiments of the present disclosure.

[0038] In addition, terms such as "vehicle" as used herein may refer to any motorized and / or mechanical machine capable of transporting or carrying people and / or cargo, such as automobiles, trucks, motorcycles, buses, bicycles, mobility scooters, etc.

[0039] Exemplary embodiments consistent with the present disclosure provide a method, system, and apparatus for managing tests for one or more software of an embedded system, such as an in-vehicle ECU. Specifically, the methods, systems, apparatuses, etc. of the exemplary embodiments can automatically manage tests according to user-defined requirements. According to an embodiment, the methods, systems, apparatuses, etc. of the exemplary embodiments can automatically generate one or more test packages and one or more test environment configuration files based on one or more user inputs, and execute tests accordingly.

[0040] In some embodiments, one or more of the generated test packages and / or one or more of the generated test environment configuration files may be verified before being utilized to execute tests. When one or more of the generated test packages and / or one or more of the generated test environment configuration files are verified, the methods, systems, apparatuses, etc. of the exemplary embodiments can automatically determine software changes based thereon and automatically execute tests based on the determination of software changes. Further, the verified test packages and / or verified test environment configuration files may be published and accessible by other users.

[0041] According to an embodiment, when a test is executed on software, the method, system, device, etc. of the exemplary embodiment can automatically collect test results and present them to the relevant user. Further, the method, system, device, etc. of the exemplary embodiment can receive one or more user inputs for updating one or more generated test packages and / or one or more generated test environment configuration files, and can automatically update them accordingly.

[0042] Thus, the method, system, device, etc. of the exemplary embodiment can automatically generate or update one or more test components according to user input based on real-time or near-real-time status and test requirements. As a result, one or more test benches can be provided and utilized on demand without geographical restrictions.

[0043] For this purpose, the exemplary embodiments of the present disclosure can automatically manage the end-to-end process of testing according to one or more user-defined requirements. The user can remotely define one or more test requirements without physically moving to a test facility as required in the prior art. Ultimately, the exemplary embodiments of the present disclosure make it possible to perform software development more efficiently, greatly reduce the burden on the user, significantly reduce development time, and greatly reduce the cost and effort for planning visits to or business trips to test facilities.

[0044] It is contemplated that the features, advantages, and significance of the above exemplary embodiments are only a part of the present disclosure, and are not intended to be comprehensive or to limit the technical scope of the present disclosure. Further descriptions of the features, components, configurations, operations, and implementations of the exemplary embodiments of the present disclosure are provided below, along with their accompanying technical advantages and significance.

[0045] FIG. 1 is a block diagram of an exemplary system configuration 100 for managing tests according to one or more embodiments. As shown in FIG. 1, the system configuration 100 may include a test automation system 110, a plurality of nodes 120-1 to 120-N, and a network 130.

[0046] Generally, the test automation system 110 may be communicatively coupled to the plurality of nodes 120-1 to 120-N via the network 130 and configured to interoperate with the plurality of nodes 120-1 to 120-N to manage tests (or one or more related information or data). Descriptions of exemplary components that may be included in the test automation system 110, as well as descriptions of related use cases, are provided below with reference to FIGS. 2 through 4.

[0047] Each of the plurality of nodes 120-1 to 120-N may include one or more devices, apparatuses, systems, or any other suitable components that may receive, host, store, utilize, deploy, process, provide, etc., one or more artifacts or components that make up a test.

[0048] For example, node 120-1 can include a device or apparatus (such as a personal computer, a server or server cluster, a workstation, etc.) that can be used for constructing, storing, executing, or simulating one or more computer-executable software applications, such as one or more virtual ECUs of a vehicle system, one or more emulated ECUs, and / or any other suitable software-based components (such as a vehicle model, a data communication module (DCM) model, a heating, ventilation, and air conditioning (HVAC) model, etc.). As another example, node 120-1 can include one or more hardware components, such as one or more fully developed physical ECUs, one or more partially developed physical ECUs, one or more vehicle hardware (such as a powertrain, an engine, etc.).

[0049] According to an embodiment, one or more of the plurality of nodes 120-1 to 120-N can include one or more interfaces, each of which can be configured to communicatively couple the associated node to the test automation system 110. For example, one or more of the plurality of nodes can include a hardware interface, a software interface (such as a program interface, an application program interface (API), etc.).

[0050] According to an embodiment, at least some of the plurality of nodes 120-1 to 120-N are located at a location that is geographically different from the test automation system 110 and / or different from another part of the plurality of nodes. According to an embodiment, at least some of the plurality of nodes 120-1 to 120-N are associated with users located at geographically different locations.

[0051] For example, node 120-1 may be associated with a first user (e.g., a development engineer) responsible for developing a first ECU, node 120-2 may be associated with a second user (e.g., a test engineer) responsible for testing the first ECU, the first user may be located at a first location, the second user may be located at a second location, and the first location may be different from the second location. Alternatively or additionally, the first node 120-1 may be associated with a first user responsible for developing a first ECU, node 120-2 may be associated with a second user responsible for developing a second ECU, the first ECU may interoperate with the second ECU, the first user may be located at a first location, the second user may be located at a second location, and the first location may be different from the second location.

[0052] According to an embodiment, at least a portion of the plurality of nodes 120-1 to 120-N may be associated with one or more test environments. For example, the portion of the nodes may have at least one software-based test environment (e.g., a software-in-the-loop (SIL) test environment, a virtual ECU (V-ECU) test environment, a model-in-the-loop (MIL) test environment, a processor-in-the-loop (PIL) test environment, etc.) and / or at least one hardware-based test environment (e.g., a hardware-in-the-loop (HIL) test environment), and may be communicatively coupled thereto (e.g., by a wired connection, a wireless connection, etc.) or deployed thereon.

[0053] Furthermore, at least a portion of the plurality of nodes 120-1 to 120-N may include one or more storage media, such as a server or a server cluster, configured to store, disclose, etc., one or more data or information (or information related thereto) provided by the test automation system 110 and / or another portion of the plurality of nodes 120-1 to 120-N.

[0054] Network 130 may include one or more wired and / or wireless networks configured to couple a plurality of nodes 120-1 to 120-N to the test automation system 110. For example, network 130 may include a cellular network (e.g., a fifth generation (5G) network, a Long-Term Evolution (LTE) network, a third generation (3G) network, a Code Division Multiple Access (CDMA) network, etc.), a Public Land Mobile Network (PLMN), a Local Area Network (LAN), a Wide Area Network (WAN), a Metropolitan Area Network (MAN), a telephone line network (e.g., a Public Switched Telephone Network (PSTN)), a private network, an ad hoc network, an intranet, the Internet, an optical fiber-based network, etc., and / or a combination of these or other types of networks.

[0055] According to an embodiment, network 130 may include a virtual network that includes one or more physical network components (e.g., Ethernet (registered trademark), a WiFi module, telecommunications network hardware, etc.) on which one or more virtual network functions (e.g., a Controller Area Network (CAN) bus, etc.) are implemented. Additionally or alternatively, network 130 may include at least one parameter network.

[0056] Next, refer to FIG. 2, which shows a block diagram of exemplary modules of the test automation system 200 according to one or more embodiments. The test bench management system 200 may correspond to the test automation system 110 described above with reference to FIG. 1, and thus, the features described herein with reference to systems 110 and 200 may be applicable to each other unless explicitly stated otherwise.

[0057] As shown in FIG. 2, the test automation system 200 can include at least one user interaction module 210, at least one testing framework module 220, at least one configuration management module 230, at least one test bench management module 240, and at least one test management module 250. As will be further described below, one or more of modules 210 - 250 can be defined by computer-executable instructions or programming code that, when executed by one or more processors (e.g., processor 430 described below with reference to FIG. 4), cause the one or more processors to perform associated operations. Alternatively or additionally, one or more of modules 210 - 250 can include one or more hardware components (e.g., components described below with reference to FIG. 4) that are configured to perform associated operations. For this purpose, modules 210 - 250 can be configured to interact with each other to manage one or more tests, as described below.

[0058] The user interaction module 210 (which may be referred to as "module 210" in this specification) may be configured to interact with one or more users. According to an embodiment, module 210 may generate one or more graphical user interfaces (GUIs) to involve one or more users. For example, module 210 may generate one or more GUIs for a user and receive one or more user inputs from the user.

[0059] According to an embodiment, module 210 may be configured to receive from one or more users one or more user inputs that may define one or more test requirements such as a test case intended by the user, a test plan intended by the user, a test cycle intended by the user, and / or the like. As will be further described below, the one or more user inputs may be utilized in generating one or more test packages and / or one or more test environment configuration files. Further, module 210 may also be configured to receive from one or more users one or more inputs including information for updating one or more generated test components.

[0060] According to an embodiment, module 210 may generate one or more GUIs to present the status of a test to a user. For example, module 210 may present in one or more GUIs the status of a configured test (e.g., pending, executed, failed, etc.), the result of a verification test, the result of an actual test, and the like.

[0061] The test framework module 220 (which may be referred to as "module 220" in this specification) may be configured to generate one or more test packages and / or one or more test environment configuration files. According to an embodiment, module 220 may be configured to obtain one or more user inputs from module 210 and, based on the obtained one or more user inputs, may be configured to generate one or more test packages and / or one or more test environment configuration files.

[0062] For example, module 220 can generate one or more test packages based on one or more user inputs and generate one or more test environment configuration files based on the one or more test packages. In some embodiments, the one or more user inputs may include information related to one or more user-defined test cases, and module 220 may create one or more test scenario templates based on the one or more user-defined test cases, and obtain one or more test package artifacts related to the one or more test scenario templates from among a plurality of nodes related to the test automation system (e.g., nodes 120-1 to 120-N) (e.g., create, search, retrieve, etc.), and combine the one or more test scenario templates with the one or more test package artifacts to generate one or more test packages, thereby generating one or more test packages.

[0063] In this regard, the term "test case" as described herein may refer to a specific set of conditions and / or steps designed to verify the functionality or behavior of software (or the system under test). A test case can define one or more actions and the expected results for a particular test scenario. According to an embodiment, a test case may include one or more of a test case ID (e.g., a unique identifier or a series of numbers assigned to the test case for tracking and reference purposes), a test objective (e.g., a description or specification of the goal or purpose of the test case), at least one test condition (e.g., a prerequisite or initial state required to execute the test case, a specific setup or configuration required to trigger the test execution, etc.), at least one test step (e.g., an action or operation to be performed to execute the test case, required inputs or interactions with the software, etc.), at least one expected result (e.g., the result or behavior expected of the software when the test case is executed successfully), and any other appropriate information.

[0064] Furthermore, the term "test package" as used herein may refer to an aggregate of test cases, test scripts / algorithms, test data, test dependencies, test execution instructions, test cycles, and other related resources collectively organized for the purpose of testing a particular software (e.g., a particular ECU). By generating and providing a "test package" based on user-defined test requirements (included in the first user input), module 220 can provide a structured approach for planning and executing tests in the way the user intends, while ensuring that all necessary test activities are included and appropriately documented.

[0065] Furthermore, as used herein, the term "test environment configuration file" may refer to a file or document containing information for obtaining (e.g., acquiring, creating, updating, etc.) and configuring at least one test environment associated with the software or system being tested. For example, a test environment configuration file may include information related to the configuration of one or more test environments required to test the software (e.g., the configuration of a software-based test environment required to test the software and related software-based components, the configuration of a hardware-based test environment required to test the hardware related to the software, etc.), information related to the test plan, information related to the test scenario, information related to the test execution conditions, and information related to the test bench configuration.

[0066] As used herein, the term "test scenario" may refer to a set or array of related test cases grouped to achieve a common test objective. Each test case within a test scenario focuses on a specific aspect or condition, and the test scenario provides a larger context or flow.

[0067] As used herein, the term "test plan" may refer to a set of information indicating an approach, objective, scope, procedure, and an overview of the configuration for testing software. A test plan can include a set or array of test scenarios, test execution procedures, and setting information such as test bench configuration and ECU test configuration.

[0068] As used herein, the term "test bench" may refer to information or parameters such as a set of tools, procedures, functional compositions, equipment, etc. that enable a test to be executed under desired conditions or configurations when compiled or utilized. Simply put, a test bench defines a test environment for testing software.

[0069] Additionally or alternatively, one or more user inputs may include information related to one or more user-defined test plans and information related to one or more user-defined test cycles, and module 220 may generate one or more test plan templates based on the one or more user-defined test plans and the one or more test packages, and generate one or more test plans by adding one or more user-defined test cases to the one or more test plan templates, and generate one or more test environment configuration files based on the one or more user-defined test cycles and the one or more test plans, thereby generating one or more test environment configuration files.

[0070] Additionally or alternatively, module 220 may update one or more generated test packages based on one or more user inputs. In some implementations, the one or more user inputs may include information related to one or more user-defined updated test cases, and module 220 may generate one or more updated test scenario templates based on the one or more user-defined updated test cases, and obtain (e.g., create, search, retrieve, etc.) one or more test package artifacts related to the one or more updated test scenario templates from among a plurality of nodes related to the test automation system (e.g., nodes 120-1 to 120-N), and update the one or more generated test packages by combining the one or more updated test package artifacts with the one or more test package artifacts to generate one or more updated test packages. According to some embodiments, module 220 may determine one or more differences between the one or more user-defined updated test cases and the current user-defined test cases, and generate one or more test scenario templates by modifying the one or more generated test scenario templates to reflect the one or more differences.

[0071] According to an embodiment, the module 220 can interact with the configuration management module 230, such as storing or committing one or more generated data (e.g., test packages, test environment configuration files, updated test packages, etc.) to the configuration management module 230, receiving a trigger for executing a test from the configuration management module 230, and providing one or more test results, test logs, test traces, etc. (received from the test management module 250) to the configuration management module 230.

[0072] According to an embodiment, the module 220 can interact with the test bench management module 240, such as providing information or data to the test bench management module 240 to initialize test bench provision. According to an embodiment, the module 220 can interact with the test management module 250, such as providing one or more generated / verified data (e.g., test packages, etc.) to the test management module 250 and receiving one or more test results, test logs, test traces, etc. from the test management module 250.

[0073] At least one configuration management module 230 (sometimes referred to herein as "module 230") can be configured to manage information or configurations related to tests. For example, the module 230 can receive one or more generated data (e.g., test packages, test environment configuration files, updated test packages, etc.) from the module 220 and then compile and store the one or more generated data therein. According to an embodiment, the module 230 can include one or more storage media (e.g., the storage 420 described below with reference to FIG. 4, a hosting server, a server cluster, etc.).

[0074] Furthermore, module 230 may also be configured to receive from one or more nodes associated with the test automation system 200 (e.g., nodes 120-1 to 120-N, etc.), one or more systems to be tested (e.g., software to be tested such as a software-based ECU), and one or more components associated therewith (e.g., one or more software-based ECUs, one or more vehicle-related models, information on hardware associated with the system to be tested, etc., one or more software that interacts with the system to be tested), and store it therein.

[0075] According to an embodiment, module 230 may be configured to continuously (or periodically) determine changes in the system under test. For example, module 230 may obtain the current status of the system under test based on a test cycle defined in at least one test package of the system under test, and compare the current status with the last known status of the system under test to determine whether a change has occurred in the system under test. According to an embodiment, the change in the system under test includes a destructive change that satisfies one or more conditions defined in at least one test package. Therefore, when determining a change in the system under test, module 230 can generate a message (e.g., a notification message, etc.) for triggering test execution and send it to module 220.

[0076] The test bench management module 240 (which may be referred to as "module 240" in this specification) may be configured to provide one or more test benches for testing. For example, each time a test plan execution is triggered, module 240 can receive information or data such as a test environment configuration file (including test bench setting information) from module 220, and can obtain (for example, retrieve, create, update, etc.) one or more test benches based on the received information. According to an embodiment, module 240 can provide one or more test benches to the test management module 250.

[0077] The test management module 250 (which may be referred to as "module 250" in this specification) may be configured to execute one or more tests. For example, module 250 can receive one or more test benches from module 240, and can initialize one or more test environments (such as a software-based environment and / or a hardware-based environment, etc.) according to the one or more test benches. Further, module 250 can receive one or more test packages from module 220, and can execute one or more tests based on the one or more initialized test environments according to the one or more test packages.

[0078] For this purpose, the modules within the test automation system 200 can be configured to interact with each other to facilitate end-to-end automation when managing a user's tests. The user can simply provide the test automation system 200 with the intended test requirements (e.g., intended test cases, intended test plans, intended test cycles, etc.), and the test automation system 200 can automatically generate the required configuration data (e.g., test packages, test environment configuration files, test benches, etc.) based on the user-defined test requirements, automatically monitor the state of the system under test (e.g., software), automatically trigger the execution of the test, automatically collect the test results, and present them to the user.

[0079] Hereinafter, exemplary use cases of the modules within the test automation system 200 according to one or more embodiments will be described with reference to FIGS. 3A and 3B. One or more of the modules 210-250 shown in FIGS. 3A and 3B are similar to the modules 210-25 described above herein with reference to FIG. 2.

[0080] First, refer to FIG. 3A, which shows a flowchart of an exemplary use case for test preparation according to one or more embodiments. As shown in FIG. 3A, in operation S3101, the module 210 can be configured to receive at least one first user input from the user (via a node associated with the user). The first user input can be associated with one or more test requirements and can include information associated with at least one user-defined test case.

[0081] In operation S3102, module 210 can provide the first user input (received in operation S3101) to module 220. Thus, in operation S3103, module 220 can be configured to generate at least one test package based on the first user input. For example, module 220 can generate at least one test scenario template based on at least one user-defined test case (included in the first user input), and can be configured to obtain one or more test package artifacts based on the at least one test scenario template from a plurality of nodes (such as nodes 120-1 to 120-N, etc.) associated with the test automation system 200. Thus, module 220 can combine the at least one test scenario template with one or more test package artifacts to generate at least one test package.

[0082] Referring further to FIG. 3A, upon generating at least one test package, in operation S3104, module 220 can provide the generated test package to module 230 for storage. Subsequently, in operation S3105, module 210 can be configured to receive at least one second user input from the user. The at least one second user input can be associated with at least one user-defined test plan.

[0083] Therefore, in operation S3106, module 210 can provide the second user input to module 220. Next, in operation S3107, module 220 can generate at least one test plan based on the second user input. For example, module 220 can generate at least one test plan template based on the second user input and at least one test package (generated in operation S3103), and can generate at least one test plan by adding at least one user-defined test case to the at least one test plan template.

[0084] Furthermore, in operation S3108, module 210 can be configured to receive at least one third user input from the user. The at least one third user input can be associated with at least one user-defined test cycle that defines how often the system (e.g., module 230) should determine whether to execute the test. Therefore, in operation S3109, module 210 can provide the third user input to module 220. Next, in operation S3110, module 220 can be configured to generate at least one test environment configuration file. The test environment configuration file can include the configuration of one or more test environments required to test the system under test, information related to the test plan, information related to the test scenario, information related to the test execution conditions, and / or information related to the test bench configuration.

[0085] In operation S3111, upon generating the test environment configuration file, module 220 can provide the same to module 230 for storage. In some implementations, before providing the test package and / or the test environment configuration file to module 230, module 220 can verify the test package and / or the test environment configuration file to ensure that the associated content is accurate and meets the user's requirements.

[0086] According to an embodiment, the module 220 can verify a test package and / or a test environment configuration file by executing a pre-test on a system under test based on the test package and / or the test environment configuration file, presenting the result of the pre-test to the user, and receiving from the user at least one fourth user input associated with one of approval of the result of the pre-test and rejection of the result of the pre-test. Therefore, based on determining that the fourth user input is associated with approval, the module 220 may determine that the test and / or the test environment configuration file is valid. Otherwise, based on determining that the fourth user input is associated with rejection, the module 220 may determine that the test and / or the test environment configuration file is invalid. The pre-test may be executed on a first node that may be different from the node (e.g., the second node) on which the actual test is executed. In this way, the test automation system can ensure the validity of the test package and / or the test environment configuration file before exposing the same to the module 230.

[0087] For this purpose, when the preparation for the test is complete, the module 230 may be configured to continuously (or periodically) monitor the status of the system under test (e.g., according to a user-defined test cycle) and automatically trigger test execution when necessary. An illustrative description of a related use case will be described below with reference to FIG. 3B.

[0088] Referring to FIG. 3B, FIG. 3B shows a flowchart of an illustrative use case of test execution according to one or more embodiments. The operations in FIG. 3B may be executed after the operations described above with reference to FIG. 3A.

[0089] As shown in FIG. 3B, in operation S3201, module 230 can determine changes to the system under test (e.g., software, etc.). For example, module 230 can continuously (or periodically) obtain the current status of the system under test (from one or more nodes related to the system under test), and by comparing the current status with the last known status of the system under test (previously obtained and stored by module 230), it can determine whether the system under test has changed from the last known status. According to an embodiment, module 230 can execute operation S3201 according to a user-defined test cycle (the relevant information is included in the relevant test package).

[0090] Also, after determining the change, module 230 may determine whether the change is a destructive change. For example, module 230 can determine whether the change meets one or more conditions defined in at least one test package. Based on the determination that the change meets one or more conditions, module 230 may determine that the change is a destructive change. In this regard, the term "destructive change" may refer to a change or modification to the system under test that destroys or interrupts the normal function of the system under test.

[0091] Therefore, in operation S3202, module 230 can generate a message (e.g., a notification flag, a warning message, etc.) and send it to module 220 to inform about the determined change to the system under test. Subsequently, in operation S3203, module 220 can collect the test package and the test environment configuration file related to the system under test (from module 230, etc.) and trigger the test execution by providing the test environment configuration file to module 240.

[0092] In operation S3204, module 240 can initialize one or more test benches. Specifically, module 240 can obtain (for example, retrieve, create, configure, etc.) one or more test benches for executing tests based on the test environment configuration file. Thus, in operation S3205, module 240 can provide one or more test benches to module 250. Further, in operation S3206, module 220 can provide a test package to module 250. In this regard, it can be understood that module 220 can execute operation S3206 simultaneously with operations S3203, S3204, or S320 without departing from the technical scope of the present disclosure.

[0093] In operation S3207, module 250 can execute a test for testing the system under test. Specifically, module 250 can determine one or more nodes (for example, nodes 120-1 to 120-N) capable of providing a test environment (for example, a software-based test environment, a hardware-based test environment, etc.) according to one or more test benches, deploy the system under test to the determined one or more nodes (together with related test components), and execute a test on the one or more nodes based on the test package.

[0094] When the test is executed, in operation S3208, module 220 can collect one or more test results associated with the test from module 250. Accordingly, in operation S3209, module 220 can archive the one or more collected test results by sending the one or more test results to module 230. When storing the one or more test results, in operation S3210, module 230 can provide the one or more test results to module 210. Accordingly, in operation S3211, module 210 can generate one or more GUIs including the one or more test results and then present the one or more GUIs to the user (via the node associated with the user).

[0095] It can be understood that the operations of the above modules 210 to 250 are merely examples of possible embodiments, and the technical scope of the present disclosure should not be limited thereby. Specifically, one or more of modules 210 to 250 can be configured to operate in a manner different from that described herein without departing from the technical scope of the present disclosure.

[0096] According to an embodiment, one or more of modules 210 to 250 in the test automation system may be defined in computer-executable instructions or programming code and may be associated with one or more hardware components of the test automation system. For example, the computer-executable instructions defining one or more of modules 210 to 2504 may be stored in one or more memory storages of the test automation system and may be executable by one or more processors of the test automation system to perform one or more operations related to modules 210 to 250 as described herein.

[0097] Referring to FIG. 4, FIG. 4 shows a block diagram of exemplary components of a test automation system 400 according to one or more embodiments. The test automation system 400 may be similar to the test automation system 200 of FIG. 2, and the components included therein may be configured to perform one or more operations of one or more modules of the test automation system 200.

[0098] As shown in FIG. 4, the test automation system 400 can include at least one communication interface 410, at least one storage 420, and at least one processor 430. However, the test automation system 400 can include more or fewer components than those shown, and / or the components included therein can be arranged in any manner different from those shown without departing from the technical scope of the present disclosure. For example, in some implementations, the test automation system 400 can include multiple storages 420 and multiple processors 430, each of which may be specialized to perform one operation of modules 210 to 250.

[0099] The communication interface 410 may include a component such as a transceiver (e.g., a transceiver, a separate receiver and transmitter, etc.) that enables the test automation system 400 (or one or more components included therein) to communicate with one or more components external to the test automation system 400 via a wired connection, a wireless connection, or a combination of wired and wireless connections. For example, the communication interface 410 may couple the test automation system 400 (or one or more components included therein) to a plurality of nodes (e.g., nodes 120-1 to 120-N in FIG. 1, etc.), thereby enabling them to communicate and interact with each other. As another example, the communication interface 410 may enable components of the test automation system 400 to communicate with each other. For example, the communication interface 410 may couple the storage 420 to the processor 430, thereby enabling them to communicate and interoperate with each other.

[0100] According to an embodiment, the communication interface 410 may include a hardware-based interface such as a bus interface, an Ethernet interface, an optical interface, a coaxial interface, an infrared interface, a radio frequency (RF) interface, a universal serial bus (USB) interface, a Wi-Fi interface, a cellular network interface, a software interface, etc. According to an embodiment, the communication interface 210 may include at least one controller area network (CAN) bus configurable to communicatively couple components (e.g., storage 420, processor 430, etc.) of the test automation system 400 to a plurality of nodes (e.g., nodes 120-1 to 120-N). Additionally or alternatively, the communication interface 410 may include a software-based interface such as an application programming interface (API), a virtual network interface (e.g., a virtual CAN bus, etc.).

[0101] According to an embodiment, the communication interface 410 may be configured to receive information from one or more components external to the test automation system 400 and provide it to the processor 430 for further processing and / or to the storage 420 for storage. For example, the communication interface 410 may receive one or more user inputs that define one or more test requirements (e.g., test cases, test plans, test cycles, etc.) from multiple nodes.

[0102] At least one storage 420 may include one or more storage media suitable for storing data, information, and / or computer-readable / computer-executable instructions. According to an embodiment, the storage 420 may include random access memory (RAM), read-only memory (ROM), and / or another type of dynamic or static storage device (e.g., flash memory, magnetic memory, and / or optical memory) for storing information and / or instructions for use by the processor 430.

[0103] Additionally or alternatively, the storage 420 may include, together with a corresponding drive, a hard disk (e.g., magnetic disk, optical disk, magneto-optical disk, and / or solid-state disk), a compact disc (CD), a digital versatile disc (DVD), a floppy disk, a cartridge, a magnetic tape, and / or another type of non-transitory computer-readable medium.

[0104] According to an embodiment, the storage 420 may be configured to store computer-executable instructions or programming code that define one or more of the modules 210-250 (described above with reference to FIGS. 2-3B). Further, the storage 420 may be configured to store information utilized by the processor 430 to facilitate the automation of test management. For example, the storage 420 may store one or more test requirements defined by one or more users, store one or more systems under test and related test components or artifacts (e.g., programming code that defines software to be tested and / or software that interoperates with the software to be tested), store one or more generated test packages and generated test environment configuration files, and be configured to store one or more test results.

[0105] The at least one processor 430 may include one or more processors that may be programmed to perform the functions or operations described herein. For example, the processor 430 may be configured to perform one or more of the operations or one or more of the actions described herein by executing computer-readable instructions stored on a storage medium (e.g., storage 420, etc.).

[0106] According to an embodiment, the processor 430 can be configured to receive one or more signals (e.g., via a communication interface 410, etc.) that define one or more instructions for performing one or more operations. Further, the processor 430 can be implemented in hardware, firmware, or a combination of hardware and software. The processor 430 can include a central processing unit (CPU), a graphics processing unit (GPU), an accelerated processing unit (APU), a microprocessor, a microcontroller, a digital signal processor (DSP), a field-programmable gate array (FPGA), an application-specific integrated circuit (ASIC), and / or another type of processing or computing component.

[0107] According to an embodiment, at least one processor 430 can be configured to execute computer-executable instructions stored in at least one memory storage (e.g., storage 420), thereby performing one or more operations for managing tests for test software (which may be referred to herein as a "system under test") of an embedded system. The software of the embedded system can include an in-vehicle ECU.

[0108] According to an embodiment, at least one processor 430 is configured to receive, from a user, at least one first user input related to one or more test requirements; generate at least one test package based on the first user input; generate at least one test environment configuration file based on the at least one test package; verify the at least one test package and the at least one test environment configuration file; determine a software change; and execute a test according to the verified at least one test package and the verified at least one test environment configuration file based on the determination of the software change.

[0109] The software is associated with (e.g., developed, managed, etc.) another user different from the user, and the other user can be located at a geographically different location from the user. Further, the user can access the test automation system 400 via a node different from the node on which the software is deployed or hosted (e.g., a user device).

[0110] According to an embodiment, the first user input can include information related to at least one user-defined test case. In this regard, at least one processor 430 is configured to generate at least one test scenario template based on the at least one user-defined test case; obtain one or more test package artifacts based on the at least one test scenario template; and generate at least one test package by combining the at least one test scenario template with the one or more test package artifacts.

[0111] According to an embodiment, at least one processor 430 is configured to receive from a user at least one second user input related to at least one user-defined test plan, generate at least one test plan template based on the second user input and at least one test package, generate at least one test plan by adding at least one user-defined test case to the at least one test plan template, receive from the user at least one third user input related to at least one user-defined test cycle, and generate at least one test environment configuration file based on the third user input and at least one test plan. In this regard, the at least one test environment configuration file can include information defining the configuration of at least one test bench.

[0112] The systems and / or methods described herein can be implemented in different forms of hardware, firmware, or a combination of hardware and software. The actual specialized control hardware or software code used to implement these systems and / or methods does not limit the embodiments. Therefore, the operations and behaviors of the systems and / or methods have been described herein without reference to specific software code, and it is understood that software and hardware can be designed to implement the systems and / or methods based on the description herein.

[0113] In some implementations, the pre-test for verifying at least one test package and at least one test environment configuration file may be a simplified test that includes testing only essential functions and / or user-specified functions. The pre-test may be executed on a node different from the node on which the test performed to determine software changes is performed (e.g., what may be referred to herein as "actual testing"). For example, the pre-test may be executed locally on a device or workstation associated with the user, and the actual test may be executed on-the-fly and dynamically selected by a test automation system on multiple nodes according to the real-time or near real-time status of multiple nodes (e.g., resource availability, performance metrics, etc.).

[0114] Additionally or alternatively, at least one processor 430 may be configured to receive approval / denial of the pre-test results from another user different from the user. For example, the other user may be a senior test engineer responsible for verifying the accuracy of a test package and / or a test environment configuration file generated according to test requirements defined by the user. In this way, the other user can efficiently and effectively ensure that the test requirements are properly defined by the user.

[0115] According to an embodiment, at least one processor 430 may be configured to determine a software change by obtaining the current status of the software and comparing the current status with the last known status of the software to determine whether a change has occurred in the software. The last known status of the software may be obtained in advance by at least one processor 430 (during the previous test cycle).

[0116] In some implementations, software changes may include breaking changes that meet one or more conditions defined within at least one test package. In this regard, at least one processor 430 may be further configured to determine whether a software change is a breaking change when determining the software change. If it is determined that the software change is a breaking change, at least one processor 430 may execute a test. Otherwise, based on determining that the software change is not a breaking change, at least one processor 430 may not execute a test.

[0117] According to an embodiment, at least one processor 430 generates at least one test bench associated with at least one test environment for testing software based on at least one verified test environment configuration file, selects at least one node (communicatively coupled to the test automation system 400) associated with at least one test environment defined within at least one test bench based on the at least one test bench, obtains the software (or associated program code) and associated test components (such as another software that interacts with the software under test), deploys the software and associated test components to the selected at least one node, and executes a test for testing the software within the selected at least one node based on at least one verified test package, and thus may be configured to execute a test.

[0118] According to an embodiment, when at least one test package and at least one test environment configuration file are verified, at least one processor 430 can be further configured to issue the verified at least one test package and the verified at least one test environment configuration file. For example, at least one processor 430 can store the verified at least one test package and the verified at least one test environment configuration file in one or more storage media (e.g., storage 420, content hosting server, etc.) accessible by other users. In this way, other users can choose to utilize the verified at least one test package and the verified at least one test environment configuration file when needed (e.g., when performing similar tests on the same or similar software).

[0119] According to an embodiment, when a test is executed, at least one processor 430 can be further configured to present one or more test results to a user. For example, at least one processor 430 can collect one or more test results related to the test, generate at least one graphical user interface (GUI) including the one or more test results, and present the at least one GUI to the user.

[0120] According to an embodiment, at least one processor 430 can be further configured to update at least one test package and / or at least one test environment configuration file. For example, at least one processor 430 can receive one or more user inputs related to one or more updated test requirements (e.g., updated test cases, updated test plans, updated test cycles, etc.), and accordingly update at least one test package and / or at least one test environment configuration file.

[0121] As an example, when receiving one or more user-defined updated test cases, at least one processor 430 generates one or more updated test scenario templates based on the one or more user-defined updated test cases, and obtains (e.g., creates, searches, retrieves, etc.) one or more test package artifacts associated with the one or more updated test scenario templates from among a plurality of nodes (e.g., nodes 120-1 to 120-N) associated with the test automation system, and combines the one or more updated test package artifacts with the one or more test package artifacts to generate one or more updated test packages, thereby updating at least one test package. According to an embodiment, at least one processor can determine one or more differences between the one or more user-defined updated test cases and the current user-defined test cases, and update the one or more generated test scenario templates to reflect the one or more differences, thereby updating the test scenario templates one or more times. At least one test environment configuration file can be updated in a similar manner by at least processor 430.

[0122] In view of the above, exemplary embodiments of the present disclosure provide a test automation system that automates an end-to-end process when managing tests for a user. Specifically, a user can simply provide intended test requirements (e.g., intended test cases, intended test plans, intended test cycles, etc.) to the test automation system, and the test automation system can automatically generate test packages and test environment configuration files based thereon. The user only needs to remotely provide test requirements from related nodes (e.g., user devices, workstations, etc.) communicatively coupled to the test automation system (e.g., via network 130), and does not need to physically move to a test facility as in related technologies.

[0123] Furthermore, before the test automation system publishes and uses the generated test package and the generated test environment configuration file for actual testing, it can verify the generated test package and / or the generated test environment configuration file. As a result, the user can confirm that the generated test package and the generated test environment configuration file are accurate, and since the user can automatically execute such operations based on the verified test package and the verified test environment configuration file, there is no need to manually monitor the status of the system during testing and manually execute the test after verifying the generated test package and the generated test environment configuration file.

[0124] Furthermore, the test automation system can automatically allocate resources for executing the test according to the verified test package and the verified test environment configuration file without user involvement. The test automation system can dynamically select a node having sufficient resources to execute the test and then execute the test on that node. Thereafter, the test automation system can automatically collect the test results and present them to the user.

[0125] For this purpose, the user can simply define the intended test requirements from any suitable node communicatively coupled to the test automation system, and the test automation system can automatically manage the test on behalf of the user. Therefore, the software test is managed in an efficient and effective manner, thereby reducing the burden on the user and shortening the software development period.

[0126] It should be understood that the specific order or hierarchy of blocks in the processes / flowcharts disclosed in this specification is an example of an illustrative approach. Based on design preferences, it should be understood that the specific order or hierarchy of blocks in a process / flowchart can be reconfigured. Further, some blocks can be combined or omitted. The appended method claims present the elements of the various blocks in an illustrative order and are not limited to the specific order or hierarchy presented.

[0127] Some embodiments can relate to systems, methods, and / or computer-readable media at any possible technical detail level of integration. Further, one or more of the above-described components can be stored on a computer-readable medium and implemented as instructions executable by at least one processor (and / or can include at least one processor). The computer-readable medium can include a computer-readable non-transitory storage medium having computer-readable program instructions for causing a processor to perform operations.

[0128] A computer-readable storage medium can be a tangible device that holds and stores instructions for use by an instruction execution device. A computer-readable storage medium may be, for example, but is not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. A list, which does not purport to list all possible examples of a computer-readable storage medium, includes portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disk read-only memory (CD-ROM), digital versatile disks (DVDs), memory sticks, floppy disks, punch cards, mechanically encoded devices such as a raised structure in a groove having recorded instructions, and any suitable combination of the foregoing. A computer-readable medium as used herein should not be construed to be a signal per se, such as a radio wave, or other freely propagating electromagnetic wave, an electromagnetic wave propagating through a waveguide or other transmission medium (e.g., an optical pulse passing through an optical fiber cable), or an electrical signal transmitted through a wire.

[0129] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or to an external computer or external storage device via a network such as, for example, the Internet, a local area network, a wide area network, and / or a wireless network. The network can comprise copper transmission cables, optical transmission fibers, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and transfers the computer-readable program instructions for storage in a computer-readable storage medium within each computing / processing device.

[0130] The computer-readable program code / instructions for performing the operations may be in any combination of one or more programming languages, including source code or object code written in assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state-setting data, configuration data for integrated circuits, or object-oriented programming languages such as Smalltalk, C++, and procedural programming languages such as the "C" programming language or similar program languages. The computer-readable program instructions can be executed entirely on the user's computer, partially on the user's computer, as a stand-alone software package, partially on the user's computer, and partially on a remote computer, or entirely on a remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computer (e.g., through the Internet using an Internet service provider). In some embodiments, for example, an electronic circuit including a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA) can execute the computer-readable program instructions by personalizing the electronic circuit using the state information of the computer-readable program instructions to perform the aspects or operations.

[0131] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for realizing the functions / operations specified in the blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that includes instructions for a manufactured article that comprises a computer, a programmable data processing apparatus, and / or other devices that function in a particular manner, such that the computer-readable storage medium contains instructions that realize the aspects of the functions / operations specified in the blocks of the flowchart and / or block diagram.

[0132] The computer-readable program instructions can also be loaded onto a computer, other programmable apparatus, or other device, such that a series of operational steps are performed on the computer, other programmable apparatus, or other device to produce a process that is realized by the computer to realize the functions / operations specified in the blocks of the flowchart and / or block diagram.

[0133] The flowcharts and block diagrams in the drawings illustrate the structure, functionality, and operation of possible realizations of systems, methods, and computer-readable media according to various embodiments. In this regard, each block in a flowchart or block diagram can represent a module, segment, or portion of instructions, which includes one or more executable instructions for implementing a specified logical function. The methods, computer systems, and computer-readable media can include additional blocks, fewer blocks, different blocks, or blocks arranged differently than those shown in FIG. 1. In some alternative realizations, the functions represented by the blocks can occur in an order different from that shown in the figures. For example, two blocks shown in succession can actually be executed simultaneously, or substantially simultaneously, or depending on the functions involved, can be executed in the reverse order of the blocks. It will also be appreciated that each block in the illustrations of the block diagrams and / or flowcharts, and combinations of blocks in the illustrations of the block diagrams and / or flowcharts, can be implemented by a system based on special-purpose hardware that performs the specified functions or operations, and combinations of special-purpose hardware and computer instructions.

[0134] The systems and / or methods described herein can be realized in different forms of hardware, firmware, or combinations of hardware and software. The actual specialized control hardware or software code used to implement these systems and / or methods does not limit the embodiments. Therefore, the operation and behavior of the systems and / or methods have been described herein without reference to specific software code, and it is understood that software and hardware can be designed to implement the systems and / or methods based on the description herein.

Claims

1. A method implemented by at least one processor for managing tests for testing software of an embedded system, the method comprising: the method further comprises: receiving, from a user, at least one first user input related to one or more test requirements; generating at least one test package based on the at least one first user input; generating at least one test environment configuration file based on the at least one test package; validating the at least one test package and the at least one test environment configuration file; determining a change to the software; executing the test according to the at least one validated test package and the at least one validated test environment configuration file based on the determination of the change to the software; comprising; the software of the embedded system includes an in-vehicle electronic control unit (ECU); method.

2. publishing the at least one validated test package and the at least one validated test environment configuration file by storing the at least one validated test package and the at least one validated test environment configuration file in one or more storage media accessible by other users; further comprising; The method according to claim 1.

3. The at least one first user input includes information related to at least one user-defined test case, generating the at least one test package includes: generating at least one test scenario template based on the at least one user-defined test case; obtaining one or more test package artifacts based on the at least one test scenario template; combining the at least one test scenario template with the one or more test package artifacts to generate the at least one test package; comprising; The method according to claim 1 or claim 2.

4. generating the at least one test environment configuration file includes: receiving, from the user, at least one second user input related to at least one user-defined test plan; Generating at least one test plan template based on the second user input and at least one of the test packages; Generating at least one test plan by adding at least one of the test plan templates to at least one of the user-defined test cases; Receiving from the user at least one third user input related to at least one user-defined test cycle; Generating at least one of the test environment configuration files based on the third user input and at least one of the test plans; including; The method according to claim 1 or claim 2.

5. At least one of the test environment configuration files includes information defining settings for at least one test bench. The method according to claim 1 or claim 2.

6. Verifying at least one of the test packages and at least one of the test environment configuration files includes: Performing a pre-test of the software based on at least one of the test packages and at least one of the test environment configuration files; Presenting the results of the pre-test to the user; Receiving from the user at least one fourth user input related to one of approval and rejection of the results of the pre-test; Determining that at least one of the test packages and at least one of the test environment configuration files is valid based on determining that the fourth user input is associated with approval; Determining that at least one of the test packages and at least one of the test environment configuration files is invalid based on determining that the fourth user input is associated with rejection; including; The method according to claim 1 or claim 2.

7. Determining a change in the software includes: Obtaining the current status of the software; Comparing the current status with the last known status of the software to determine whether a change has occurred in the software; including; The method according to claim 1 or claim 2.

8. The change to the software includes a breaking change that meets one or more conditions defined in at least one of the test packages, The method according to claim 1 or claim 2.

9. Executing the test includes, generating at least one test bench related to at least one test environment for testing the software based on at least one of the verified test environment configuration files; selecting at least one node related to at least one test environment defined in at least one of the test benches based on at least one of the test benches; deploying the software to the selected at least one node; performing a test for testing the software on the selected at least one node based on at least one of the verified test packages; including, The method according to claim 1 or claim 2.

10. collecting one or more test results related to the test; generating at least one graphical user interface (GUI) including the one or more test results; presenting the at least one GUI to the user; further including, The method according to claim 1 or claim 2.

11. A system for managing a test for testing software of an embedded system, the system comprising: the system includes, a storage for storing instructions; receiving, from a user, at least one first user input related to one or more test requirements; generating at least one test package based on the at least one first user input; generating at least one test environment configuration file based on at least one of the test packages; verifying at least one of the test packages and at least one of the test environment configuration files; determining a change to the software; at least one processor configured to execute the instructions for executing the test according to at least one of the verified test packages and at least one of the verified test environment configuration files based on determining the change to the software; and comprising, the software of the embedded system includes an in-vehicle electronic control unit (ECU), system.

12. At least one of the processors is configured to: Publish at least one verified test package and at least one verified test environment configuration file by storing the at least one verified test package and the at least one verified test environment configuration file in one or more storage media accessible by other users; And is further configured to execute the instructions for; The system according to claim 11. **Claim 13** The first user input includes information related to at least one user-defined test case; At least one of the processors is configured to: Generate at least one test scenario template based on at least one of the user-defined test cases; Obtain one or more test package artifacts based on at least one test scenario template; Combine at least one test scenario template with one or more test package artifacts to generate at least one test package; And is further configured to execute the instructions for generating at least one test package; The system according to claim 11 or claim 12. **Claim 14** At least one of the processors is configured to: Receive from the user at least one second user input related to at least one user-defined test plan; Generate at least one test plan template based on the second user input and at least one of the test packages; Generate at least one test plan by adding at least one test plan template to at least one of the user-defined test cases; Receive from the user at least one third user input related to at least one user-defined test cycle; Generate at least one test environment configuration file based on the third user input and at least one of the test plans; And is further configured to execute the instructions for generating at least one test environment configuration file; The system according to claim 11 or claim 12. **Claim 15** At least one of the test environment configuration files includes information defining the settings of at least one test bench. The system according to claim 11 or claim 12.

16. At least one of the processors Based on at least one of the test packages and at least one of the test environment configuration files, perform a pre-test of the software, Present the results of the pre-test to the user, Receive from the user at least one fourth user input related to one of approval of the results of the pre-test and rejection of the results of the pre-test, Based on determining that the fourth user input is associated with approval, determine that at least one of the test packages and at least one of the test environment configuration files are valid, Based on determining that the fourth user input is associated with rejection, determine that at least one of the test packages and at least one of the test environment configuration files are invalid, Thereby, it is further configured to execute the instructions for verifying at least one of the test packages and at least one of the test environment configuration files. The system according to claim 11 or claim 12.

17. At least one of the processors Obtain the current status of the software, Compare the current status with the last known status of the software to determine whether a change has occurred in the software, Thereby, it is further configured to execute the instructions for determining a change in the software. The system according to claim 11 or claim 12.

18. The change in the software includes a destructive change that satisfies one or more conditions defined in at least one of the test packages. The system according to claim 11 or claim 12.

19. At least one of the processors Based on at least one of the verified test environment configuration files, generate at least one test bench related to at least one test environment for testing the software, Based on at least one test bench, select at least one node related to at least one test environment defined in at least one of the test benches. Deploy the software to the selected at least one node, Based on the at least one verified test package, perform a test for testing the software on the selected at least one node, Thereby being further configured to execute the instructions for executing the test, The system according to claim 11 or claim 12.

20. At least one of the processors, Collect one or more test results related to the test, Generate at least one graphical user interface (GUI) including the one or more test results, Present at least one of the GUIs to the user, Thereby being further configured to execute the instructions for, The system according to claim 11 or claim 12.

Images