System and method for managing tests

Abstract

The invention relates to a system and a method for managing tests. The present disclosure provides systems, methods, and devices for managing a test for testing software of an embedded system. In accordance with an embodiment, the method is implemented by at least one processor, comprising: receiving at least one first user input associated with one or more test requirements from a user, generating at least one test pack based on the first user input, generating at least one test environment composition file based on the at least one test pack, and a testing unit that verifies the at least one test pack and the at least one test environment configuration file, determines a change in the software, and performs a test in accordance with the verified at least one test pack and the verified at least one test environment configuration file on the basis of the determination of the change in the software, the software of the embedded system including an in-vehicle electronic control unit (ECU).

Description

Technical Field

[0001] Systems and methods consistent with exemplary embodiments of the present disclosure relate to test management, and more particularly, to managing the testing of software associated with an embedded system of a vehicle. Background Art

[0002] Testing of software requires ensuring that the software functions as intended, meets specified requirements, and functions reliably in various scenarios. Software testing is an important part of the software development life cycle (SDLC), and software testing is performed before the software is deployed to an actual system to identify defects, bugs, or vulnerabilities in the software.

[0003] When the software includes complex functions and / or needs to interoperate with other software, the testing of the software becomes complex and may involve multiple processes and users / stakeholders. For example, in the development of vehicle-related functions in vehicle systems such as lane change assist and mobile smart key, multiple electronic control units (ECUs) may be developed to perform the intended functions and interoperate with each other.

[0004] Regarding this, the ECUs can be managed by different users respectively and / or can be located at geographically different locations respectively. For example, the first ECU is sometimes developed by a first developer located at a first location (e.g., an in-house development engineer of an automobile manufacturer), and the first ECU may need to interoperate with a second ECU developed by a second developer located at a second location (e.g., a supplier). As another example, the first ECU is sometimes tested by a third user located at a third location (e.g., a test engineer). Moreover, the first ECU sometimes interoperates with hardware (e.g., a physical ECU, etc.) located at a fourth location, and thus, the testing of the first ECU requires the participation of the hardware.

[0005] Therefore, in the prior art, when a user wants to test software (e.g., a software-based ECU), the user needs to access a physical test facility centrally equipped with the software and associated test components (e.g., software components such as software-based ECUs developed by other users, hardware components such as physical ECUs, etc.). Therefore, in software testing, the user may need to physically access the test facility to set up and execute the test, and thus, it may take time and be a burden for the user.

[0006] In addition, in the prior art, it is sometimes difficult for users to set up, edit, and adjust tests in a test facility. This is because, especially in cases where new functions or software that have not been tested in the past need to be tested in a complex test scenario, the test components available in the test facility are general and limited, and the associated test packages are restricted and may not meet the intended test requirements or test scenarios.

[0007] Moreover, sometimes the test setup is complex and time-consuming. Therefore, most tests are executed at the point when the setup process is completed without verifying the accuracy of the test setup. Especially in cases where the user setting up the test is inexperienced and prone to making mistakes during the test setup, this may lead to inaccurate test results. Also, even when an error in the test setup is found during the test, the user sometimes cannot quickly correct the error on the spot.

[0008] In addition, in the prior art, users sometimes cannot access information about the test components available in a test facility before accessing the test facility. For example, a vendor sometimes cannot obtain information about the available test packages and / or associated restrictions / constraints of a test facility managed by an automobile manufacturer. Therefore, it is difficult for the vendor to accurately construct test requirements or a test plan without physically accessing the test facility.

[0009] Moreover, in the prior art, most (if not all) of the end-to-end processes of testing, such as the definition of test requirements, the determination of appropriate test packages based on the test requirements, the selection and setup of an appropriate test environment, the collection of associated test components, the deployment of the software under test and associated test components to the test environment, the execution of tests on the software based on the determined test packages, the acquisition of test results, and the reproduction of test results, are performed separately, manually managed by different users and / or on different systems, which is inefficient, burdensome, and may lead to human errors.

[0010] In view of the above, the development of existing software may take time, impose a heavy burden on users, and may also be delayed due to the unreliability of test management. SUMMARY OF THE INVENTION

[0011] According to an embodiment, there are provided a method and a system for efficiently and effectively managing one or more tests for one or more software for testing an embedded system of a vehicle. For example, exemplary embodiments of the present disclosure provide a method and a system for processing the end-to-end process of test management, such as collecting user input, providing test packages, providing test environment configuration files, verifying test packages and / or test environment configuration files, and executing one or more tests.

[0012] According to an embodiment, a method for managing a test for testing software of an embedded system is provided. The method may be implemented by at least one processor and may include: receiving from a user at least one first user input associated with one or more test requirements; generating at least one test package based on the first user input; generating at least one test environment configuration file based on the at least one test package; validating the at least one test package and the at least one test environment configuration file; determining a change in the software; and based on the determination of the change in the software, executing the test according to the validated at least one test package and the validated at least one test environment configuration file. The software of the embedded system may include an in-vehicle electronic control unit (ECU).

[0013] According to an embodiment, the method may further include: storing the validated at least one test package and the validated at least one test environment configuration file in one or more storage media accessible by other users, thereby making the validated at least one test package and the validated at least one test environment configuration file public. In addition, the method may include: collecting one or more test results associated with the test; generating at least one graphical user interface (GUI) including the one or more test results; and presenting the at least one GUI to the user.

[0014] According to an embodiment, the first user input may include information associated with at least one user-defined test case, and generating at least one test package may include: generating at least one test scenario template based on the at least one user-defined test case; obtaining one or more test package artifacts based on the at least one test scenario template; and combining the at least one test scenario template with the one or more test package artifacts to generate at least one test package.

[0015] According to an embodiment, generating at least one test environment configuration file may include: receiving from the user at least one second user input associated with at least one user-defined test plan; generating at least one test plan template based on the second user input and the at least one test package; generating at least one test plan by adding at least one test plan template to the at least one user-defined test case; receiving from the user at least one third user input associated with at least one user-defined test cycle; and generating at least one test environment configuration file based on the third user input and the at least one test plan. The at least one test environment configuration file may include information defining settings for at least one test bench.

[0016] According to an embodiment, validating at least one test package and at least one test environment configuration file may include: performing a pre-test of the software based on the at least one test package and the at least one test environment configuration file; presenting the results of the pre-test to a user; receiving from the user at least one fourth user input associated with one of approval and rejection of the results of the pre-test; determining that the at least one test package and the at least one test environment configuration file are valid based on determining that the fourth user input is associated with approval; and determining that the at least one test package and the at least one test environment configuration file are invalid based on determining that the fourth user input is associated with rejection.

[0017] According to an embodiment, determining a change in software may include: obtaining the current state of the software; and comparing the current state with the last known state of the software to determine whether a change has occurred in the software. A change in the software may include a disruptive change that satisfies one or more conditions defined in at least one test package.

[0018] According to an embodiment, performing a test may include: generating at least one test platform associated with at least one test environment for testing the software based on the validated at least one test environment configuration file; selecting at least one node associated with the at least one test environment defined in the at least one test platform based on the at least one test platform; deploying the software on the selected at least one node; and performing a test for testing the software on the selected at least one node based on the validated at least one test package.

[0019] According to an embodiment, there is provided a system for managing a test for testing software of an embedded system. The system may include: a storage device storing instructions; and at least one processor configured to execute instructions for performing the following actions: receiving from a user at least one first user input associated with one or more test requirements, generating at least one test package based on the first user input, generating at least one test environment configuration file based on the at least one test package, validating the at least one test package and the at least one test environment configuration file, determining a change in the software; and performing a test according to the validated at least one test package and the validated at least one test environment configuration file based on determining the change in the software. The software of the embedded system may include an in-vehicle electronic control unit (ECU).

[0020] According to an embodiment, at least one processor may also be configured to execute instructions for performing the following actions: storing the at least one verified test package and the at least one verified test environment configuration file in one or more storage media accessible by other users, thereby making the at least one verified test package and the at least one verified test environment configuration file public. In addition, at least one processor may also be configured to execute instructions for performing the following actions: collecting one or more test results associated with a test, generating at least one graphical user interface (GUI) including the one or more test results, and presenting the at least one GUI to a user.

[0021] According to an embodiment, the first user input includes information associated with at least one user-defined test case, and at least one processor may also be configured to execute instructions for generating at least one test package by performing the following actions: generating at least one test scenario template based on the at least one user-defined test case, obtaining one or more test package artifacts based on the at least one test scenario template, and combining the at least one test scenario template with the one or more test package artifacts to generate at least one test package.

[0022] According to an embodiment, at least one processor may also be configured to execute instructions for generating at least one test environment configuration file by performing the following actions: receiving from a user at least one second user input associated with at least one user-defined test plan, generating at least one test plan template based on the second user input and the at least one test package, generating at least one test plan by appending the at least one test plan template to the at least one user-defined test case, receiving from the user at least one third user input associated with at least one user-defined test cycle, and generating at least one test environment configuration file based on the third user input and the at least one test plan. The at least one test environment configuration file may include information defining settings for at least one test platform.

[0023] According to an embodiment, at least one processor may also be configured to execute instructions for verifying the at least one test package and the at least one test environment configuration file by performing the following actions: performing a pre-test of software based on the at least one test package and the at least one test environment configuration file, presenting the results of the pre-test to a user, receiving from the user at least one fourth user input associated with either approval or rejection of the results of the pre-test, determining that the at least one test package and the at least one test environment configuration file are valid based on a determination that the fourth user input is associated with approval, and determining that the at least one test package and the at least one test environment configuration file are invalid based on a determination that the fourth user input is associated with rejection.

[0024] According to an embodiment, at least one processor may also be configured to execute instructions for determining a change in software by performing the following actions: obtaining a current state of the software, and comparing the current state with a last known state of the software to determine whether the software has changed. A change in the software may include a disruptive change that satisfies one or more conditions defined in at least one test package.

[0025] According to an embodiment, at least one processor may also be configured to execute instructions for performing a test by performing the following actions: generating at least one test platform associated with at least one test environment for testing software based on at least one verified test environment configuration file, selecting at least one node associated with at least one test environment defined in at least one of the at least one test platforms, deploying the software on the selected at least one node, and performing a test for testing the software on the selected at least one node based on at least one verified test package.

[0026] Additional solutions are described in part in the following description and are apparent from the description or can be achieved by practicing the embodiments presented in the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Hereinafter, features, advantages, and significance of exemplary embodiments of the present disclosure will be described with reference to the drawings in which like reference numerals denote like elements.

[0028] Figure 1 is a block diagram of an exemplary system configuration for managing tests according to one or more embodiments.

[0029] Figure 2 is a block diagram of an exemplary module of a test automation system according to one or more embodiments.

[0030] Figure 3A is a flowchart of an exemplary use case for test preparation according to one or more embodiments.

[0031] Figure 3B is a flowchart of an exemplary use case for test execution according to one or more embodiments.

[0032] Figure 4 is a block diagram of an exemplary component of a test automation system according to one or more embodiments. DETAILED DESCRIPTION

[0033] Refer to the accompanying drawings for the following detailed description of the exemplary embodiments. The foregoing disclosure provides examples and descriptions, but is not intended to be exhaustive or to limit the embodiments to the precise forms disclosed. Modifications and variations are possible in light of the foregoing disclosure, or may be acquired through the practice of the embodiments. Also, one or more features or components of one or more embodiments may be incorporated into other embodiments (or one or more features of other embodiments), or may be combined therewith. Additionally, it will be understood that in the descriptions of the actions provided below, one or more actions may be omitted, one or more actions may be added, one or more actions may be performed simultaneously (at least in part) and the order of one or more actions may be switched.

[0034] Even if specific combinations of features are recited in the claims and / or disclosed in the specification, such combinations are not intended to limit the disclosure of possible implementations. In fact, most of these features can be combined in ways not specifically recited in the claims and / or not disclosed in the specification. Each of the dependent claims listed below may depend directly only on one claim, but the disclosure of possible implementations includes each dependent claim in combination with all the other claims within the set of claims.

[0035] Unless so explicitly stated, any element, action, or instruction used in this specification should not be construed as decisive or essential. Additionally, as used in this specification, the articles "a" and "an" are intended to include one or more items and may be used interchangeably with "one or more." The term "one" or similar language is used when only one item is intended. Further, as used in this specification, terms such as "has," "have," "includes," "comprises," etc. are intended to be open terms without limitation. Also, unless explicitly stated otherwise, the phrase "based on" is intended to mean "at least in part based on ~." Moreover, expressions such as "at least one of [A] and [B]" or "at least one of [A] or [B]" should be understood to include only A, only B, or both A and B.

[0036] References throughout this specification to "one embodiment," "an embodiment," "a non-limiting preferred embodiment," or like terms mean that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment of the solution. Thus, the phrases "in one embodiment," "in an embodiment," "in a non-limiting preferred one embodiment," and like terms throughout this specification may all refer to the same embodiment, but not necessarily so.

[0037] Moreover, the features, advantages, and characteristics of the present disclosure described may be combined in any suitable manner in one or more embodiments. Those skilled in the art should recognize that, in light of the description of this specification, the present disclosure may be implemented without one or more of the specific features or advantages of a particular embodiment. In another example, additional features and advantages may be recognized in a particular embodiment that may not be present in all embodiments of the present disclosure.

[0038] In addition, when terms such as "vehicle" are used in this specification, it may refer to any electrified and / or mechanical machine such as an automobile, a truck, a motorcycle, a bus, a bicycle, a mobility scooter, etc. that can carry or transport people and / or goods.

[0039] Exemplary embodiments consistent with the present disclosure provide methods, systems, and apparatuses for managing the testing of one or more software for testing in-vehicle ECUs and other embedded systems. Specifically, the methods, systems, apparatuses, etc. of the exemplary embodiments can automatically manage the testing according to user-defined requirements. According to an embodiment, the methods, systems, apparatuses, etc. of the exemplary embodiments can automatically generate one or more test packages and one or more test environment configuration files based on one or more user inputs, and can automatically execute the testing according to them.

[0040] In some embodiments, the one or more generated test packages and / or the one or more generated test environment configuration files may be verified before being used to execute the testing. After verifying the one or more generated test packages and / or the one or more generated test environment configuration files, the methods, systems, apparatuses, etc. of the exemplary embodiments can automatically determine software changes based on this, and automatically execute the testing based on the determined software changes. Moreover, the verified test packages and / or the verified test environment configuration files can be made public and accessible by other users.

[0041] According to an embodiment, after executing the testing on the software, the methods, systems, apparatuses, etc. of the exemplary embodiments can automatically collect the test results and present them to the associated users. Moreover, the methods, systems, apparatuses, etc. of the exemplary embodiments can receive one or more user inputs for updating one or more of the generated test packages and / or one or more of the generated test environment configuration files, and can automatically update them in response thereto.

[0042] Therefore, the methods, systems, apparatuses, etc. of the exemplary embodiments can automatically generate or update one or more test components according to user inputs based on real-time or near-real-time status and test requirements. As a result, one or more test platforms can be provided and utilized on demand without geographical restrictions.

[0043] For this purpose, exemplary embodiments of the present disclosure can automatically manage an end-to-end process of a test according to one or more user-defined requirements. Without the need for the user to physically move to a test facility as required in the prior art, the user can remotely define one or more test requirements. Ultimately, the exemplary embodiments of the present disclosure can more efficiently develop software, significantly reduce the burden on the user, greatly reduce the development time, and significantly reduce the costs and labor for planning access to the test facility and business trips.

[0044] The features, advantages, and significance attempts of the above exemplary embodiments are only a part of the present disclosure and are not intended to be comprehensive or to limit the technical scope of the present disclosure. Further descriptions of the features, components, configurations, actions, and implementations of the exemplary embodiments of the present disclosure, along with the accompanying technical advantages and meanings, are also provided below.

[0045] Figure 1 is a block diagram of an exemplary system configuration 100 for managing tests according to one or more embodiments. As Figure 1 shown, the system configuration 100 may include a test automation system 110, a plurality of nodes 120-1 to 120-N, and a network 130.

[0046] Generally, the test automation system 110 may be configured to be communicatively coupled to the plurality of nodes 120-1 to 120-N via the network 130, interact with the plurality of nodes 120-1 to 120-N, and manage tests (or one or more associated information or data). Hereinafter, reference Figures 2 to 4 is made to provide descriptions of exemplary components that may be included in the test automation system 110 and descriptions of associated use cases.

[0047] Each of the plurality of nodes 120-1 to 120-N may include one or more devices, machines, systems, or any other suitable components that can receive, host, store, effectively utilize, deploy, process, provide, etc., one or more artifacts or components that make up a test.

[0048] For example, node 120-1 may include a device or machine (e.g., a personal computer, a server or a server cluster, a workstation, etc.) for constructing, storing, executing, or simulating one or more virtual ECUs, one or more simulated ECUs, and / or any other suitable software-based components (e.g., a vehicle model, a data communication module (DCM) model, a heating, ventilation, and air conditioning (HVAC) model, etc.) that can be utilized in a vehicle system, as well as one or more computer-executable software applications. As another example, node 120-1 may include one or more fully developed physical ECUs, one or more partially developed physical ECUs, and one or more hardware components such as one or more vehicle hardwares (e.g., a powertrain, an engine, etc.).

[0049] According to an embodiment, one or more of the plurality of nodes 120-1 to 120-N may include one or more interfaces, each of which may be configured to communicatively couple the associated node with the test automation system 110. For example, one or more of the plurality of nodes may include a hardware interface, a software interface (e.g., a program interface, an application programming interface (API), etc.).

[0050] According to an embodiment, at least a portion of the plurality of nodes 120-1 to 120-N is located at a location geographically different from the test automation system 110 and / or different from another portion of the plurality of nodes. According to an embodiment, at least a portion of the plurality of nodes 120-1 to 120-N is associated with a user located at a geographically different location.

[0051] For example, node 120-1 may be associated with a first user (e.g., a development engineer) responsible for developing the first ECU, node 120-2 may be associated with a second user (e.g., a test engineer) responsible for testing the first ECU, the first user may be located at a first location, the second user may be located at a second location, and the first location may be different from the second location. Alternatively or additionally, the first node 120-1 may be associated with a first user responsible for the development of the first ECU, node 120-2 may be associated with a second user responsible for the development of the second ECU, the first ECU may interact with the second ECU, the first user may be located at a first location, the second user may be located at a second location, and the first location may be different from the second location.

[0052] According to an embodiment, at least a portion of the plurality of nodes 120-1 to 120-N may be associated with one or more test environments. For example, the portion of the nodes may have at least one software-based test environment (e.g., software-in-the-loop (SIL) test environment, virtual ECU (V-ECU) test environment, model-in-the-loop (MIL) test environment, processor-in-the-loop (PIL) test environment, etc.) and / or at least one hardware-based test environment (e.g., hardware-in-the-loop (HIL) test environment), and may be communicatively coupled (e.g., wired coupling, wireless coupling, etc.) or deployed to the test environment.

[0053] Moreover, at least a portion of the plurality of nodes 120-1 to 120-N may include one or more storage media such as a server or a server cluster that can be configured to store, disclose, etc., one or more data or information (or information associated therewith) provided by the test automation system 110 and / or another portion of the plurality of nodes 120-1 to 120-N.

[0054] The network 130 may include one or more wired and / or wireless networks that can be configured to couple the plurality of nodes 120-1 to 120-N to the test automation system 110. For example, the network 130 may include a cellular network (e.g., fifth-generation (5G) network, long-term evolution (LTE) network, third-generation (3G) network, code division multiple access (CDMA) network, etc.), a public land mobile network (PLMN), a local area network (LAN), a wide area network (WAN), a metropolitan area network (MAN), a telephone line network (e.g., public switched telephone network (PSTN)), a private network, an ad hoc network, an intranet, the Internet, a fiber-based network, etc. and / or a combination of these or other types of networks.

[0055] According to an embodiment, network 130 may include a virtual network, which may include one or more physical network components (e.g., Ethernet (registered trademark), WiFi module, telecommunication network hardware, etc.) that implement one or more virtual network functions (e.g., Controller Area Network (CAN) bus, etc.). Network 130 may additionally or alternatively include at least one parameter network.

[0056] Next, referring to the block diagram showing exemplary modules of a test automation system 200 according to one or more embodiments Figure 2 of. The test automation system 200 may correspond to the test automation system 110 described above. Therefore, unless otherwise clearly stated, the features described for the systems 110 and 200 in this specification may be applicable to each other. Figure 1 As described above, the test automation system 200 may include at least one user interaction module 210, at least one testing framework module 220, at least one configuration management module 230, at least one test platform management module 240, and at least one test management module 250. As will be further described below, one or more of the modules 210-250 may be defined by computer-executable instructions or programming code, which, when executed by one or more processors (e.g., the processor 430 described below with reference to

[0057] As Figure 2 shown, cause the one or more processors to perform actions associated therewith. One or more of the modules 210-250 may alternatively or additionally include one or more hardware components (e.g., the components described below with reference to Figure 4 ), which may be configured to perform actions associated therewith. For this purpose, the modules 210-250 may be configured to interact with each other as described below to manage one or more tests. Figure 4 ), which may be configured to perform actions associated therewith. For this purpose, the modules 210-250 may be configured to interact with each other as described below to manage one or more tests.

[0058] The user interaction module 210 (sometimes referred to as "module 210" in this specification) may be configured to communicate with one or more users. According to an embodiment, module 210 is capable of generating one or more graphical user interfaces (GUIs) to enable one or more users to participate. For example, module 210 may generate one or more GUIs for the user and receive one or more user inputs from the user.

[0059] According to an embodiment, module 210 may be configured to receive one or more user inputs from one or more users, and the one or more user inputs may define one or more test requirements of a test case of a user intention, a test plan of a user intention, a test cycle of a user intention, and / or the like. As will be further described below, the one or more user inputs may be utilized when generating one or more test packages and / or one or more test environment configuration files. Moreover, module 210 may also be configured to receive from one or more users one or more inputs including information for updating one or more generated test components.

[0060] According to an embodiment, module 210 can generate one or more GUIs to present the status of a test to a user. For example, module 210 may present the status of a configured test (e.g., in progress, executed, failed, etc.), the result of a verification test, the result of an actual test, etc. in one or more GUIs.

[0061] The test framework module 220 (sometimes referred to as "module 220" in this specification) may be configured to generate one or more test packages and / or one or more test environment configuration files. According to an embodiment, module 220 may be configured to obtain one or more user inputs from module 210, and may be configured to generate one or more test packages and / or one or more test environment configuration files based on the obtained one or more user inputs.

[0062] For example, module 220 can generate one or more test packages based on one or more user inputs, and generate one or more test environment configuration files based on one or more test packages. In some embodiments, the one or more user inputs may include information associated with one or more user-defined test cases, and module 220 may generate one or more test packages by performing the following actions: creating one or more test scenario templates based on one or more user-defined test cases, obtaining one or more test package artifacts (e.g., creating, retrieving, extracting, etc.) associated with one or more test scenario templates from a plurality of nodes (e.g., nodes 120-1 to 120-N) associated with a test automation system, combining one or more test scenario templates with one or more test package artifacts, and generating one or more test packages.

[0063] In this regard, the term "test case" as described in this specification sometimes refers to a specific set of conditions and / or steps designed to verify the functionality or behavior of software (or the system under test). A test case can define one or more actions and expected results for a specific test scenario. According to an embodiment, a test case may include a test case ID (e.g., a unique identifier or a series of numbers assigned to the test case for tracking and reference), a test purpose (e.g., a description or specification of the goal, objective of the test case), at least one test condition (e.g., preconditions or initial state required to execute the test case, specific settings or configurations required to trigger test execution, etc.), at least one test step (e.g., actions or operations performed to execute the test case, required inputs, interactions with the software, etc.), at least one expected result (e.g., the result or behavior expected of the software when the test case is executed normally, etc.), and one or more of any other appropriate information.

[0064] Moreover, in this specification, the term "test package" may refer to a collection of other related resources organized together for testing test cases, test scenarios / algorithms, test data, test dependencies, test execution instructions, test cycles, and specific software (e.g., a specific ECU). By generating and providing a "test package" based on user-defined test requirements (included in the first user input), module 220 can provide a structured method for planning and executing tests in a way that the user intends while ensuring that all required test activities are included and properly documented.

[0065] Moreover, in this specification, the term "test environment configuration file" may refer to a file or document that includes information for obtaining (e.g., obtaining, creating, updating, etc.) and configuring at least one test environment associated with the software or system under test. For example, a test environment configuration file may include information related to the configuration of one or more test environments required to test the software (e.g., the configuration of a software-based test environment required to test the software and associated software-based components, the configuration of a hardware-based test environment required to test the hardware associated with the software, etc.), information related to the test plan, information related to the test scenario, information related to the test execution conditions, and information related to the test platform configuration.

[0066] In this specification, the term "test scenario" may refer to a set or arrangement of related test cases grouped together for a common test purpose. Each test case within a test scenario focuses on a specific aspect or condition, and the test scenario provides a larger context or process.

[0067] In this specification, the term "test plan" may refer to a set of information representing the method, purpose, scope, process, and composition outline of testing software. A test plan may include a set of or arrangement of test scenarios, test execution processes, and setting information such as test platform configuration and ECU test configuration.

[0068] In this specification, the term "test platform" may refer to a series of tools, processes, functional compositions, spare parts, etc. of information or parameters that can execute tests under desired conditions or configurations during compilation or utilization. Simply put, the test platform defines the test environment for testing software.

[0069] Additionally or alternatively, one or more user inputs may include information associated with one or more user-defined test plans and information associated with one or more user-defined test cycles. Module 220 may generate one or more test environment configuration files by performing the following actions: generating one or more test plan templates based on one or more user-defined test plans and one or more test packages; generating one or more test plans by appending one or more user-defined test cases to one or more test plan templates; and generating one or more test environment configuration files based on one or more user-defined test cycles and one or more test plans.

[0070] Additionally or alternatively, module 220 can update one or more generated test packages based on one or more user inputs. In some implementations, one or more user inputs may include information associated with one or more user-defined updated test cases. Module 220 may update one or more generated test packages by performing the following actions: generating one or more updated test scenario templates based on one or more user-defined updated test cases, obtaining (e.g., creating, retrieving, extracting, etc.) one or more test package artifacts associated with one or more updated test scenario templates from multiple nodes (e.g., nodes 120-1 to 120-N) associated with the test automation system, combining one or more updated test package artifacts with one or more test package artifacts, and generating one or more updated test packages. According to some embodiments, module 220 can determine one or more differences between one or more user-defined updated test cases and current user-defined test cases, and can generate one or more test scenario templates by modifying one or more generated test scenario templates in a manner that reflects one or more differences.

[0071] According to an embodiment, the module 220 can interact with the build management module 230, for example, storing or submitting one or more generated data (e.g., test packages, test environment build files, updated test packages, etc.) to the build management module 230, receiving a trigger for executing a test from the build management module 230, providing one or more test results, test logs, test traces, etc. (received from the test management module 250) to the build management module 230, etc.

[0072] According to an embodiment, the module 220 can interact with the test platform management module 240, for example, providing information or data to the test platform management module 240 to initialize test bench provision. According to an embodiment, the module 220 can interact with the test management module 250, for example, providing one or more generated / verified data (e.g., test packages, etc.) to the test management module 250, receiving one or more test results, test logs, test traces, etc. from the test management module 250, etc.

[0073] At least one build management module 230 (sometimes referred to as "module 230" in this specification) can be configured to manage information or builds associated with a test. For example, the module 230 can receive one or more generated data (e.g., test packages, test environment build files, updated test packages, etc.) from the module 220, and then organize and store the one or more generated data therein. According to an embodiment, the module 230 can include one or more storage media (e.g., the storage 420 described below with reference to Figure 4 a hosting server, a server cluster, etc.).

[0074] Moreover, the module 230 can also be configured to receive from one or more nodes (e.g., nodes 120-1 to 120-N, etc.) associated with the test automation system 200 one or more systems to be tested (e.g., software as a test object such as a software-based ECU, etc.) and one or more components associated therewith (e.g., one or more software that interacts with the system to be tested, such as an ECU based on one or more software, one or more vehicle-associated models, information on hardware associated with the system to be tested, etc.), and store them therein.

[0075] According to an embodiment, module 230 may also be configured to continuously (or periodically) determine changes to the DUT system. For example, module 230 can obtain the current state of the DUT system based on a test cycle defined in at least one test package of the DUT system, compare the current state with the last known state of the DUT system, and determine whether a change has occurred in the DUT system. According to an embodiment, a change to the DUT system includes a disruptive change that meets one or more conditions defined in at least one test package. Therefore, after determining a change to the DUT system, module 230 can generate a message (such as a notification message, etc.) for triggering test execution and send it to module 220.

[0076] The test platform management module 240 (sometimes referred to as "module 240" in this specification) may be configured to provide one or more test platforms for testing. For example, each time a test plan execution is triggered, module 240 can receive information or data such as a test environment configuration file (including test platform setting information) from module 220, and can obtain (e.g., extract, create, update, etc.) one or more test platforms based on the received information. According to an embodiment, module 240 can provide one or more test platforms to the test management module 250.

[0077] The test management module 250 (sometimes referred to as "module 250" in this specification) may be configured to execute one or more tests. For example, module 250 can receive one or more test platforms from module 240, and can initialize one or more test environments (e.g., software-based environments and / or hardware-based environments, etc.) according to the one or more test platforms. Moreover, module 250 can receive one or more test packages from module 220, and can execute one or more tests based on the one or more initialized test environments according to the one or more test packages.

[0078] For this purpose, the modules within the test automation system 200 may be configured to interact with each other to facilitate end-to-end automation in managing a user's tests. The user can only provide the intended test requirements (e.g., intended test cases, intended test plans, intended test cycles, etc.) to the test automation system 200, and the test automation system 200 can automatically generate the required configuration data (e.g., test packages, test environment configuration files, test platforms, etc.) based on the user-defined test requirements, automatically monitor the state of the DUT system (e.g., software), automatically trigger the execution of tests, automatically collect test results, and present them to the user.

[0079] Hereinafter, with reference to Figure 3A and Figure 3B , exemplary use cases of the modules within the test automation system 200 according to one or more embodiments will be described.Figure 3A and Figure 3B one or more of the modules 210 - 250 shown is / are the same as the modules 210 - 250 described above in this specification. Figure 2

[0080] First, refer to the flowchart of an exemplary use case representing the preparation of a test according to one or more embodiments. Figure 3A . As Figure 3A shown, in operation S3101, the module 210 may be configured to receive at least one first user input from the user (via a node associated with the user). The first user input may be associated with one or more test requirements and may include information associated with at least one user - defined test case.

[0081] In operation S3102, the module 210 may provide the first user input (received in operation S3101) to the module 220. Thus, in operation S3103, the module 220 may be configured to generate at least one test package based on the first user input. For example, the module 220 may be configured to generate at least one test scenario template based on at least one user - defined test case included in the first user input, and obtain one or more test package artifacts from a plurality of nodes (e.g., nodes 120 - 1 to 120 - N, etc.) associated with the test automation system 200. Thus, the module 220 can combine at least one test scenario template with one or more test package artifacts to generate at least one test package.

[0082] If further refer to Figure 3A , after generating at least one test package, in operation S3104, the module 220 can provide the generated test package to the module 230 for storage. Then, in operation S3105, the module 210 may be configured to receive at least one second user input from the user. The at least one second user input may be associated with at least one user - defined test plan.

[0083] Thus, in operation S3106, the module 210 can provide the second user input to the module 220. Then, in operation S3107, the module 220 can generate at least one test plan based on the second user input. For example, the module 220 may generate at least one test plan template based on the second user input and at least one test package (generated in operation S3103), and can generate at least one test plan by appending at least one user - defined test case to the at least one test plan template.

[0084] Moreover, in operation S3108, module 210 can be configured to receive at least one third user input from the user. The at least one third user input can be associated with at least one user-defined test period that defines how often the determination system (e.g., module 230) should perform tests. Thus, in operation S3109, module 210 can provide the third user input to module 220. Then, in operation S3110, module 220 can be configured to generate at least one test environment configuration file. The test environment configuration file can include the configuration of one or more test environments required to test the system under test, information associated with the test plan, information associated with the test scenario, information associated with the test execution conditions, and / or information associated with the test platform configuration.

[0085] In operation S3111, after generating the test environment configuration file, module 220 can provide the same file to module 230 for storage. In some implementations, before providing the test package and / or the test environment configuration file to module 230, module 220 can verify the test package and / or the test environment configuration file to ensure that the associated content is accurate and meets the user's requirements.

[0086] According to an embodiment, module 220 performs a pre-test on the system under test based on the test package and / or the test environment configuration file, presents the results of the pre-test to the user, and receives from the user at least one fourth user input associated with either approval or rejection of the results of the pre-test, thereby enabling verification of the test package and / or the test environment configuration file. Thus, module 220 can determine that the test and / or the test environment configuration file is valid based on a determination that the fourth user input is associated with approval. Otherwise, module 220 can determine that the test and / or the test environment configuration file is invalid based on a determination that the fourth user input is associated with rejection. The pre-test can be performed at a first node, which can be different from the node (e.g., the second node) where the actual test is performed. In this way, the test automation system can ensure the validity of the test package and / or the test environment configuration file before exposing the same to module 230.

[0087] For this purpose, to complete the preparation for the test, module 230 can be configured to continuously (or periodically) (e.g., according to a user-defined test period) monitor the state of the system under test and automatically trigger test execution when needed. Hereinafter, Figure 3B a description of an exemplary use case associated therewith will be described.

[0088] Referring to Figure 3B , Figure 3BA flowchart showing an exemplary use case of test execution according to one or more embodiments. Figure 3B The action of Figure 3A can be performed after the action as described above with reference to

[0089] As Figure 3B shown, in action S3201, module 230 can determine a change in the system under test (such as software, etc.). For example, module 230 continuously (or periodically) obtains the current state of the system under test (from one or more nodes associated with the system under test), and compares the current state with the last known state of the system under test (previously obtained and stored by module 230), thereby being able to determine whether the system under test has changed from the last known state. According to an embodiment, module 230 can perform action S3201 according to a user-defined test cycle (the associated information is included in the associated test package).

[0090] In addition, after determining the change, module 230 can determine whether the change is a disruptive change. For example, module 230 can determine whether the change meets one or more conditions defined in at least one test package. Module 230 can determine that the change is a disruptive change based on the determination that the change meets one or more conditions. In this regard, the term "disruptive change" may refer to a change or modification of the system under test that disrupts or interrupts the normal function of the system under test.

[0091] Therefore, in action S3202, module 230 can generate a message (such as a notification flag, warning message, etc.) and send it to module 220 to notify the determined change in the system under test. Then, in action S3203, module 220 collects the test environment configuration file associated with the test package and the system under test (from module 230, etc.), and provides the test environment configuration file to module 240, thereby being able to trigger test execution.

[0092] In action S3204, module 240 can initialize one or more test platforms. Specifically, module 240 can obtain (such as extract, create, configure, etc.) one or more test platforms for performing tests based on the test environment configuration file. Therefore, in action S3205, module 240 can provide one or more test platforms to module 250. Moreover, in action S3206, module 220 can provide the test package to module 250. In this regard, it can be understood that module 220 can perform action S3206 simultaneously with action S3203, S3204, or S320 without departing from the technical scope of the present disclosure.

[0093] In operation S3207, module 250 can perform a test for testing the test object system. Specifically, module 250 can determine one or more nodes (e.g., nodes 120-1 to 120-N) that can provide a test environment (e.g., a software-based test environment, a hardware-based test environment, etc.) according to one or more test platforms, can deploy the test object system on the determined one or more nodes (together with associated test components), and can perform a test on the one or more nodes based on a test package.

[0094] After performing the test, in operation S3208, module 220 can collect one or more test results associated with the test from module 250. Thus, in operation S3209, module 220 can archive the collected one or more test results by sending the one or more test results to module 230. After storing the one or more test results, in operation S3210, module 230 can provide the one or more test results to module 210. Thus, in operation S3211, module 210 can generate one or more GUIs including the one or more test results and then present the one or more GUIs to the user (via a node associated with the user).

[0095] It can be understood that the operations of the above modules 210 to 250 are only examples of possible embodiments, and the technical scope of the present disclosure should not be limited thereby. Specifically, one or more of modules 210 to 250 can be configured to operate by different methods described in this specification without departing from the technical scope of the present disclosure.

[0096] According to an embodiment, one or more of modules 210 to 250 within the test automation system can be defined in computer-executable instructions or programming code and can be associated with one or more hardware components of the test automation system. For example, the computer-executable instructions defining one or more of modules 210 to 250 can be stored in one or more storage memories of the test automation system, and as described in this specification, the computer-executable instructions can be executed by one or more processors of the test automation system to perform one or more operations associated with modules 210 to 250.

[0097] Referring to Figure 4 , Figure 4 FIG. is a block diagram showing exemplary components of a test automation system 400 according to one or more embodiments. The test automation system 400 can be the same as Figure 2 the test automation system 200, and the components included therein can be configured to perform one or more operations of one or more modules of the test automation system 200.

[0098] AsFigure 4 As shown, the test automation system 400 may include at least one communication interface 410, at least one memory 420, and at least one processor 430. However, it can be understood that the test automation system 400 may include more or fewer components than shown and / or, without departing from the technical scope of the present disclosure, the components included therein may be configured in any method different from that shown. For example, in some implementations, the test automation system 400 may include multiple memories 420 and multiple processors 430, which may be respectively dedicated to performing one action among modules 210 to 250.

[0099] The communication interface 410 may include components such as a transceiver (e.g., a transceiver, a separate receiver, and a transmitter, etc.) through which the test automation system 400 (or one or more components included therein) can communicate with one or more external components of the test automation system 400 via a wired connection, a wireless connection, or a combination of a wired connection and a wireless connection, etc. For example, the communication interface 410 may couple the test automation system 400 (or one or more components included therein) with multiple nodes (e.g., Figure 1 nodes 120-1 to 120-N, etc. of the like), thereby enabling communication with them and interacting with them respectively. As another example, the communication interface 410 may enable the components of the test automation system 400 to communicate with each other. For example, the communication interface 410 may couple the memory 420 with the processor 430, thereby enabling them to communicate and operate with each other.

[0100] According to an embodiment, the communication interface 410 may include hardware-based interfaces such as a bus interface, an Ethernet (registered trademark) interface, an optical interface, a coaxial interface, an infrared interface, a radio frequency (RF) interface, a universal serial bus (USB) interface, a Wi-Fi interface, a cellular network interface, a software interface, etc. According to an embodiment, the communication interface 210 may include at least one controller area network (CAN) bus that can be configured to communicatively couple the components of the test automation system 400 (e.g., the memory 420, the processor 430, etc.) with multiple nodes (e.g., nodes 120-1 to 120-N). Additionally or alternatively, the communication interface 410 may include software-based interfaces such as an application programming interface (API), a virtual network interface (e.g., a virtual CAN bus, etc.).

[0101] According to an embodiment, the communication interface 410 may be configured to receive information from one or more components external to the test automation system 400 and provide the information to the processor 430 for further processing and / or to the storage 420 for storage. For example, the communication interface 410 may receive one or more user inputs that define one or more test requirements (e.g., test cases, test plans, test cycles, etc.) from multiple nodes.

[0102] At least one storage 420 may include one or more storage media suitable for storing data, information, and / or computer-readable / computer-executable instructions. According to an embodiment, the storage 420 may include other types of dynamic or static storage devices (e.g., flash memory, magnetic memory, and / or optical memory) that store information and / or instructions for use by the random access memory (RAM), read-only memory (ROM), and / or the processor 430.

[0103] Additionally or alternatively, the storage 420 may include, together with corresponding drives, a hard disk (e.g., magnetic disk, optical disk, magneto-optical disk, and / or solid-state drive), a compact disc (CD), a digital versatile disc (DVD), a floppy disk, a cassette tape, a magnetic tape, and / or other types of non-transitory computer-readable media.

[0104] According to an embodiment, the storage 420 may be configured to store computer-executable instructions or programming code that define one or more of the modules 210-250 (refer to Figures 2 to 3B as described above). Moreover, the storage 420 may be configured to store information utilized by the processor 430 to facilitate the automation of test management. For example, the storage 420 may be configured to store one or more test requirements defined by one or more users, store one or more test object systems and associated test components or artifacts (e.g., programming code that defines the software under test and / or programming code that defines the software that interoperates with the software under test), store one or more generated test packages and generated test environment configuration files, and store one or more test results.

[0105] At least one processor 430 may include one or more processors that can be programmed to perform the functions or actions described in this specification. For example, the processor 430 may be configured to perform one or more actions or one or more operations described in this specification by executing computer-readable instructions stored in a storage medium (e.g., the storage 420, etc.).

[0106] According to an embodiment, the processor 430 may be configured to receive (e.g., via the communication interface 410, etc.) one or more signals that define one or more instructions for performing one or more actions. Further, the processor 430 may be implemented by hardware, firmware, or a combination of hardware and software. The processor 430 may include a central processing unit (CPU), a graphics processing unit (GPU), an accelerated processing unit (APU), a microprocessor, a microcontroller, a digital signal processor (DSP), a field-programmable gate array (FPGA), an application-specific integrated circuit (ASIC), and / or other types of processing or computing components.

[0107] According to an embodiment, at least one processor 430 may be configured to execute computer-executable instructions stored in at least one storage memory (e.g., the memory 420), thereby performing one or more actions for managing a test of test software (sometimes referred to as a "test object system" in this specification) for an embedded system. The software of the embedded system may include an in-vehicle ECU.

[0108] According to an embodiment, at least one processor 430 may be configured to perform the following actions: receive from a user at least one first user input associated with one or more test requirements; generate at least one test package based on the first user input; generate at least one test environment configuration file based on the at least one test package; verify the at least one test package and the at least one test environment configuration file; determine a change in the software; and based on determining the change in the software, execute the test according to the verified at least one test package and the verified at least one test environment configuration file.

[0109] The software is associated with other users different from the user (e.g., for development, management, etc.), and the other users may be located in a geographically different location from the user. Further, the user can access the test automation system 400 via a node different from the node where the software is deployed or hosted (e.g., a user device, etc.).

[0110] According to an embodiment, the first user input may include information associated with at least one user-defined test case. In this regard, at least one processor 430 may be configured to generate at least one test package by performing the following actions: generating at least one test scenario template based on at least one user-defined test case; obtaining one or more test package artifacts based on at least one test scenario template; and combining at least one test scenario template with one or more test package artifacts to generate at least one test package.

[0111] According to an embodiment, at least one processor 430 may be configured to generate at least one test environment configuration file by performing the following actions: receiving from a user at least one second user input associated with at least one user-defined test plan; generating at least one test plan template based on the second user input and at least one test package; generating at least one test plan by appending at least one user-defined test case to at least one test plan template; receiving from the user at least one third user input associated with at least one user-defined test cycle; and generating at least one test environment configuration file based on the third user input and at least one test plan. In this regard, at least one test environment configuration file may include information defining the configuration of at least one test platform.

[0112] The systems and / or methods described in this specification can be implemented in different forms of hardware, firmware, or a combination of hardware and software. The actual specialized control hardware or software code for implementing these systems and / or methods does not limit the implementation form. Therefore, the actions and behaviors of the systems and / or methods are described in this specification without reference to specific software code, and it should be understood that software and hardware can be designed to implement the systems and / or methods based on the description in this specification.

[0113] In some implementations, the pre-test for validating at least one test package and at least one test environment configuration file may be a simplified test that includes only essential functions and / or user-specified functions. The pre-test may be executed at a node different from the node where the test for judging software changes is executed (e.g., what may be referred to as "actual testing" in this specification). For example, the pre-test may be executed locally on a machine or workstation associated with the user, and the actual test may be executed on multiple nodes dynamically selected on-the-fly by a test automation system according to the real-time or near real-time status of multiple nodes (e.g., resource availability, performance metrics, etc.).

[0114] Additionally or alternatively, at least one processor 430 may be configured to receive approval / denial of the results of a pre-test from other users different from the user. For example, the other users may be senior test engineers responsible for verifying the accuracy of test packages and / or test environment configuration files generated according to test requirements defined by the user. In this way, the other users can efficiently and effectively ensure that the test requirements are properly defined by the user.

[0115] According to an embodiment, at least one processor 430 may be configured to determine a change in software by performing the following actions: obtaining the current state of the software; and comparing the current state with the last known state of the software to determine whether the software has changed. The last known state of the software may be pre-obtained by at least one processor 430 (in a previous test cycle).

[0116] In some implementations, a change in software may include a disruptive change that satisfies one or more conditions defined in at least one test package. In this regard, at least one processor 430 may also be configured to, after determining a change in software, determine whether the change in software is a disruptive change. After determining that the change in software is a disruptive change, at least one processor 430 can execute a test. Otherwise, at least one processor 430 may also not execute a test based on determining that the change in software is not a disruptive change.

[0117] According to an embodiment, at least one processor 430 may be configured to execute a test by performing the following actions: generating at least one test platform associated with at least one test environment for testing software based on the verified at least one test environment configuration file; selecting at least one node (communicatively coupled to the test automation system 400) associated with at least one test environment defined within at least one test platform based on the at least one test platform; obtaining the software (or associated program code) and associated test components (e.g., other software that interacts with the software under test, etc.); deploying the software and the associated test components to the selected at least one node; and executing a test for testing the software within the selected at least one node based on the verified at least one test package.

[0118] According to an embodiment, after validating at least one test package and at least one test environment configuration file, at least one processor 430 may also be configured to issue the validated at least one test package and the validated at least one test environment configuration file. For example, at least one processor 430 can store the validated at least one test package and the validated at least one test environment configuration file in one or more storage media (e.g., storage 420, content hosting server, etc.) that can be accessed by other users. In this way, other users can choose to utilize the validated at least one test package and the validated at least one test environment configuration file when needed (e.g., when performing the same test on the same or similar software, etc.).

[0119] According to an embodiment, after performing a test, at least one processor 430 may also be configured to present one or more test results to a user. For example, at least one processor 430 can collect one or more test results associated with the test, generate at least one graphical user interface (GUI) including the one or more test results, and present the at least one GUI to the user.

[0120] According to an embodiment, at least one processor 430 may also be configured to update at least one test package and / or at least one test environment configuration file. For example, at least one processor 430 can receive one or more user inputs associated with one or more updated test requirements (e.g., updated test cases, updated test plans, updated test cycles, etc.), and accordingly update at least one test package and / or at least one test environment configuration file.

[0121] As an example, in the case of receiving one or more user-defined updated test cases, at least one processor 430 can update at least one test package by performing the following actions: generating one or more updated test scenario templates based on one or more user-defined updated test cases, obtaining (e.g., creating, retrieving, extracting, etc.) one or more test package artifacts associated with one or more updated test scenario templates from a plurality of nodes (e.g., nodes 120-1 to 120-N) associated with the test automation system, and combining one or more updated test package artifacts with one or more test package artifacts to generate one or more updated test packages. According to an embodiment, at least one processor can determine one or more differences between one or more user-defined updated test cases and current user-defined test cases, and can update the test scenario templates by modifying one or more generated test scenario templates in a manner that reflects the one or more differences. At least one test environment configuration file can be updated by at least processor 430 by the same method.

[0122] Considering the above, an exemplary embodiment of the present disclosure provides a test automation system that automates the end-to-end process when managing tests for a user. Specifically, a user can provide only the intended test requirements (e.g., intended test cases, intended test plans, intended test cycles, etc.) to the test automation system, and the test automation system can automatically generate test packages and test environment configuration files based on this. The user only needs to remotely provide the test requirements (e.g., via network 130, etc.) from associated nodes (e.g., user devices, workstations, etc.) communicatively coupled to the test automation system, without physically moving to the test facility as shown in the related art.

[0123] Moreover, the test automation system can verify the generated test packages and / or the generated test environment configuration files before making them public and using them for actual tests. Thereby, the user can confirm that the generated test packages and the generated test environment configuration files are accurate, and the user can automatically perform actions based on the verified test packages and the verified test environment configuration files. Therefore, there is no need to manually monitor the state of the system during the test, nor to manually execute the test after verifying the generated test packages and the generated test environment configuration files.

[0124] Moreover, the test automation system can automatically allocate resources for executing tests according to the verified test packages and the verified test environment configuration files without the participation of the user. The test automation system can dynamically select a node having sufficient resources for executing tests, and then execute the test on that node. Then, the test automation system can automatically collect the test results and present them to the user.

[0125] For this purpose, a user can define a desired test requirement only from any suitable node communicatively coupled to the test automation system, and the test automation system can automatically manage the test on behalf of the user. Thus, the testing of software is managed in an efficient and effective manner, whereby the burden on the user is reduced and the software development period is shortened.

[0126] It should be understood that the specific order or hierarchy of the blocks in the process / flowchart disclosed in this specification is an example of an exemplary method. It should be understood that the specific order or hierarchy of the blocks in the process / flowchart can be reconfigured based on design preferences. Moreover, some blocks can be combined or omitted. The appended method claims present the elements of the various blocks in an exemplary order, but are not limited to the specific order or hierarchy presented.

[0127] Some embodiments may be related to systems, methods, and / or computer-readable media at any possible level of integration of any technology. Moreover, one or more of the above-described components may be stored on a computer-readable medium and implemented as instructions executable by at least one processor (and / or may include at least one processor). The computer-readable medium may include a computer-readable non-transitory storage medium having computer-readable program instructions for causing a processor to perform operations.

[0128] A computer-readable storage medium may be an entity device capable of holding and storing instructions for use by an instruction execution device. A computer-readable storage medium is not limited to, for example, the following media and may be an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the above. In a more specific example of a computer-readable storage medium, which does not include all, the following are included: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disk read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a punched card, or a mechanically encoded device such as a raised structure in a groove having instructions recorded thereon, and any suitable combination of the above. The computer-readable medium used herein should not be construed as a transient signal per se, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagated through a waveguide or other transmission medium (e.g., an optical pulse passing through an optical fiber cable), or an electrical signal transmitted through a wire.

[0129] The computer-readable program instructions described in this specification can be downloaded from a computer-readable storage medium to various computing / processing devices, or can be downloaded, for example, via networks such as the Internet, local area network, wide area network, and / or wireless network to an external computer or external storage device. The network can include copper transmission optical fiber, optical transmission optical fiber, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter or network interface in each computing / processing device receives the computer-readable program instructions from the network and transmits the computer-readable program instructions for storage in the computer-readable storage medium in each computing / processing device.

[0130] The computer-readable program code / instructions for performing an action can be assembly instructions, instruction-set-architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state-setting data, configuration data for an integrated circuit, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages such as Smalltalk, C++, and procedural programming languages such as the "C" programming language or similar programming languages. The computer-readable program instructions can be executed entirely on the user's computer, partially on the user's computer, executed as an independent software package partially on the user's computer, and partially on a remote computer, or can be executed entirely on a remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through any type of network including a local area network (LAN) or wide area network (WAN), or the connection can be made to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, for example, an electronic circuit including a programmable logic circuit, a field-programmable gate array (FPGA), or a programmable logic array (PLA) can be personalized using the state information of the computer-readable program instructions for executing a solution or operation, thereby enabling the execution of the computer-readable program instructions.

[0131] These computer-readable program instructions can be provided to create a machine in the processor of a general-purpose computer, special-purpose computer, or other programmable data processing device, such that the instructions executed via the processor of the computer or other programmable data processing device create a method for implementing the functions / actions specified in the blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium, which can instruct a computer, programmable data processing device, and / or other devices to function in a particular manner, such that the computer-readable storage medium with the stored instructions has a manufactured article including instructions for implementing the solution of the functions / actions specified in the blocks of the flowchart and / or block diagram.

[0132] The computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other devices to cause a series of operational steps to be performed on the computer, other programmable apparatus, or other devices to generate a computer-implemented process, such that the instructions executed on the computer, other programmable apparatus, or other devices implement the functions / actions specified in the boxes of the flowchart and / or block diagram.

[0133] The flowcharts and block diagrams in the figures illustrate the structure, functions, and operations of the possible implementations of systems, methods, and computer-readable media in various embodiments. In this regard, each box in the flowchart or block diagram may represent a module, segment, or portion of instructions that include one or more executable instructions for implementing the specified logical function. The methods, computer systems, and computer-readable media may include additional boxes, fewer boxes, different boxes, or boxes configured differently from those shown in Figure 1 The boxes shown in the figures. In some alternative implementations, the functions shown in the boxes may be performed in an order different from that shown in the figures. For example, two consecutive boxes shown may actually be performed simultaneously or substantially simultaneously, or sometimes the boxes may be performed in the reverse order depending on the functions involved. It should also be noted that each box in the examples of the block diagrams and / or flowcharts and combinations of boxes in the examples of the block diagrams and / or flowcharts may be implemented by a system based on special-purpose hardware that performs the specified functions or actions, implementing combinations of special-purpose hardware and computer instructions.

[0134] The systems and / or methods described in this specification may be implemented in different forms of hardware, firmware, or a combination of hardware and software. The actual specialized control hardware or software code for implementing these systems and / or methods is not limiting of the implementation. Thus, the operations and behaviors of the systems and / or methods are described in this specification without reference to specific software code, and it should be understood that software and hardware can be designed to implement the systems and / or methods based on the description in this specification.

Claims

1. A method for managing a test, the test being a test for testing software of an embedded system, the method being implemented by at least one processor, wherein: The method comprises: receiving at least one first user input associated with one or more test elements from a user; generating at least one test package based on the first user input; generating at least one test environment configuration file based on at least one of the test packages; Verifying at least one of the test packages and at least one of the test environment configuration files; Determining changes to the software; and Based on the determination of the change of the software, the test is performed according to at least one of the verified test packages and at least one of the verified test environment configuration files, The software of the embedded system includes an on-board electronic control unit ECU.

2. The method according to claim 1, further comprising: The at least one verified test package and the at least one verified test environment configuration file are stored in one or more storage media accessible to other users, thereby making the at least one verified test package and the at least one verified test environment configuration file public.

3. The method according to claim 1 or 2, wherein: The first user input includes information associated with at least one user-defined test case, Generating at least one of the test packages comprises: generating at least one test scenario template based on at least one of the user-defined test cases; obtaining one or more test package artifacts based on at least one test scenario template; and At least one of the test scenario templates is combined with one or more of the test package artifacts to generate at least one of the test packages.

4. The method according to any one of claims 1 to 3, wherein: Generating at least one of the test environment configuration files includes: receiving at least one second user input from the user associated with at least one user-defined test plan; generating at least one test plan template based on the second user input and at least one of the test packages; Generating at least one test plan by appending at least one of the test plan templates to at least one of the user-defined test cases; receiving at least one third user input from the user associated with at least one user-defined test cycle; and At least one of the test environment configuration files is generated based on the third user input and at least one of the test plans.

5. The method according to any one of claims 1 to 4, wherein: At least one of the test environment configuration files includes information defining settings for at least one test platform.

6. The method according to any one of claims 1 to 5, wherein: Verifying at least one of the test packages and at least one of the test environment configuration files includes: Performing a preliminary test of the software based on at least one of the test packages and at least one of the test environment configuration files; presenting the results of the pre-test to the user; receiving at least one fourth user input from the user associated with one of approval of the results of the pre-test and rejection of the results of the pre-test; Based on the determination that the fourth user input is associated with the approval, determining that at least one of the test packages and at least one of the test environment configuration files are valid; and Based on the determination that the fourth user input is associated with the rejection, it is determined that at least one of the test packages and at least one of the test environment configuration files are invalid.

7. The method according to any one of claims 1 to 6, wherein: Determining the changes to the software includes: Obtaining the current status of the software; and The current state is compared with the last known state of the software to determine whether the software has been changed.

8. The method according to any one of claims 1 to 7, wherein: The changes to the software include destructive changes that satisfy one or more conditions defined in at least one of the test packages.

9. The method according to any one of claims 1 to 8, wherein: Execution of the test involves: generating at least one test platform associated with at least one test environment for testing the software based on at least one of the verified test environment configuration files; selecting, based on at least one test platform, at least one node associated with at least one test environment defined in at least one of the test platforms; Deploy the software on the selected at least one node; and A test for testing the software at the selected at least one node is performed based on the verified at least one test package.

10. The method according to any one of claims 1 to 9, further comprising: collecting one or more test results associated with the test; generating at least one graphical user interface GUI including one or more of the test results; as well as At least one of the GUIs is presented to the user.

11. A system for managing a test, the test being a test for testing software of an embedded system, wherein: The system has: a memory storing instructions; and At least one processor is configured to execute the instructions for performing the following actions: receiving at least one first user input associated with one or more test elements from a user, generating at least one test package based on the first user input, generating at least one test environment configuration file based on at least one of the test packages, verifying at least one of the test packages and at least one of the test environment configuration files, Determine changes to the software, Based on the determination of the change of the software, the test is performed according to at least one of the verified test packages and at least one of the verified test environment configuration files, Wherein, the software of the embedded system includes an on-board electronic control unit ECU.

12. The system according to claim 11, wherein: At least one of the processors is further configured to execute the instructions to: The at least one verified test package and the at least one verified test environment configuration file are stored in one or more storage media accessible to other users, thereby making the at least one verified test package and the at least one verified test environment configuration file public.

13. The system according to claim 11 or 12, wherein: The first user input includes information associated with at least one user-defined test case, At least one of the processors is further configured to execute the instructions for generating at least one of the test packages by: generating at least one test scenario template based on at least one of the user-defined test cases, obtaining one or more test package artifacts based on at least one test scenario template, At least one of the test scenario templates is combined with one or more of the test package artifacts to generate at least one of the test packages.

14. The system according to any one of claims 11 to 13, wherein: At least one of the processors is further configured to execute the instructions for generating at least one of the test environment configuration files by performing the following actions: receiving at least one second user input associated with at least one user-defined test plan from the user, generating at least one test plan template based on the second user input and at least one of the test packages, generating at least one test plan by appending at least one of the test plan templates to at least one of the user-defined test cases, receiving at least one third user input from the user associated with at least one user-defined test cycle, At least one of the test environment configuration files is generated based on the third user input and at least one of the test plans.

15. The system according to any one of claims 11 to 14, wherein: At least one of the test environment configuration files includes information defining settings for at least one test platform.

16. The system according to any one of claims 11 to 15, wherein: At least one of the processors is further configured to execute the instructions for validating at least one of the test packages and at least one of the test environment configuration files by performing the following actions: performing a preliminary test of the software based on at least one of the test packages and at least one of the test environment configuration files, presenting the results of the pre-test to the user, receiving at least one fourth user input from the user associated with one of approval of the results of the pre-test and rejection of the results of the pre-test, Based on the determination that the fourth user input is associated with the approval, it is determined that at least one of the test packages and at least one of the test environment configuration files are valid, Based on the determination that the fourth user input is associated with the rejection, it is determined that at least one of the test packages and at least one of the test environment configuration files are invalid.

17. The system according to any one of claims 11 to 16, wherein: At least one of the processors is further configured to execute the instructions for determining a change in the software by performing the following actions: Get the current status of the software, The current state is compared with the last known state of the software to determine whether the software has been changed.

18. The system according to any one of claims 11 to 17, wherein: The changes to the software include destructive changes that satisfy one or more conditions defined in at least one of the test packages.

19. The system according to any one of claims 11 to 18, wherein: At least one of the processors is further configured to execute the instructions for performing the test by: generating at least one test platform associated with at least one test environment for testing the software based on at least one of the verified test environment configuration files, selecting, based on at least one test platform, at least one node associated with at least one test environment defined in at least one of the test platforms, deploying the software on the selected at least one node, A test for testing the software at the selected at least one node is performed based on the verified at least one test package.

20. The system according to any one of claims 11 to 19, wherein: At least one of the processors is further configured to execute the instructions to: collecting one or more test results associated with the test, generating at least one graphical user interface GUI including one or more of said test results, At least one of the GUIs is presented to the user.

Images