System and method for managing tests for testing embedded systems in a vehicle

Abstract

To provide a system, method, and device for managing a test for testing software of an embedded system.SOLUTION: A method may be implemented by at least one processor and may include the steps of: receiving, from a user, at least one first user input associated with one or more test requirements; generating, based on the first user input, at least one test package; generating, based on the at least one test package, at least one test environment configuration file; validating the at least one test package and the at least one test environment configuration file; determining a change in the software; and based on determining the change in the software, executing a test according to the at least one validated test package and the at least one validated test environment configuration file. The software of the embedded system may include an on-vehicle electronic control unit (ECU).SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] Systems and methods consistent with exemplary embodiments of the present disclosure relate to test management, and more particularly, to managing testing of software associated with embedded systems in vehicles. [Background technology]

[0002] Software testing is required to ensure that software functions as intended, meets specified requirements, and performs reliably in a variety of scenarios. Software testing is an important part of the Software Development Lifecycle (SDLC) and is performed to identify defects, errors, or bugs in the software before it is deployed into live systems.

[0003] When software contains complex functionality and / or is required to interoperate with other software, software testing can become complex and involve multiple steps and users / stakeholders. For example, in the development of vehicle-related features in vehicle systems such as lane change assist or mobile smart key, multiple electronic control units (ECUs) may be developed and interoperate with each other to perform the intended functions.

[0004] In this regard, each of the ECUs may be managed by a different user and / or located in a different geographic location. For example, a first ECU may be developed by a first developer (e.g., an in-house development engineer at an automobile manufacturer) located at a first location, and the first ECU may need to interoperate with a second ECU developed by a second developer (e.g., a vendor) located at a second location. As another example, the first ECU may be tested by a third user (e.g., a test engineer) located at a third location. Furthermore, the first ECU may interoperate with hardware (e.g., a physical ECU, etc.) located at a fourth location, and thus testing of the first ECU requires hardware involvement.

[0005] Thus, in the prior art, when a user wants to test software (e.g., a software-based ECU), the user needs to visit a physical test facility where the software and associated test components (e.g., software components such as software-based ECUs developed by other users, hardware components such as physical ECUs, etc.) are collectively deployed. Therefore, testing the software can be time-consuming and burdensome for the user because the user may need to physically visit the test facility to set up and run the tests.

[0006] Additionally, conventional techniques can make it difficult for users to set up, edit, and adjust tests at a test facility, especially when previously untested new features or software need to be tested with complex test cases, because the test components available at the test facility are generic and limited, and the associated test packages are limited and may not be able to meet the intended test requirements or test cases.

[0007] Furthermore, because test setup can be complex and time-consuming, tests are often run once the setup procedure is complete without verifying the accuracy of the test setup. This can lead to inaccurate test results, especially if the user setting up the test is inexperienced and prone to making mistakes when setting up the test. Furthermore, even if an error in the test setup is discovered during the test, the user may not be able to quickly correct the error on the spot.

[0008] Furthermore, in the prior art, a user may not be able to access information about test components available at a test facility before visiting the test facility. For example, a vendor may not be able to obtain information about available test packages and / or associated limitations / constraints at a test facility managed by an automobile manufacturer, making it extremely difficult for the vendor to accurately formulate test requirements or test plans without physically visiting the test facility.

[0009] Furthermore, in the prior art, most (if not all) parts of the end-to-end process of testing, such as defining test requirements, determining an appropriate test package based on the test requirements, selecting and configuring an appropriate test environment, collecting relevant test components, deploying the software to be tested and the relevant test components to the test environment, running tests on the software based on the determined test package, obtaining test results, and reproducing the test results, are performed separately and managed manually by different users and / or on different systems, which is inefficient, burdensome, and prone to human error.

[0010] In view of the above, traditional software development is time consuming, burdensome to users, and can be delayed due to uncertainties in test management. Summary of the Invention

[0011] According to embodiments, methods and systems are provided for efficiently and effectively managing one or more tests for testing one or more software of an embedded system of a vehicle. For example, exemplary embodiments of the present disclosure provide methods and systems for handling the end-to-end process of test management, such as collecting user input, providing a test package, providing a test environment configuration file, validating the test package and / or the test environment configuration file, and running one or more tests.

[0012] According to an embodiment, a method for managing tests for testing software of an embedded system is provided. The method can be implemented on at least one processor and can include receiving at least one first user input from a user related to one or more test requirements, generating at least one test package based on the first user input, generating at least one test environment configuration file based on the at least one test package, validating the at least one test package and the at least one test environment configuration file, determining software changes, and running tests in accordance with the validated at least one test package and the validated at least one test environment configuration file based on determining the software changes. The software of the embedded system can include an on-board electronic control unit (ECU).

[0013] According to an embodiment, the method may further include publishing the validated at least one test package and the validated at least one test environment configuration file by storing the validated at least one test package and the validated at least one test environment configuration file on one or more storage media accessible by other users. The method may also include collecting one or more test results associated with the tests, generating at least one graphical user interface (GUI) including the one or more test results, and presenting the at least one GUI to a user.

[0014] According to an embodiment, the first user input may include information related to at least one user-defined test case, and generating the at least one test package may include generating at least one test scenario template based on the at least one user-defined test case, obtaining one or more test package artifacts based on the at least one test scenario template, and combining the at least one test scenario template with the one or more test package artifacts to generate the at least one test package.

[0015] According to an embodiment, generating the at least one test environment configuration file may include receiving at least one second user input from a user related to at least one user-defined test plan, generating at least one test plan template based on the second user input and the at least one test package, generating the at least one test plan by adding at least one user-defined test case to the at least one test plan template, receiving at least one third user input from a user related to at least one user-defined test cycle, and generating the at least one test environment configuration file based on the third user input and the at least one test plan. The at least one test environment configuration file may include information defining a configuration of at least one test bench.

[0016] According to an embodiment, validating at least one test package and at least one test environment configuration file may include: conducting a pre-test of software based on the at least one test package and the at least one test environment configuration file; presenting results of the pre-test to a user; receiving at least one fourth user input from the user associated with one of an approval of the results of the pre-test and a rejection of the results of the pre-test; determining that the at least one test package and the at least one test environment configuration file are valid based on determining that the fourth user input is associated with the approval; and determining that the at least one test package and the at least one test environment configuration file are invalid based on determining that the fourth user input is associated with the rejection.

[0017] According to an embodiment, determining a change in the software can include obtaining a current status of the software and comparing the current status to a last known status of the software to determine whether a change has occurred in the software. The change in the software can include a breaking change that meets one or more conditions defined in at least one test suite.

[0018] According to an embodiment, executing the test may include generating at least one test bench associated with at least one test environment for testing the software based on the validated at least one test environment configuration file; selecting at least one node associated with the at least one test environment defined in the at least one test bench based on the at least one test bench; deploying the software to the selected at least one node; and executing a test to test the software on the selected at least one node based on the validated at least one test package.

[0019] According to an embodiment, a system for managing tests for testing software of an embedded system is provided. The system can include a storage device that stores instructions and at least one processor configured to execute the instructions to receive at least one first user input from a user related to one or more test requirements, generate at least one test package based on the first user input, generate at least one test environment configuration file based on the at least one test package, validate the at least one test package and the at least one test environment configuration file, determine software changes, and run tests in accordance with the validated at least one test package and the validated at least one test environment configuration file based on the determined software changes. The software of the embedded system can include an on-board electronic control unit (ECU).

[0020] According to an embodiment, the at least one processor may be further configured to execute instructions for publishing the validated at least one test package and the validated at least one test environment configuration file by storing the validated at least one test package and the validated at least one test environment configuration file on one or more storage media accessible by other users. Additionally, the at least one processor may be further configured to execute instructions for collecting one or more test results associated with the tests, generating at least one graphical user interface (GUI) including the one or more test results, and presenting the at least one GUI to a user.

[0021] According to an embodiment, the first user input includes information related to at least one user-defined test case, and the at least one processor may be further configured to execute instructions for generating the at least one test package by generating at least one test scenario template based on the at least one user-defined test case, obtaining one or more test package artifacts based on the at least one test scenario template, and combining the at least one test scenario template with the one or more test package artifacts to generate the at least one test package.

[0022] According to an embodiment, the at least one processor may be further configured to execute instructions for generating the at least one test environment configuration file by receiving at least one second user input from a user related to the at least one user-defined test plan, generating at least one test plan template based on the second user input and the at least one test package, generating the at least one test plan by adding the at least one test plan template to the at least one user-defined test case, and receiving at least one third user input from the user related to the at least one user-defined test cycle, and generating at least one test environment configuration file based on the third user input and the at least one test plan. The at least one test environment configuration file may include information defining a configuration of the at least one test bench.

[0023] According to an embodiment, the at least one processor may be further configured to execute instructions to validate the at least one test package and the at least one test environment configuration file by: conducting a pre-test of the software based on the at least one test package and the at least one test environment configuration file; presenting results of the pre-test to a user; receiving at least one fourth user input from the user associated with one of an approval of the results of the pre-test and a rejection of the results of the pre-test; determining that the at least one test package and the at least one test environment configuration file are valid based on determining that the fourth user input is associated with the approval; and determining that the at least one test package and the at least one test environment configuration file are invalid based on determining that the fourth user input is associated with the rejection.

[0024] According to an embodiment, the at least one processor may be further configured to execute instructions for determining a change in the software by obtaining a current status of the software and comparing the current status with a last known status of the software to determine whether a change has occurred in the software. The change in the software may include a breaking change that meets one or more conditions defined in the at least one test suite.

[0025] According to an embodiment, the at least one processor may be further configured to execute instructions for executing a test by: generating, based on the validated at least one test environment configuration file, at least one test bench associated with at least one test environment for testing software; selecting, based on the at least one test bench, at least one node associated with the at least one test environment defined in the at least one test bench; deploying the selected at least one node software; and conducting a test for testing the selected at least one node software based on the validated at least one test package.

[0026] Additional aspects will be set forth in part in the description that follows, and in part will be obvious from the description, or may be realized by practice of the presented embodiments of the present disclosure. [Brief explanation of the drawings]

[0027] The features, advantages, and benefits of exemplary embodiments of the present disclosure will be described below with reference to the accompanying drawings, in which like numerals refer to like elements.

[0028] [Figure 1] FIG. 1 is a block diagram of an exemplary system configuration for managing tests, according to one or more embodiments.

[0029] [Figure 2]FIG. 2 is a block diagram of example modules of a test automation system according to one or more embodiments.

[0030] [Figure 3A] FIG. 3A is a flow diagram of an example use case for preparing a test, according to one or more embodiments.

[0031] [Figure 3B] FIG. 3B is a flow diagram of an exemplary use case for test execution, according to one or more embodiments.

[0032] [Figure 4] FIG. 4 is a block diagram of example components of a test automation system according to one or more embodiments. DETAILED DESCRIPTION OF THE INVENTION

[0033] The following detailed description of example embodiments refers to the accompanying drawings. The foregoing disclosure provides illustration and description, but is not intended to be exhaustive or to limit the embodiments to the precise form disclosed. Modifications and variations are possible in light of the above disclosure or may be acquired by practice of the embodiments. Furthermore, one or more features or components of one or more embodiments may be incorporated into or combined with other embodiments (or one or more features of other embodiments). Additionally, in the descriptions of operations provided below, it is understood that one or more operations may be omitted, one or more operations may be added, one or more operations may be performed concurrently (at least in part), and the order of one or more operations may be switched.

[0034] Although particular combinations of features are recited in the claims and / or disclosed herein, such combinations are not intended to limit the disclosure of possible implementations. Indeed, many of the features may be combined in ways not specifically recited in the claims and / or disclosed herein. Although each dependent claim listed below may depend directly on only one claim, the disclosure of possible implementations includes each dependent claim in combination with all other claims in the claim set.

[0035] No element, act, or instruction used herein should be construed as critical or essential unless expressly stated as such. Also, as used herein, the articles "a" and "an" are intended to include one or more items and can be used interchangeably with "one or more." When only one item is intended, the term "a" or similar language is used. Also, as used herein, terms such as "have," "having," "include," "including," and the like are intended to be open-ended and unrestricted. Furthermore, the phrase "based on" is intended to mean "based at least in part on," unless expressly stated otherwise. Furthermore, phrases such as "at least one of [A] and [B]" or "at least one of [A] or [B]" should be understood to include only A, only B, or both A and B.

[0036] References throughout this specification to "one embodiment," "an embodiment," "a non-limiting preferred embodiment," or similar terms mean that a particular feature, structure, or characteristic described in connection with the illustrated embodiment is included in at least one embodiment of the solution. Thus, the phrases "in one embodiment," "in an embodiment," "a non-limiting preferred embodiment," and similar terms throughout this specification may, but do not necessarily, all refer to the same embodiment.

[0037] Furthermore, the described features, advantages, and characteristics of the present disclosure may be combined in any suitable manner in one or more embodiments. Those skilled in the art will recognize, in light of the description herein, that the present disclosure may be practiced without one or more of the specific features or advantages of a particular embodiment. In other instances, additional features and advantages may be recognized in certain embodiments that may not be present in all embodiments of the present disclosure.

[0038] Additionally, terms such as "vehicle," as used herein, may refer to any motorized and / or mechanical machine capable of carrying or transporting people and / or cargo, such as an automobile, truck, motorcycle, bus, bicycle, mobility scooter, etc.

[0039] Exemplary embodiments consistent with the present disclosure provide methods, systems, and apparatuses for managing tests for testing one or more software components of an embedded system, such as an automotive ECU. Specifically, the exemplary embodiment methods, systems, apparatuses, etc., can automatically manage tests according to user-defined requirements. According to embodiments, the exemplary embodiment methods, systems, apparatuses, etc., can automatically generate one or more test packages and one or more test environment configuration files based on one or more user inputs, and can automatically execute tests accordingly.

[0040] In some embodiments, the generated test package(s) and / or the generated test environment configuration file(s) may be validated before being utilized to execute tests. Once the generated test package(s) and / or the generated test environment configuration file(s) are validated, the method, system, apparatus, etc. of the exemplary embodiments may automatically determine software changes therefrom and automatically execute tests based on the determined software changes. Furthermore, the validated test package(s) and / or the validated test environment configuration file(s) may be made public and accessible to other users.

[0041] According to embodiments, upon executing tests on the software, the exemplary embodiment methods, systems, apparatus, etc. may automatically collect test results and present them to an associated user. Additionally, the exemplary embodiment methods, systems, apparatus, etc. may receive one or more user inputs to update one or more generated test packages and / or one or more generated test environment configuration files and automatically update them accordingly.

[0042] Thus, the exemplary embodiment methods, systems, apparatus, etc. can automatically generate or update one or more test components according to user input based on real-time or near-real-time status and test requirements, resulting in one or more test benches being provided and available on demand without geographical limitations.

[0043] To this end, exemplary embodiments of the present disclosure can automatically manage the end-to-end testing process according to one or more user-defined requirements. A user can remotely define one or more test requirements without having to physically travel to a test facility, as is required in the prior art. Ultimately, exemplary embodiments of the present disclosure can enable more efficient software development, significantly reducing user burden, development time, and the cost and effort of planning visits and travel to test facilities.

[0044] The features, advantages, and significance of the above exemplary embodiments are intended to be only a part of the present disclosure and are not intended to be exhaustive or to limit the technical scope of the present disclosure. Further descriptions of the features, components, configurations, operations, and implementations of the exemplary embodiments of the present disclosure, as well as the technical advantages and significance thereof, are provided below.

[0045] 1 is a block diagram of an exemplary system configuration 100 for managing tests, according to one or more embodiments. As shown in FIG. 1, system configuration 100 may include a test automation system 110, multiple nodes 120-1 through 120-N, and a network 130.

[0046] Generally, test automation system 110 may be communicatively coupled to multiple nodes 120-1 through 120-N via network 130 and configured to interoperate with and manage tests (or one or more related information or data) with multiple nodes 120-1 through 120-N. A description of example components that may be included in test automation system 110, as well as descriptions of associated use cases, is provided below with reference to FIGS. 2 through 4.

[0047] Each of the plurality of nodes 120-1 to 120-N may include one or more devices, equipment, systems, or any other suitable components that may receive, host, store, utilize, deploy, process, provide, etc., one or more artifacts or components that constitute a test.

[0048] For example, node 120-1 may include a device or equipment (e.g., a personal computer, a server or server cluster, a workstation, etc.) that may be utilized to build, store, execute, or simulate one or more computer-executable software applications, such as one or more virtual ECUs, one or more emulated ECUs, and / or any other suitable software-based components of a vehicle system (e.g., a vehicle model, a data communications module (DCM) model, a heating, ventilation, and air conditioning (HVAC) model, etc.) As another example, node 120-1 may include one or more hardware components, such as one or more fully developed physical ECUs, one or more partially developed physical ECUs, one or more vehicle hardware (e.g., a powertrain, an engine, etc.).

[0049] According to an embodiment, one or more of the plurality of nodes 120-1 through 120-N may include one or more interfaces, each of which may be configured to communicatively couple the associated node to test automation system 110. For example, one or more of the plurality of nodes may include a hardware interface, a software interface (e.g., a program interface, an application program interface (API), etc.), etc.

[0050] According to an embodiment, at least some of the plurality of nodes 120-1 through 120-N are geographically located in a different location than test automation system 110 and / or in a different location than another portion of the plurality of nodes. According to an embodiment, at least some of the plurality of nodes 120-1 through 120-N are associated with users that are geographically located in different locations.

[0051] For example, node 120-1 may be associated with a first user (e.g., a development engineer) responsible for developing a first ECU, and node 120-2 may be associated with a second user (e.g., a test engineer) responsible for testing the first ECU, where the first user may be located at a first location and the second user may be located at a second location, and the first location may be different from the second location. Alternatively or additionally, first node 120-1 may be associated with a first user responsible for developing the first ECU, and node 120-2 may be associated with a second user responsible for developing a second ECU, where the first ECU may interact with the second ECU, where the first user may be located at a first location and the second user may be located at a second location, and the first location may be different from the second location.

[0052] According to an embodiment, at least a portion of the plurality of nodes 120-1 through 120-N may be associated with one or more test environments. For example, the portion of nodes may have, be communicatively coupled to (e.g., wired, wireless, etc.) or deployed to at least one software-based test environment (e.g., a software-in-the-loop (SIL) test environment, a virtual ECU (V-ECU) test environment, a model-in-the-loop (MIL) test environment, a processor-in-the-loop (PIL) test environment, etc.) and / or at least one hardware-based test environment (e.g., a hardware-in-the-loop (HIL) test environment).

[0053] Additionally, at least some of the plurality of nodes 120-1 through 120-N may include one or more storage media, such as a server or server cluster, that may be configured to store, publish, etc., one or more data or information (or information related thereto) provided by test automation system 110 and / or other portions of the plurality of nodes 120-1 through 120-N.

[0054] Network 130 may include one or more wired and / or wireless networks that may be configured to couple multiple nodes 120-1 through 120-N to test automation system 110. For example, network 130 may include a cellular network (e.g., a fifth-generation (5G) network, a long-term evolution (LTE) network, a third-generation (3G) network, a code division multiple access (CDMA) network, etc.), a public land mobile network (PLMN), a local area network (LAN), a wide area network (WAN), a metropolitan area network (MAN), a telephone network (e.g., a public switched telephone network (PSTN)), a private network, an ad hoc network, an intranet, the Internet, an optical fiber-based network, etc., and / or combinations of these or other types of networks.

[0055] According to an embodiment, network 130 may include a virtual network that may include one or more physical network components (e.g., Ethernet, WiFi modules, telecommunications network hardware, etc.) on which one or more virtual network functions (e.g., Controller Area Network (CAN) bus, etc.) are implemented. Additionally or alternatively, network 130 may include at least one parameter network.

[0056] Reference is now made to Figure 2, which illustrates a block diagram of exemplary modules of a test automation system 200, according to one or more embodiments. Test bench management system 200 may correspond to test automation system 110 described above with reference to Figure 1, and therefore features described herein with reference to systems 110 and 200 may be applicable to one another unless expressly stated otherwise.

[0057] As shown in FIG. 2 , test automation system 200 may include at least one user interaction module 210, at least one testing framework module 220, at least one configuration management module 230, at least one test bench management module 240, and at least one test management module 250. As described further below, one or more of modules 210-250 may be defined by computer-executable instructions or programming code that, when executed by one or more processors (e.g., processor 430 described below with reference to FIG. 4 ), causes the one or more processors to perform operations associated therewith. Alternatively or additionally, one or more of modules 210-250 may include one or more hardware components (e.g., components described below with reference to FIG. 4 ) that may be configured to perform operations associated therewith. To this end, modules 210-250 may be configured to interoperate with one another to manage one or more tests, as described below.

[0058] User interaction module 210 (sometimes referred to herein as "module 210") may be configured to interact with one or more users. According to an embodiment, module 210 may generate one or more graphical user interfaces (GUIs) to engage one or more users. For example, module 210 may generate one or more GUIs for a user and may receive one or more user inputs from the user.

[0059] According to an embodiment, module 210 may be configured to receive one or more user inputs from one or more users, which may define one or more user-intended test requirements, such as user-intended test cases, user-intended test plans, user-intended test cycles, and / or the like. As described further below, the one or more user inputs may be utilized in generating one or more test packages and / or one or more test environment configuration files. Additionally, module 210 may also be configured to similarly receive one or more inputs from one or more users, including information for updating one or more generated test components.

[0060] According to an embodiment, module 210 can generate one or more GUIs to present the status of the tests to a user. For example, module 210 may present the status of configured tests (e.g., pending, executed, failed, etc.), the results of validation tests, the results of actual tests, etc. in one or more GUIs.

[0061] Test framework module 220 (sometimes referred to herein as “module 220”) may be configured to generate one or more test packages and / or one or more test environment configuration files. According to an embodiment, module 220 may be configured to obtain one or more user inputs from module 210 and to generate one or more test packages and / or one or more test environment configuration files based on the obtained one or more user inputs.

[0062] For example, module 220 may generate one or more test packages based on one or more user inputs and generate one or more test environment configuration files based on the one or more test packages. In some embodiments, the one or more user inputs may include information related to one or more user-defined test cases, and module 220 may generate one or more test packages by creating one or more test scenario templates based on the one or more user-defined test cases, obtaining (e.g., creating, searching, retrieving, etc.) one or more test package artifacts associated with the one or more test scenario templates from among multiple nodes associated with the test automation system (e.g., nodes 120-1 through 120-N), and combining the one or more test scenario templates with the one or more test package artifacts to generate the one or more test packages.

[0063] In this regard, the term "test case" as used herein may refer to a specific set of conditions and / or steps designed to verify the functionality or behavior of software (or a system under test). A test case may define one or more actions and expected results for a particular test scenario. According to an embodiment, a test case may include one or more of: a test case ID (e.g., a unique identifier or sequence of numbers assigned to a test case for tracking and reference purposes); a test objective (e.g., a description or specification of the goal or objective of the test case); at least one test condition (e.g., prerequisites or initial conditions required to execute the test case, specific setup or configuration required to trigger test execution, etc.); at least one test step (e.g., an action or operation performed to execute the test case, required inputs or interactions with the software, etc.); at least one expected result (e.g., an expected result or behavior of the software when the test case is successfully executed, etc.); and any other suitable information.

[0064] Additionally, as used herein, the term "test package" may refer to a collection of test cases, test scripts / algorithms, test data, test dependencies, test execution instructions, test cycles, and other related resources collectively organized for the purpose of testing specific software (e.g., a specific ECU). By generating and providing a "test package" based on user-defined test requirements (contained in first user input), module 220 can provide a structured approach for planning and executing tests in the manner intended by the user, while ensuring that all necessary test activities are included and properly documented.

[0065] Additionally, as used herein, the term "test environment configuration file" may refer to a file or document that contains information for obtaining (e.g., obtaining, creating, updating, etc.) and configuring at least one test environment associated with the software or system under test. For example, a test environment configuration file may include information related to the configuration of one or more test environments needed to test the software (e.g., the configuration of a software-based test environment needed to test the software and associated software-based components, the configuration of a hardware-based test environment needed to test hardware associated with the software, etc.), information related to test plans, information related to test scenarios, information related to test execution conditions, and information related to test bench configuration.

[0066] As used herein, the term "test scenario" may refer to a collection or sequence of related test cases grouped together to achieve a common testing goal. Each test case within a test scenario focuses on a particular aspect or condition, and the test scenario provides a larger context or flow.

[0067] As used herein, the term "test plan" may refer to a collection of information that outlines the approach, objectives, scope, procedures, and structure for testing software. A test plan may include a collection or sequence of test scenarios, test execution procedures, and configuration information such as test bench configuration and ECU test configuration.

[0068] As used herein, the term "testbench" may refer to a set of tools, procedures, functional compositions, equipment, and other information or parameters that, when compiled or utilized, allow a test to be executed under desired conditions or configurations. Simply put, a testbench defines a test environment for testing software.

[0069] Additionally or alternatively, the one or more user inputs may include information related to one or more user-defined test plans and information related to one or more user-defined test cycles, and module 220 may generate the one or more test environment configuration files by generating one or more test plan templates based on the one or more user-defined test plans and one or more test packages, generating the one or more test plans by adding one or more user-defined test cases to the one or more test plan templates, and generating one or more test environment configuration files based on the one or more user-defined test cycles and the one or more test plans.

[0070] Additionally or alternatively, module 220 can update one or more generated test packages based on one or more user inputs. In some implementations, the one or more user inputs may include information related to one or more user-defined updated test cases, and module 220 may update the one or more generated test packages by generating one or more updated test scenario templates based on the one or more user-defined updated test cases, obtaining (e.g., creating, searching, retrieving, etc.) one or more test package artifacts associated with the one or more updated test scenario templates from among multiple nodes associated with the test automation system (e.g., nodes 120-1 through 120-N), and combining the one or more updated test package artifacts with one or more test package artifacts to generate one or more updated test packages. According to some embodiments, module 220 may determine one or more differences between the one or more user-defined updated test cases and the current user-defined test cases, and may generate one or more test scenario templates by modifying the one or more generated test scenario templates to reflect the one or more differences.

[0071] According to an embodiment, module 220 can interact with configuration management module 230, such as storing or committing one or more generated data (e.g., test packages, test environment configuration files, updated test packages, etc.) to configuration management module 230, receiving triggers for running tests from configuration management module 230, and providing one or more test results, test logs, test traces, etc. (received from test management module 250) to configuration management module 230.

[0072] According to an embodiment, module 220 can interact with test bench management module 240, such as by providing information or data to test bench management module 240 to initialize test bench provision. According to an embodiment, module 220 can interact with test management module 250, such as by providing one or more generation / validation data (e.g., test packages, etc.) to test management module 250, and by receiving one or more test results, test logs, test traces, etc. from test management module 250.

[0073] At least one configuration management module 230 (sometimes referred to herein as "module 230") may be configured to manage information or configurations related to tests. For example, module 230 may receive one or more generated data (e.g., test packages, test environment configuration files, updated test packages, etc.) from module 220 and then organize and store the one or more generated data therein. According to an embodiment, module 230 may include one or more storage media (e.g., storage 420, hosting servers, server clusters, etc., described below with reference to FIG. 4).

[0074] Additionally, module 230 may also be configured to receive and store therein, from one or more nodes associated with test automation system 200 (e.g., nodes 120-1 through 120-N), one or more systems to be tested (e.g., software to be tested, such as software-based ECUs), as well as one or more components associated therewith (e.g., one or more software interacting with the system to be tested, such as one or more software-based ECUs, one or more vehicle-related models, information on hardware associated with the system to be tested).

[0075] According to an embodiment, module 230 may be configured to continuously (or periodically) determine changes in the system under test. For example, module 230 may obtain a current status of the system under test based on a test cycle defined in at least one test package for the system under test and compare the current status with the last known status of the system under test to determine whether a change occurs in the system under test. According to an embodiment, a change in the system under test includes a breaking change that satisfies one or more conditions defined in at least one test package. Therefore, upon determining a change in the system under test, module 230 may generate and send a message (e.g., a notification message) to module 220 to trigger test execution.

[0076] Test bench management module 240 (sometimes referred to herein as “module 240”) may be configured to provide one or more test benches for testing. For example, module 240 may receive information or data, such as a test environment configuration file (containing test bench setting information), from module 220 each time a test plan execution is triggered, and may obtain (e.g., retrieve, create, update, etc.) one or more test benches based on the received information. According to an embodiment, module 240 may provide one or more test benches to test management module 250.

[0077] Test management module 250 (sometimes referred to herein as "module 250") may be configured to execute one or more tests. For example, module 250 may receive one or more test benches from module 240 and may initialize one or more test environments (e.g., software-based environments and / or hardware-based environments) according to the one or more test benches. Furthermore, module 250 may receive one or more test packages from module 220 and may execute one or more tests based on the one or more initialized test environments according to the one or more test packages.

[0078] To this end, modules within test automation system 200 can be configured to interoperate with each other to facilitate end-to-end automation in managing a user's tests. A user can simply provide intended test requirements (e.g., intended test cases, intended test plans, intended test cycles, etc.) to test automation system 200, which can automatically generate requested configuration data (e.g., test packages, test environment configuration files, test benches, etc.) based on the user-defined test requirements, automatically monitor the state of the system under test (e.g., software), automatically trigger test execution, and automatically collect and present test results to the user.

[0079] Exemplary use cases for modules within test automation system 200 according to one or more embodiments are described below with reference to Figures 3A and 3B. One or more of modules 210-250 shown in Figures 3A and 3B are similar to modules 210-25 described herein above with reference to Figure 2.

[0080] Reference is first made to Figure 3A, which illustrates a flow diagram of an exemplary use case for preparing a test, according to one or more embodiments. As shown in Figure 3A, at operation S3101, module 210 may be configured to receive at least one first user input from a user (via a node associated with the user). The first user input may be associated with one or more test requirements and may include information associated with at least one user-defined test case.

[0081] At operation S3102, module 210 may provide the first user input (received at operation S3101) to module 220. Accordingly, at operation S3103, module 220 may be configured to generate at least one test package based on the first user input. For example, module 220 may be configured to generate at least one test scenario template based on the at least one user-defined test case (included in the first user input) and retrieve one or more test package artifacts based on the at least one test scenario template from multiple nodes associated with test automation system 200 (e.g., nodes 120-1 through 120-N, etc.). Accordingly, module 220 may combine the at least one test scenario template with the one or more test package artifacts to generate at least one test package.

[0082] 3A , upon generating the at least one test package, module 220 may provide the generated test package to module 230 for storage at operation S3104. Subsequently, at operation S3105, module 210 may be configured to receive at least one second user input from a user. The at least one second user input may be associated with at least one user-defined test plan.

[0083] Thus, in operation S3106, module 210 may provide a second user input to module 220. Then, in operation S3107, module 220 may generate at least one test plan based on the second user input. For example, module 220 may generate at least one test plan template based on the second user input and the at least one test package (generated in operation S3103), and may generate the at least one test plan by adding at least one user-defined test case to the at least one test plan template.

[0084] Further, in operation S3108, module 210 may be configured to receive at least one third user input from a user. The at least one third user input may be associated with at least one user-defined test cycle that defines how often the system (e.g., module 230) should determine whether to run a test. Accordingly, in operation S3109, module 210 may provide the third user input to module 220. Next, in operation S3110, module 220 may be configured to generate at least one test environment configuration file. The test environment configuration file may include one or more test environment configurations required to test the system under test, information related to a test plan, information related to a test scenario, information related to test execution conditions, and / or information related to a test bench configuration.

[0085] Upon generating the test environment configuration file in operation S3111, module 220 may provide the same to module 230 for storage. In some implementations, before providing the test package and / or test environment configuration file to module 230, module 220 may validate the test package and / or test environment configuration file to ensure that the associated content is accurate and meets the user's requirements.

[0086] According to an embodiment, module 220 can validate the test package and / or the test environment configuration file by running a pre-test on the system under test based on the test package and / or the test environment configuration file, presenting the results of the pre-test to a user, and receiving at least one fourth user input from the user associated with one of an approval for the results of the pre-test and a rejection for the results of the pre-test. Thus, based on determining that the fourth user input is associated with an approval, module 220 can determine that the test and / or the test environment configuration file is valid. Otherwise, based on determining that the fourth user input is associated with a rejection, module 220 can determine that the test and / or the test environment configuration file is invalid. The pre-test can be run on a first node, which can be different from the node (e.g., a second node) on which the actual test will be run. In this manner, the test automation system can ensure the validity of the test package and / or the test environment configuration file before exposing the same to module 230.

[0087] To this end, once test preparation is complete, module 230 may be configured to continuously (or periodically) monitor the status of the system under test (e.g., according to a user-defined test cycle) and automatically trigger test execution when necessary, a description of an exemplary use case relating thereto is provided below with reference to FIG.

[0088]

[0033] Referring to Figure 3B, Figure 3B illustrates a flow diagram of an example use case for test execution, according to one or more embodiments. The operations of Figure 3B may be performed after the operations described above with reference to Figure 3A.

[0089] 3B , in operation S3201, module 230 may determine changes to the system under test (e.g., software, etc.). For example, module 230 may continuously (or periodically) obtain a current status of the system under test (from one or more nodes associated with the system under test) and compare the current status with a last known status of the system under test (previously obtained and stored by module 230) to determine whether the system under test has changed since the last known status. According to an embodiment, module 230 may perform operation S3201 according to a user-defined test cycle (relevant information being included in an associated test package).

[0090] After determining the change, module 230 may also determine whether the change is a breaking change. For example, module 230 may determine whether the change satisfies one or more conditions defined in at least one test package. Based on determining that the change satisfies the one or more conditions, module 230 may determine that the change is a breaking change. In this regard, the term "breaking change" may refer to an alteration or modification of the system under test that destroys or disrupts the normal functionality of the system under test.

[0091] Thus, in operation S3202, module 230 may generate and send a message (e.g., a notification flag, a warning message, etc.) to module 220 to inform it of the determined change in the system under test. Subsequently, in operation S3203, module 220 may trigger test execution by collecting test environment configuration files associated with the test package and the system under test (e.g., from module 230) and providing the test environment configuration files to module 240.

[0092] In operation S3204, module 240 may initialize one or more test benches. Specifically, module 240 may obtain (e.g., retrieve, create, configure, etc.) one or more test benches for executing tests based on a test environment configuration file. Accordingly, in operation S3205, module 240 may provide one or more test benches to module 250. Furthermore, in operation S3206, module 220 may provide a test package to module 250. In this regard, it may be understood that module 220 may perform operation S3206 simultaneously with operations S3203, S3204, or S320 without departing from the technical scope of the present disclosure.

[0093] In operation S3207, module 250 may execute tests to test the system under test. Specifically, module 250 may determine one or more nodes (e.g., nodes 120-1 to 120-N) that can provide a test environment (e.g., a software-based test environment, a hardware-based test environment, etc.) according to one or more test benches, may deploy the system under test (together with associated test components) to the determined one or more nodes, and may execute tests on the one or more nodes based on the test package.

[0094] Upon running the test, in operation S3208, module 220 may collect one or more test results associated with the test from module 250. Accordingly, in operation S3209, module 220 may archive the collected one or more test results by transmitting the one or more test results to module 230. Upon storing the one or more test results, in operation S3210, module 230 may provide the one or more test results to module 210. Accordingly, in operation S3211, module 210 may generate one or more GUIs that include the one or more test results and then present the one or more GUIs to the user (via a node associated with the user).

[0095] It can be understood that the operations of modules 210-250 described above are merely examples of possible implementations, and the scope of the present disclosure should not be limited thereby. In particular, one or more of modules 210-250 can be configured to operate in a manner different from that described herein without departing from the scope of the present disclosure.

[0096] According to an embodiment, one or more of the modules 210-250 in the test automation system may be defined in computer-executable instructions or programming code and may be associated with one or more hardware components of the test automation system. For example, computer-executable instructions defining one or more of the modules 210-250 may be stored in one or more memory storages of the test automation system and may be executable by one or more processors of the test automation system to perform one or more operations associated with the modules 210-250, as described herein.

[0097] Referring to Figure 4, Figure 4 shows a block diagram of example components of a test automation system 400, according to one or more embodiments. Test automation system 400 may be similar to test automation system 200 of Figure 2, and the components included therein may be configured to perform one or more operations of one or more modules of test automation system 200.

[0098] 4, test automation system 400 may include at least one communication interface 410, at least one storage 420, and at least one processor 430, although it may be understood that test automation system 400 may include more or fewer components than shown and / or the components included therein may be arranged in any manner different from that shown without departing from the scope of the present disclosure. For example, in some implementations, test automation system 400 may include multiple storages 420 and multiple processors 430, each of which may be dedicated to performing the operations of one of modules 210-250.

[0099] Communications interface 410 may include transceiver-like components (e.g., a transceiver, a separate receiver and transmitter, etc.) that enable test automation system 400 (or one or more components included therein) to communicate with one or more components external to test automation system 400 via wired connections, wireless connections, a combination of wired and wireless connections, etc. For example, communications interface 410 may couple test automation system 400 (or one or more components included therein) to multiple nodes (e.g., nodes 120-1 through 120-N in FIG. 1 ), thereby enabling them to communicate and interoperate with each other. As another example, communications interface 410 may enable components of test automation system 400 to communicate with each other. For example, communications interface 410 may couple storage 420 to processor 430, thereby enabling them to communicate and interoperate with each other.

[0100] According to embodiments, communication interface 410 may include a hardware-based interface, such as a bus interface, an Ethernet interface, an optical interface, a coaxial interface, an infrared interface, a radio frequency (RF) interface, a universal serial bus (USB) interface, a Wi-Fi interface, a cellular network interface, a software interface, etc. According to embodiments, communication interface 210 may include at least one controller area network (CAN) bus configurable to communicatively couple components of test automation system 400 (e.g., storage 420, processor 430, etc.) to multiple nodes (e.g., nodes 120-1 through 120-N). Additionally or alternatively, communication interface 410 may include a software-based interface, such as an application programming interface (API), a virtual network interface (e.g., a virtual CAN bus, etc.), etc.

[0101] According to an embodiment, communication interface 410 may be configured to receive information from one or more components external to test automation system 400 and provide it to processor 430 for further processing and / or to storage 420 for storage. For example, communication interface 410 may receive one or more user inputs from multiple nodes defining one or more test requirements (e.g., test cases, test plans, test cycles, etc.).

[0102] At least one storage 420 may include one or more storage media suitable for storing data, information, and / or computer-readable / computer-executable instructions. According to an embodiment, storage 420 may include random access memory (RAM), read-only memory (ROM), and / or another type of dynamic or static storage device (e.g., flash memory, magnetic memory, and / or optical memory) that stores information and / or instructions for use by processor 430.

[0103] Additionally or alternatively, storage 420 may include hard disks (e.g., magnetic disks, optical disks, magneto-optical disks, and / or solid-state disks), compact disks (CDs), digital versatile disks (DVDs), floppy disks, cartridges, magnetic tape, and / or another type of non-transitory computer-readable medium, along with corresponding drives.

[0104] According to an embodiment, storage 420 may be configured to store computer-executable instructions or programming code defining one or more of modules 210-250 (described above with reference to FIGS. 2-3B). Additionally, storage 420 may be configured to store information utilized by processor 430 to facilitate test management automation. For example, storage 420 may be configured to store one or more test requirements defined by one or more users, store one or more systems under test and associated test components or artifacts (e.g., programming code defining the software to be tested and / or defining software that interoperates with the software to be tested), store one or more generated test packages and generated test environment configuration files, and store one or more test results.

[0105] At least one processor 430 may include one or more processors that may be programmed to perform the functions or operations described herein. For example, processor 430 may be configured to perform one or more operations or actions described herein by executing computer-readable instructions stored on a storage medium (e.g., such as storage 420).

[0106] According to an embodiment, processor 430 may be configured to receive one or more signals (e.g., via communications interface 410, etc.) that define one or more instructions for performing one or more operations. Furthermore, processor 430 may be implemented in hardware, firmware, or a combination of hardware and software. Processor 430 may include a central processing unit (CPU), a graphics processing unit (GPU), an accelerated processing unit (APU), a microprocessor, a microcontroller, a digital signal processor (DSP), a field-programmable gate array (FPGA), an application-specific integrated circuit (ASIC), and / or another type of processing or computing component.

[0107] According to an embodiment, at least one processor 430 may be configured to execute computer-executable instructions stored in at least one memory storage (e.g., storage 420) to thereby perform one or more operations for managing tests for embedded system test software (sometimes referred to herein as a “system under test”). The embedded system software may include an on-board ECU.

[0108] According to an embodiment, the at least one processor 430 may be configured to receive at least one first user input from a user related to one or more test requirements; generate at least one test package based on the first user input; generate at least one test environment configuration file based on the at least one test package; validate the at least one test package and the at least one test environment configuration file; determine software changes; and run tests in accordance with the validated at least one test package and the validated at least one test environment configuration file based on the determined software changes.

[0109] The software may be associated with (e.g., developed, managed, etc.) a different user than the user, who may be located in a different geographic location than the user. Furthermore, the user may access the test automation system 400 through a node (e.g., user equipment, etc.) that is different from the node on which the software is deployed or hosted.

[0110] According to an embodiment, the first user input may include information related to at least one user-defined test case. In this regard, the at least one processor 430 may be configured to generate the at least one test suite by generating at least one test scenario template based on the at least one user-defined test case, obtaining one or more test package artifacts based on the at least one test scenario template, and combining the at least one test scenario template with the one or more test package artifacts to generate the at least one test suite.

[0111] According to an embodiment, the at least one processor 430 may be configured to generate the at least one test environment configuration file by receiving at least one second user input from a user related to the at least one user-defined test plan, generating at least one test plan template based on the second user input and the at least one test package, generating the at least one test plan by adding at least one user-defined test case to the at least one test plan template, receiving at least one third user input from the user related to the at least one user-defined test cycle, and generating the at least one test environment configuration file based on the third user input and the at least one test plan. In this regard, the at least one test environment configuration file may include information defining a configuration of the at least one test bench.

[0112] The systems and / or methods described herein may be implemented in different forms of hardware, firmware, or a combination of hardware and software. The actual specialized control hardware or software code used to implement these systems and / or methods is not intended to limit the implementation. As such, the operation and behavior of the systems and / or methods have been described herein without reference to specific software code, and it will be understood that software and hardware can be designed to implement the systems and / or methods based on the description herein.

[0113] In some implementations, the pre-testing for validating the at least one test package and the at least one test environment configuration file may be a simplified test that includes testing only required functionality and / or user-specified functionality, and the pre-testing may be performed on a node different from the node on which the testing performed to determine the software change will be performed (e.g., may be referred to herein as “actual testing”). For example, the pre-testing may be performed locally on equipment or a workstation associated with the user, and the actual testing may be performed on multiple nodes dynamically selected on-the-fly by a test automation system according to the real-time or near-real-time status (e.g., resource availability, performance metrics, etc.) of the multiple nodes.

[0114] Additionally or alternatively, at least one processor 430 may be configured to receive approval / rejection of the pre-test results from another user different from the user. For example, the other user may be a senior test engineer responsible for verifying the accuracy of the test package and / or test environment configuration file generated according to the test requirements defined by the user. In this manner, the other user can efficiently and effectively ensure that the test requirements are properly defined by the user.

[0115] According to an embodiment, the at least one processor 430 may be configured to determine changes to the software by obtaining a current status of the software and comparing the current status with a last known status of the software to determine whether changes have occurred to the software. The last known status of the software may have been previously obtained (during a previous test cycle) by the at least one processor 430.

[0116] In some implementations, the software change may include a breaking change that satisfies one or more conditions defined in the at least one test package. In this regard, upon determining the software change, the at least one processor 430 may be further configured to determine whether the software change is a breaking change. Upon determining that the software change is a breaking change, the at least one processor 430 may perform a test. Otherwise, upon determining that the software change is not a breaking change, the at least one processor 430 may not perform a test.

[0117] According to an embodiment, the at least one processor 430 may be configured to perform the testing by: generating, based on the validated at least one test environment configuration file, at least one test bench associated with at least one test environment for testing the software; selecting, based on the at least one test bench, at least one node (communicatively coupled to the test automation system 400) associated with the at least one test environment defined in the at least one test bench; obtaining the software (or associated program code) and associated test components (e.g., other software that interacts with the software under test); deploying the software and associated test components to the selected at least one node; and executing the tests to test the software in the selected at least one node based on the validated at least one test package.

[0118] According to an embodiment, upon validating the at least one test package and the at least one test environment configuration file, the at least one processor 430 may be further configured to publish the validated at least one test package and the validated at least one test environment configuration file. For example, the at least one processor 430 may store the validated at least one test package and the validated at least one test environment configuration file in one or more storage media (e.g., storage 420, a content hosting server, etc.) accessible by other users. In this manner, other users may select to utilize the validated at least one test package and the validated at least one test environment configuration file when needed (e.g., when running similar tests on the same or similar software).

[0119] According to an embodiment, upon running a test, at least one processor 430 may be further configured to present one or more test results to a user. For example, at least one processor 430 may collect one or more test results associated with the test, generate at least one graphical user interface (GUI) that includes the one or more test results, and present the at least one GUI to a user.

[0120] According to an embodiment, at least one processor 430 may be further configured to update at least one test package and / or at least one test environment configuration file. For example, at least one processor 430 may receive one or more user inputs related to one or more updated test requirements (e.g., updated test cases, updated test plans, updated test cycles, etc.) and may update at least one test package and / or at least one test environment configuration file accordingly.

[0121] As an example, upon receiving one or more user-defined updated test cases, at least one processor 430 may update at least one test suite by generating one or more updated test scenario templates based on the one or more user-defined updated test cases, obtaining (e.g., creating, retrieving, retrieving, etc.) one or more test package artifacts associated with the one or more updated test scenario templates from among multiple nodes associated with the test automation system (e.g., nodes 120-1 through 120-N), and combining the one or more updated test package artifacts with the one or more test package artifacts to generate one or more updated test suites. According to an embodiment, the at least one processor may determine one or more differences between the one or more user-defined updated test cases and the current user-defined test cases, and may update one or more of the test scenario templates by modifying one or more generated test scenario templates to reflect the one or more differences. At least one test environment configuration file may be updated by at least processor 430 in a similar manner.

[0122] In view of the above, exemplary embodiments of the present disclosure provide a test automation system that automates the end-to-end process in managing tests for a user. Specifically, a user can simply provide intended test requirements (e.g., intended test cases, intended test plans, intended test cycles, etc.) to the test automation system, and the test automation system can automatically generate test packages and test environment configuration files based thereon. The user only needs to provide the test requirements remotely from an associated node (e.g., user equipment, workstation, etc.) communicatively coupled to the test automation system (e.g., via a network 130) and does not need to physically travel to a test facility as in related art.

[0123] Furthermore, the test automation system can validate the generated test package and / or the generated test environment configuration file before publishing and utilizing the generated test package and the generated test environment configuration file for actual testing, thereby allowing a user to verify the accuracy of the generated test package and the generated test environment configuration file, and allowing the user to automatically perform such actions based on the validated test package and the validated test environment configuration file, thereby eliminating the need to manually monitor the status of the system under test and manually run the tests after validating the generated test package and the generated test environment configuration file.

[0124] Furthermore, the test automation system can automatically allocate resources for executing tests according to the validated test package and the validated test environment configuration file without user intervention. The test automation system can dynamically select a node with sufficient resources to execute the test and then execute the test on that node. The test automation system can then automatically collect and present the test results to the user.

[0125] To this end, a user can simply define the intended test requirements from any suitable node communicatively coupled to the test automation system, and the test automation system can automatically manage the tests on behalf of the user. Thus, software testing is managed in an efficient and effective manner, thereby reducing the burden on the user and shortening the software development time.

[0126] It should be understood that the specific order or hierarchy of blocks in the processes / flowcharts disclosed herein is an example of an example approach. Based on design preferences, it should be understood that the specific order or hierarchy of blocks in the processes / flowcharts may be rearranged. Further, some blocks may be combined or omitted. The accompanying method claims present elements of the various blocks in an example order and are not limited to the specific order or hierarchy presented.

[0127] Some embodiments may relate to systems, methods, and / or computer-readable media at any possible level of technical detail of integration. Furthermore, one or more of the above components described above may be implemented as instructions stored on a computer-readable medium and executable by at least one processor (and / or may include at least one processor). The computer-readable medium may include a computer-readable non-transitory storage medium having computer-readable program instructions for causing a processor to perform operations.

[0128] A computer-readable storage medium can be a tangible device that can hold and store instructions for use by an instruction-execution device. A computer-readable storage medium can be, for example, but not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the above. A non-exhaustive list of more specific examples of computer-readable storage media includes the following: portable computer diskettes, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital versatile disc (DVD), memory sticks, floppy disks, mechanically encoded devices such as punch cards or ridge structures in grooves having instructions recorded thereon, and any suitable combination of the above. Computer-readable media, as used herein, should not be construed as transitory signals per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through a waveguide or other transmission medium (e.g., light pulses passing through a fiber optic cable), or electrical signals transmitted over wires.

[0129] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium into each computing / processing device, or can be downloaded to an external computer or external storage device over a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, optical fiber transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in a computer-readable storage medium within each computing / processing device.

[0130] The computer readable program code / instructions for carrying out operations may be either source code or object code written in any combination of one or more programming languages, including assembler instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, configuration data for an integrated circuit, or object-oriented programming languages ​​such as Smalltalk, C++, etc., and procedural programming languages ​​such as the "C" programming language or similar programming languages. The computer readable program instructions may be executed entirely on the user's computer, partially on the user's computer, partially on the user's computer as a standalone software package, and partially on a remote computer, or entirely on a remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection may be to an external computer (e.g., through the Internet using an Internet Service Provider). In some embodiments, electronic circuitry including, for example, a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA) can execute computer-readable program instructions by utilizing state information in the computer-readable program instructions to personalize the electronic circuitry to perform aspects or operations.

[0131] These computer-readable program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus to fabricate a machine, such that the instructions, executed by the processor of the computer or other programmable data processing apparatus, create means for implementing the functions / acts specified in the flowchart and / or block diagram blocks. These computer-readable program instructions can also be stored on a computer-readable storage medium that can instruct a computer, programmable data processing apparatus, and / or other device to function in a particular manner, such that the computer-readable storage medium having stored thereon instructions comprises an article of manufacture containing instructions that implement aspects of the functions / acts specified in the flowchart and / or block diagram blocks.

[0132] The computer-readable program instructions can also be loaded into a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus, or other device to generate a computer-implemented process such that the instructions, when executed on the computer, other programmable apparatus, or other device, implement the functions / acts specified in the flowchart and / or block diagram blocks.

[0133] The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer-readable media according to various embodiments. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction set, which comprises one or more executable instructions for implementing the specified logical function(s). The methods, computer systems, and computer-readable media may include additional, fewer, different, or differently arranged blocks than shown in FIG. 1. In some alternative implementations, the functions depicted in the blocks may occur in a different order than that depicted in the figures. For example, two blocks shown in succession may in fact be executed concurrently or substantially concurrently, or the blocks may be executed in the reverse order, depending on the functionality involved. It will also be noted that each block in the block diagram and / or flowchart illustrations, and combinations of blocks in the block diagram and / or flowchart illustrations, may be implemented by special-purpose hardware-based systems that perform the specified functions or operations and execute a combination of special-purpose hardware and computer instructions.

[0134] The systems and / or methods described herein may be implemented in different forms of hardware, firmware, or a combination of hardware and software. The actual specialized control hardware or software code used to implement these systems and / or methods is not intended to limit the implementation. As such, the operation and behavior of the systems and / or methods have been described herein without reference to specific software code, and it will be understood that software and hardware can be designed to implement the systems and / or methods based on the description herein. The present disclosure encompasses the following aspects. (1) 1. A method, implemented by at least one processor, for managing tests for testing software of an embedded system, comprising: The method comprises: receiving at least one first user input from a user related to one or more test requirements; generating at least one test package based on the first user input; generating at least one test environment configuration file based on the at least one test package; validating at least one of the test packages and at least one of the test environment configuration files; determining modifications to the software; executing the tests according to the validated at least one test package and the validated at least one test environment configuration file based on determining the software changes; Including, The software of the embedded system includes an on-board electronic control unit (ECU); method. (2) publishing the validated at least one test package and the validated at least one test environment configuration file by storing the validated at least one test package and the validated at least one test environment configuration file on one or more storage media accessible by other users; further comprising: The method described in (1) above. (3) the first user input includes information related to at least one user-defined test case; Generating the at least one test package includes: generating at least one test scenario template based on the at least one user-defined test case; Obtaining one or more test package artifacts based on at least one test scenario template; combining at least one of the test scenario templates with one or more of the test package artifacts to generate at least one of the test packages; Including, The method according to (1) or (2) above. (4) Generating at least one said test environment configuration file includes: receiving at least one second user input from the user related to at least one user-defined test plan; generating at least one test plan template based on the second user input and the at least one test package; generating at least one test plan by adding at least one of the user-defined test cases to at least one of the test plan templates; receiving at least one third user input from the user related to at least one user-defined test cycle; generating at least one of the test environment configuration files based on the third user input and at least one of the test plans; Including, The method according to (1) or (2) above. (5) at least one of the test environment configuration files includes information defining settings for at least one test bench; The method according to (1) or (2) above. (6) Validating the at least one test package and the at least one test environment configuration file includes: conducting a pre-test of the software based on at least one of the test packages and at least one of the test environment configuration files; presenting the results of the pre-test to the user; receiving at least one fourth user input from the user relating to one of an approval of the results of the pre-test and a rejection of the results of the pre-test; determining that the at least one test package and the at least one test environment configuration file are valid based on determining that the fourth user input is associated with an authorization; and determining that at least one of the test packages and at least one of the test environment configuration files is invalid based on determining that the fourth user input is associated with a rejection; and Including, The method according to (1) or (2) above. (7) Determining the modification of the software includes: obtaining a current status of the software; comparing the current status with the last known status of the software to determine if a change has occurred in the software; Including, The method according to (1) or (2) above. (8) the software changes include breaking changes that satisfy one or more conditions defined in at least one of the test packages; The method according to (1) or (2) above. (9) performing the test generating at least one test bench associated with at least one test environment for testing the software based on the validated at least one test environment configuration file; selecting, based on at least one test bench, at least one node associated with at least one test environment defined in said at least one test bench; deploying the software to the at least one selected node; conducting a test to test the software on the at least one selected node based on the verified at least one test package; Including, The method according to (1) or (2) above. (10) collecting one or more test results associated with said test; generating at least one graphical user interface (GUI) including one or more of said test results; presenting at least one of said GUIs to said user; further comprising: The method according to (1) or (2) above. (11) A test management system for testing software of an embedded system, comprising: The system comprises: a storage for storing instructions; receiving at least one first user input from a user related to one or more test requirements; generating at least one test package based on the first user input; generating at least one test environment configuration file based on the at least one test package; Validating at least one of the test packages and at least one of the test environment configuration files; determining modifications to the software; running the tests according to the validated at least one test package and the validated at least one test environment configuration file based on determining the software changes; at least one processor configured to execute the instructions for: Equipped with The software of the embedded system includes an on-board electronic control unit (ECU); system. (12) At least one of the processors publishing the validated at least one test package and the validated at least one test environment configuration file by storing the validated at least one test package and the validated at least one test environment configuration file on one or more storage media accessible by other users; further configured to execute the instructions for: The system according to (11) above. (13) the first user input includes information related to at least one user-defined test case; At least one of the processors generating at least one test scenario template based on the at least one user-defined test case; Obtaining one or more test package artifacts based on at least one test scenario template; combining at least one of the test scenario templates with one or more of the test package artifacts to generate at least one of the test packages; and further configured to execute the instructions to generate the at least one test package by: The system according to (11) or (12) above. (14) At least one of the processors receiving at least one second user input from the user related to at least one user-defined test plan; generating at least one test plan template based on the second user input and the at least one test package; generating at least one test plan by adding at least one of the user-defined test cases to at least one of the test plan templates; receiving at least one third user input from the user related to at least one user-defined test cycle; generating at least one of the test environment configuration files based on the third user input and at least one of the test plans; and further configured to execute the instructions to generate at least one of the test environment configuration files by The system according to (11) or (12) above. (15) at least one of the test environment configuration files includes information defining settings for at least one test bench; The system according to (11) or (12) above. (16) At least one of the processors conducting a pre-test of the software based on at least one of the test packages and at least one of the test environment configuration files; presenting the results of the pre-test to the user; receiving at least one fourth user input from the user relating to one of an approval of the results of the pre-test and a rejection of the results of the pre-test; determining that the at least one test package and the at least one test environment configuration file are valid based on determining that the fourth user input is associated with an authorization; determining that at least one of the test packages and at least one of the test environment configuration files is invalid based on determining that the fourth user input is associated with a rejection; and further configured to execute the instructions to validate the at least one test package and the at least one test environment configuration file by The system according to (11) or (12) above. (17) At least one of the processors Obtaining a current status of the software; comparing the current status with the last known status of the software to determine if a change has occurred to the software; and further configured to execute the instructions to determine a modification of the software by The system according to (11) or (12) above. (18) the software changes include breaking changes that satisfy one or more conditions defined in at least one of the test packages; The system according to (11) or (12) above. (19) At least one of the processors generating at least one test bench associated with at least one test environment for testing the software based on the validated at least one test environment configuration file; selecting, based on the at least one test bench, at least one node associated with the at least one test environment defined in the at least one test bench; Deploying the software to the at least one selected node; conducting a test to test the software on the at least one selected node based on the verified at least one test package; and further configured to execute the instructions for performing the test by The system according to (11) or (12) above. (20) At least one of the processors collecting one or more test results associated with said test; generating at least one graphical user interface (GUI) including one or more of said test results; presenting at least one of the GUIs to the user; further configured to execute the instructions for: The system according to (11) or (12) above.

Claims

1. 1. A method, implemented by at least one processor, for managing tests for testing software of an embedded system, comprising: The method comprises: receiving at least one first user input from a user related to one or more test requirements; generating at least one test package including test scripts or test algorithms for testing the software based on the first user input; generating at least one test environment configuration file based on the at least one test package; validating at least one of the test packages and at least one of the test environment configuration files; determining modifications to the software; executing the tests according to the validated at least one test package and the validated at least one test environment configuration file based on determining the software changes; Including, Validating the at least one test package and the at least one test environment configuration file includes: conducting a pre-test of the software based on at least one of the test packages and at least one of the test environment configuration files; presenting the results of the pre-test to the user; receiving at least one fourth user input from the user relating to one of an approval of the results of the pre-test and a rejection of the results of the pre-test; determining that the at least one test package and the at least one test environment configuration file are valid based on determining that the fourth user input is associated with an approval; and determining that at least one of the test packages and at least one of the test environment configuration files is invalid based on determining that the fourth user input is associated with a rejection; and Including, method.

2. publishing the validated at least one test package and the validated at least one test environment configuration file by storing the validated at least one test package and the validated at least one test environment configuration file on one or more storage media accessible by other users; further comprising: The method of claim 1.

3. the first user input includes information related to at least one user-defined test case; Generating the at least one test package includes: generating at least one test scenario template based on the at least one user-defined test case; Obtaining one or more test package artifacts based on at least one test scenario template; combining at least one of the test scenario templates with one or more of the test package artifacts to generate at least one of the test packages; Including, The method according to claim 1 or claim 2.

4. Generating at least one said test environment configuration file includes: receiving at least one second user input from the user related to at least one user-defined test plan; generating at least one test plan template based on the second user input and the at least one test package; generating at least one test plan by adding at least one of the user-defined test cases to at least one of the test plan templates; receiving at least one third user input from the user related to at least one user-defined test cycle; generating at least one of the test environment configuration files based on the third user input and at least one of the test plans; Including, The method according to claim 1 or claim 2.

5. at least one said test environment configuration file includes information defining settings for at least one test bench; The method according to claim 1 or claim 2.

6. Determining the modification of the software includes: obtaining a current status of the software; comparing the current status with the last known status of the software to determine if a change has occurred in the software; Including, The method according to claim 1 or claim 2.

7. the software changes include breaking changes that satisfy one or more conditions defined in at least one of the test packages; The method according to claim 1 or claim 2.

8. performing the test generating at least one test bench associated with at least one test environment for testing the software based on the validated at least one test environment configuration file; selecting, based on at least one test bench, at least one node associated with at least one test environment defined in said at least one test bench; deploying the software to the at least one selected node; conducting a test to test the software on the at least one selected node based on the verified at least one test package; Including, The method according to claim 1 or claim 2.

9. collecting one or more test results associated with said test; generating at least one graphical user interface (GUI) including one or more of said test results; presenting at least one of the GUIs to the user; further comprising: The method according to claim 1 or claim 2.

10. A test management system for testing software of an embedded system, comprising: The system comprises: a storage for storing instructions; receiving at least one first user input from a user related to one or more test requirements; generating at least one test package including test scripts or test algorithms for testing the software based on the first user input; generating at least one test environment configuration file based on the at least one test package; Validating at least one of the test packages and at least one of the test environment configuration files; determining modifications to the software; executing the tests according to the validated at least one test package and the validated at least one test environment configuration file based on determining the software changes; at least one processor configured to execute the instructions for: Equipped with At least one of the processors conducting a pre-test of the software based on at least one of the test packages and at least one of the test environment configuration files; presenting the results of the pre-test to the user; receiving at least one fourth user input from the user relating to one of an approval of the results of the pre-test and a rejection of the results of the pre-test; determining that the at least one test package and the at least one test environment configuration file are valid based on determining that the fourth user input is associated with an approval; determining that the at least one test package and the at least one test environment configuration file are invalid based on determining that the fourth user input is associated with a rejection; and further configured to execute the instructions to validate the at least one test package and the at least one test environment configuration file by system.

11. At least one of the processors publishing the validated at least one test package and the validated at least one test environment configuration file by storing the validated at least one test package and the validated at least one test environment configuration file on one or more storage media accessible by other users; further configured to execute the instructions for: The system of claim 10.

12. the first user input includes information related to at least one user-defined test case; At least one of the processors generating at least one test scenario template based on the at least one user-defined test case; Obtaining one or more test package artifacts based on at least one test scenario template; combining at least one of the test scenario templates with one or more of the test package artifacts to generate at least one of the test packages; and further configured to execute the instructions to generate the at least one test package by:

12. A system according to claim 10 or claim 11.

13. At least one of the processors receiving at least one second user input from the user related to at least one user-defined test plan; generating at least one test plan template based on the second user input and the at least one test package; generating at least one test plan by adding at least one of the user-defined test cases to at least one of the test plan templates; receiving at least one third user input from the user related to at least one user-defined test cycle; generating at least one of the test environment configuration files based on the third user input and at least one of the test plans; and further configured to execute the instructions for generating at least one of the test environment configuration files by 12. A system according to claim 10 or claim 11.

14. at least one said test environment configuration file includes information defining settings for at least one test bench; 12. A system according to claim 10 or claim 11.

15. At least one of the processors Obtaining a current status of the software; comparing the current status with the last known status of the software to determine if a change has occurred to the software; and further configured to execute the instructions to determine a modification of the software by 12. A system according to claim 10 or claim 11.

16. the software changes include breaking changes that satisfy one or more conditions defined in at least one of the test packages; 12. A system according to claim 10 or claim 11.

17. At least one of the processors generating at least one test bench associated with at least one test environment for testing the software based on the validated at least one test environment configuration file; selecting, based on the at least one test bench, at least one node associated with the at least one test environment defined in the at least one test bench; Deploying the software to the selected at least one node; conducting a test to test the software on the at least one selected node based on the verified at least one test package; and further configured to execute the instructions for performing the test by 12. A system according to claim 10 or claim 11.

18. At least one of the processors collecting one or more test results associated with said test; generating at least one graphical user interface (GUI) including one or more of said test results; presenting at least one of the GUIs to the user; further configured to execute the instructions for:

12. A system according to claim 10 or claim 11.

Images