Horizontally scalable distributed system and method for automated firmware testing

Abstract

A system having a horizontally scalable distributed architecture for automated firmware testing includes: test stations for testing firmware products, the test stations being divided into pools, each pool including multiple test stations; and multiple execution entities, each execution entity being configured to execute tests corresponding to an associated pool. Each competing test station sends a test initiation event to a corresponding execution entity. The corresponding execution entity receives the test initiation event from the competing test stations and executes a run test command on a selected test station among the competing test stations, causing the selected test station to execute the test based on a test sequence.

Description

Technical Field

[0001] An embodiment of the present invention relates to a scheme for testing firmware of an electronic device. Background Art

[0002] Electronic devices are widely used in various forms. Examples of large electronic devices include desktop computers, workstations, three-dimensional (3D) televisions, smart TVs, digital audio recorders, digital audio players, digital photo recorders, digital photo players, and / or digital video recorders and digital video players. There are also various portable electronic devices, such as mobile phones, smart phones, e-books, MP3 players, portable multimedia players (PMPs), and portable game consoles.

[0003] Electronic devices can be implemented using various components including a memory system (or storage device), including hardware (HW) and software (SW). Examples of memory systems include hard disk drives (HDDs), solid-state drives (SSDs), universal serial bus (USB) memory devices, and memory cards such as secure digital (SD) cards and universal flash storage (UFS). It may be necessary to test the components of an electronic device. In this context, embodiments of the present invention are proposed. Summary of the Invention

[0004] Aspects of the present invention include systems and methods having a horizontally scalable distributed architecture for automated firmware testing.

[0005] In one aspect, a system includes: a plurality of test stations for testing a plurality of firmware products, the plurality of test stations being divided into a plurality of pools, each pool including a plurality of test stations; and a plurality of execution instances, each execution instance being configured to execute a test corresponding to an associated pool. Each competing test station among the plurality of test stations transmits a test initiation event to a corresponding execution instance. The corresponding execution instance receives the test initiation event from the competing test station and executes a run-test command on a selected test station among the competing test stations, causing the selected test station to perform a test execution based on a test sequence.

[0006] On the other hand, a method for operating a test system includes: dividing a plurality of test stations for testing a plurality of firmware products into a plurality of pools, each pool including a plurality of test stations; delivering a test start event by each competing test station among the plurality of test stations to a corresponding execution entity among a plurality of execution entities, each execution entity being used to execute a test corresponding to the associated pool; receiving the test start event from the competing test station by the corresponding execution entity; and executing a run test command on a selected test station among the competing test stations by the corresponding execution entity, so that the selected test station performs test execution based on a test sequence.

[0007] Other aspects of the invention will become apparent from the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] Figure 1 is a schematic diagram illustrating a test laboratory according to an embodiment of the present invention.

[0009] Figure 2 is a diagram illustrating a plurality of stations divided into pools according to an embodiment of the present invention.

[0010] Figure 3 is a diagram illustrating a mechanism in which an execution entity is scaled according to an embodiment of the present invention.

[0011] Figure 4 FIG. 4 is a diagram illustrating communication between a test station pool and an execution entity according to an embodiment of the present invention.

[0012] Figure 5 is a diagram illustrating states of an agent according to an embodiment of the present invention.

[0013] Figure 6 is a diagram illustrating the structure of an execution entity according to an embodiment of the present invention.

[0014] Figure 7 is a diagram illustrating a test execution process according to an embodiment of the present invention.

[0015] Figure 8 is a diagram illustrating performance of a test system according to an embodiment of the present invention. DETAILED DESCRIPTION

[0016] Each embodiment is described in more detail below with reference to the accompanying drawings. However, the present invention can be implemented in different forms and should not be construed as being limited to the embodiments set forth herein. On the contrary, these embodiments are provided to make this disclosure thorough and complete and to fully convey the scope of the invention to those skilled in the art. In addition, references to "an embodiment," "another embodiment," etc. herein do not necessarily refer to only one embodiment, and different references to any such phrases do not necessarily refer to the same embodiment. The term "embodiment" used herein does not necessarily refer to all embodiments. Throughout the disclosure, the same reference numerals represent the same parts in the drawings and embodiments of the present invention.

[0017] The present invention may be implemented in a variety of ways, including as a process; as an apparatus; as a system; as a computer program product embedded in a computer-readable storage medium; and / or as a processor, for example, a processor adapted to execute instructions stored on a memory coupled to the processor and / or as provided by a memory coupled to the processor. In this specification, these embodiments or any other form that the present invention may take may be referred to as techniques. In general, the order of the steps of the disclosed processes may be changed within the scope of the present invention. Unless otherwise stated, components such as processors or memories described as being adapted to perform tasks may be implemented as general components or as specific components, with general components being temporarily configured to perform tasks at a given time and specific components being manufactured to perform tasks. As used herein, the term "processor" or the like refers to one or more devices, circuits, and / or processing cores adapted to process data (e.g., computer program instructions).

[0018] Detailed descriptions of embodiments of the present invention and accompanying drawings illustrating aspects of the present invention are provided below. The present invention is described in conjunction with these embodiments, but the invention is not limited to any embodiment. The scope of the present invention is limited only by the claims. The present invention encompasses numerous alternatives, modifications, and equivalents within the scope of the claims. In order to provide a thorough understanding of the present invention, numerous specific details are set forth in the following description. These details are provided for illustrative purposes; the present invention may be practiced according to the claims without some or all of these specific details. For the sake of clarity, technical material that is known in the technical fields related to the present invention has not been described in detail so as not to unnecessarily obscure the present invention.

[0019] There is a need to test components (or products) of electronic devices. In particular, there is a need to test software (or firmware) products of electronic devices. Firmware is a specific type of computer software that provides low-level control for specific hardware of a device. Typical examples of electronic devices that contain firmware are embedded systems, consumer electronics, computers, computer peripherals, and memory systems (or storage devices). Firmware can be stored in non-volatile memory devices such as read-only memory (ROM), electrically programmable read-only memory (EPROM), or flash memory.

[0020] Automated software (or firmware) testing requires running a large number of tests simultaneously. Test stations or machines in isolated environments are used to execute separate, limited test sequences. The increase in the number of tests executed simultaneously leads to Figure 1 The increase in the number of test stations shown.

[0021] refer to Figure 1 , the test laboratory 10 includes a test controller or distributor 11 and L test stations (or machines), individually and collectively designated by 12, and K products to be tested, individually and collectively designated by 13. iTests 14 will be run on L test stations to test K products, where i=1, ..., K. Test distributor 11 may include a processor 11A for controlling the tests on the K products. For example, test lab 10 may test the firmware of a component or product (i.e., a firmware product) embedded in an electronic device such as a memory system (or storage device).

[0022] In the test lab 10, the coordination of test stations 12 and the processing of tests 14 results in a heavy computational load. This computational load cannot be handled solely through vertical scaling of the test lab 10. Typically, there are many more tests to be executed than test stations available for execution. This raises another issue: high concurrency during test station allocation. Therefore, a test system according to an embodiment can provide an architecture that allows horizontal scaling to be applied to the task of executing automated tests at scale, while also handling high concurrency between tests at test stations.

[0023] To manage a larger number of test stations, embodiments may partition (or group) multiple machines into pools and connect each pool to a separate subset of system components: independent execution entities. Figure 2 In the illustrated embodiment, the test system may include a plurality of test stations 100 for testing a plurality of products. For example, the products may include embedded firmware, i.e., firmware products. The plurality of test stations 100 may be divided into a plurality of pools 200. For example, the first pool 201 includes a plurality (e.g., M) of test stations TS11 to TS1M, the second pool 202 includes M test stations TS21 to TS2M, and the nth pool 20N includes M test stations TSN1 to TSNM. Although Figure 2 Each pool is shown as having the same number of test stations, but the number of test stations in each pool can be different. In some embodiments, the test stations are grouped according to set characteristics of the product to be tested (e.g., a firmware product of a memory system): the hardware used in the memory system, the operating system used in the memory system, the available memory (e.g., RAM) and / or the number of memory devices (e.g., HDD) in the memory system.

[0024] Furthermore, the test system may include multiple execution entities 300. Each execution entity can execute tests on test stations in a corresponding pool. For example, the first execution entity 301 executes tests on test stations in the first pool 201, the second execution entity 302 executes tests on test stations in the second pool 202, and the Nth execution entity 30N executes tests on test stations in the Nth pool 20N. An execution entity can be a single environment with multiple components responsible for the overall test execution process. Each execution entity can manage a pool of test stations. A set of tests to be executed is associated with test stations in a specific pool.

[0025] according to Figure 2 In the embodiment, horizontal scaling can be achieved by increasing the number of independent execution entities. In addition, the embodiment provides a solution to the high concurrency problem by reversing the distribution flow between test stations and tests. In other words, the embodiment implements a solution in which test stations compete for tests instead of tests competing for test stations. This solution may significantly reduce concurrency (i.e., simultaneous execution) because the number of test stations is always much smaller than the number of tests. Figures 3 to 7 ,describe Figure 2 The structure and scheme of the test system in .

[0026] Figure 3 is a diagram illustrating a mechanism in which an execution entity is scaled according to an embodiment of the present invention.

[0027] refer to Figure 3 , the plurality of execution entities 300 may include N execution entities 301 to 30N (where N is the number of execution entities) in parallel and connected to a master (or main) entity 400. The N execution entities 301 to 30N and the master entity 400 may be Figure 1 The main entity 400 is a component of the test controller 11 in the test system that is responsible for the test execution process. In addition, the main entity 400 may include a user interface connected to the test controller 11, as well as an interface between the test controller 11 and the user. To initiate a test, the main entity 400 may send a message associated with the test to the message system 500.

[0028] The messaging system 500 can organize communications between the master entity 400 and the execution entities 301 through 30N. The messaging system 500 can include a message broker 510, an entity test queue 520, and a test result queue 530. The message broker 510 can place messages associated with tests from the master interface into the appropriate entity test queue 520 using a set queue routing mechanism. The entity test queue 520 can include entity test queues 521 through 52N, which correspond to the execution entities 301 through 30N, respectively. For example, the message broker 510 can place messages associated with tests for the execution entity 301 into the entity test queue 521.

[0029] Each of the execution entities 301 to 30N can independently obtain messages from its own queue and process tests based on these messages. After completing the test, each of the execution entities 301 to 30N can put the test results into the corresponding outgoing test result queue 530.

[0030] Figure 4 FIG. 1 is a diagram illustrating communication between the test station pool 201 and the first execution entity 301 according to an embodiment of the present invention.

[0031] refer to Figure 4 Test station pool 201 may include multiple test stations, each of which may include an agent, which is specialized system software running on the test station. In other words, test station pool 201 may include multiple agents 211 through 21M. Each agent is responsible for executing remote commands on a corresponding test station. First execution entity 301 may connect to multiple agents 211 through 21M via a predefined interface (e.g., a remote procedure call (RPC) protocol).

[0032] Multiple agents 211 to 21M may compete for access to the first execution entity 301. In other words, multiple test stations corresponding to the multiple agents 211 to 21M may compete for testing by transmitting test requests (i.e., test initiation events) to the first execution entity 301. When a test initiation event is received from the competing multiple test stations, the first execution entity 301 may select a test initiation event from among the competing test initiation events for test execution. In some embodiments, the first execution entity 301 may select the test initiation event that is first received and stored in a queue (e.g., Figure 6 Test start events in the idle queue of .

[0033] Figure 5 is a diagram illustrating states of an agent according to an embodiment of the present invention.

[0034] refer to Figure 5 During the lifecycle of a corresponding test station, an agent can be in one of several states: "Idle", "Running", and "Completed". The state "Idle" indicates that the test station is not busy and is ready to start executing some test sequence. The state "Running" indicates that the test station is executing tests. The state "Completed" indicates that the test station has completed test execution and is ready to provide test results.

[0035] Figure 6 30N is a diagram showing the structure of the execution entity 301 according to an embodiment of the present invention. Each of the remaining execution entities 302 to 30N may have the same structure.

[0036] refer to Figure 6 , the execution entity 301 may Figure 3 is connected to the main entity 400 as shown, and can be Figure 4 2. As shown, the test station pool 201 is connected to the test station pool 201. The test station pool 201 may include test stations with activated agents 211 to 21M therein. Each agent may be connected to the execution entity 301. When each agent changes state, the event may be reported to the execution entity 301.

[0037] The execution entity 301 may include a test engine 310, an agent manager 320, state queues 330A to 330C, an incoming command queue 340A, and an external event queue 340B. The incoming command queue 340A may correspond to Figure 3 The entity test queue 521, the external event queue 340B may correspond to Figure 3 Outgoing test result queue 530.

[0038] The agent manager 320 can communicate with the agent 201 via a specific protocol (e.g., RPC) and can act as a proxy link, processing events from test stations in the test station pool 201 and converting these events into messages in separate queues 330A through 330C, depending on the type of event from the agent 211. The agent 211 can generate three types of events corresponding to the state of each agent (idle, running, and completed). In addition, the agent manager 320 can provide a specific interface, such as a Hypertext Transfer Protocol (HTTP) interface, to other components of the execution entity 301. Through the HTTP interface, information about individual test stations can be obtained and commands can be executed on them.

[0039] In some embodiments, as described below, various logic units are provided within the test engine 310. Events for the agent 211 may be obtained by separate portions of the test engine 310.

[0040] The test engine 310 may include a preparer (or intervener) 311, a resource allocator 312, an operation cache 313, a test launcher 314A, a test completer 314B, and a test state changer 314C. The operation cache 313 may be a persistent memory that stores data about running test sequences. For example, the data about running test sequences includes the test sequence to be run, the status of the agent (e.g., which agent is currently running), and meta-information associated with the test.

[0041] The preparer / intervenor 311 can receive command messages from the master entity 400 regarding starting / stopping test sequences via the incoming command queue 340A. In other words, the preparer / intervenor 311 can consume the command messages on the incoming command queue 340A and process them. The preparer / intervenor 311 can convert the test sequences and save them in the operation cache 313, and prepare files containing the artifacts required to run the tests on the test stations. Artifacts represent the items involved in the automated testing process (e.g., the exact firmware build to be tested, the package with the test framework, the executable test itself, etc.). The resource allocator 312 can dynamically allocate resources to the test stations. In some embodiments, the resource allocator 312 can allocate resources to the test stations based on their priorities. One embodiment of priority-based dynamic resource allocation is described in U.S. Patent Application Serial No. 16 / 825,721, entitled "Priority-Based Dynamic Resource Allocation for Product Testing," the contents of which are incorporated herein by reference.

[0042] The test initiator 314A can react to idle events from a test station in the test station pool 201, which are stored in the idle queue 330A. The test initiator 314A can select a test to be initiated at a specific station (issuing an idle signal) and form a command to initiate the test. The test state changer 314C can update the test status in the operation cache 313. The test completer 314B can react to completion events from a test station in the test station pool 201. The test completer 314B can obtain test logs from the test stations, analyze and convert the test logs, and provide information about the test logs as test results to the host entity 400 via the external event queue 340B.

[0043] As described above, multiple test stations (each test station corresponding to M multi-agents in the associated test station pool 201) can compete for testing by transmitting test requests (i.e., test initiation events) to the execution entity 301. Upon receiving a test initiation event from the competing test stations, the execution entity 301 can select a test initiation event from among the competing test initiation events for test execution. In some embodiments, the execution entity 301 can select the test initiation event that was first received and stored in the idle queue 330A among the competing test initiation events.

[0044] Figure 7 1 is a diagram illustrating a test execution process according to an embodiment of the present invention. As an example and not a limitation, the test execution process may be Figure 6 The same test execution process can be performed between other agents and the execution entity 301.

[0045] refer to Figure 7The test execution process is triggered by a test activity. In some embodiments, the test activity is generated by an end user via a user interface as part of the master entity 400. In response to the test activity, the master entity 400 may place a command to start the test sequence into the incoming command queue 340A of the execution entity 301 (operation 705).

[0046] The start test sequence command may be processed by the preparer 311. The preparer 311 may prepare the necessary artifacts and place the data associated with the artifacts into the operation cache 313 (operation 710). Thereafter, the test sequence is prepared for execution (operation 715).

[0047] Some test stations in the test station pool 201 may become idle. When a test station becomes idle, the associated agent 211 transmits an event indicating that the test station is in an idle state (ie, an idle event) to the agent manager 320 via the RPC protocol (operation 720).

[0048] Agent manager 320 converts the idle event into a message in idle queue 330A (ie, an idle message). Test launcher 314A processes the idle message by selecting the appropriate test from operation cache 313 and calling the run test HTTP endpoint of agent manager 320 (operation 725).

[0049] The agent manager 320 may execute a run test command on the agent 211 via the RPC protocol (operation 730). Under the control of the agent manager 320, the agent 211 may execute the test (operation 735). Once the test is completed, the agent 211 sends a signal indicating that the event is completed to the agent manager 320 (operation 740). The agent manager 320 may convert the completed event into a message placed in the completion queue 330B. The test completer 314B may process the test log generated by the test and push the test results to the external event queue 340B (operation 745). The test results may be provided to the main entity 400 (operation 750).

[0050] Figure 8 is a diagram illustrating performance of a test system according to an embodiment of the present invention.

[0051] The test system is a software system prototype based on the described architecture and implemented using the NextEra technology stack (C#, Python, .NET Core, MS SQL Server, RabbitMQ, ZeroIce, Docker). The test pool has 4,000 test stations. The regression testing campaign consists of 40,000 tests, each lasting up to 90 seconds.

[0052] According to preliminary performance / load tests, a single instance of the test system allowed to successfully handle 4000 test stations. In this experiment, the test stations were replaced by docker containers connected to a separate single execution entity. As a test load, commands were sent to the incoming queue (i.e., Figure 6 The incoming command queue 340A) is used to run the test. Each test is designed to have an upper bound on the run time (not more, but possibly less than a certain value). Metrics are collected using a specific monitoring stack such as Prometheus and Grafana. The trace is equivalent to Figure 8 The test results in the graph are shown in Figure 2. Figure 8 In

[15] , the performance indicators include pending (unstarted) test (i.e., trace) count 810, running test (i.e., trace) count 820, and the number of tests started per minute (i.e., trace count 830). Figure 8 As can be seen from the graphical results, all tests were assigned, executed, and processed within a specific time (for example, from 18:00 to 18:16, approximately 17 minutes).

[0053] As described above, embodiments provide a solution for achieving horizontal scaling by increasing the number of independent execution entities. Furthermore, embodiments provide a solution to the high concurrency problem by using a solution where test stations compete for tests rather than tests competing for test stations. This solution can significantly reduce concurrency because the number of test stations is always much smaller than the number of tests.

[0054] Although the above embodiments have been illustrated and described in some detail for the purposes of clarity and understanding, the present invention is not limited to the details provided. As will be appreciated by those skilled in the art based on the foregoing disclosure, there are many alternative ways of implementing the present invention. Therefore, the disclosed embodiments are illustrative rather than restrictive. The present invention is intended to encompass all modifications and alternatives that fall within the scope of the claims.

Claims

1. A testing system comprising: A plurality of test stations for testing a plurality of firmware products, wherein the test stations are divided into a plurality of pools, each pool including a plurality of test stations; as well as Multiple execution entities, each of which executes the tests corresponding to the associated pool, Each competing test station in the plurality of test stations transmits a competing test start event to a corresponding execution entity, wherein the corresponding execution entity receives the competing test start event from the competing test stations, and executes a run test command on a selected test station among the competing test stations, so that the selected test station performs test execution based on a test sequence, and Multiple testing stations in each pool compete for testing. 2 . The test system according to claim 1 , wherein the selected test station is a test station among the competing test stations that first transmits a corresponding test start event to the corresponding execution entity. 3 . The test system according to claim 1 , wherein the corresponding execution entity is ready for execution in response to a start test sequence command from a user interface. 4 . The test system of claim 3 , wherein the corresponding execution entity receives a test completion event and a test result from the selected test station, and provides the test result to the user interface in response to the test completion event. 5 . The test system according to claim 4 , wherein the corresponding execution entity comprises an incoming queue for storing the start test sequence command and an external queue for storing the test results. 6 . The test system of claim 1 , wherein the plurality of test stations are divided into the plurality of pools based on set characteristics, wherein the test stations in a given pool have at least one set characteristic in common.

7. The test system of claim 6, wherein the set characteristics include hardware used in a memory system, an operating system used in the memory system, and the amount of available memory and memory devices of the memory system.

8. The test system according to claim 2, wherein the corresponding execution entity comprises: Status queue; an agent manager coupled to the plurality of test stations and configured to store status information about the plurality of test stations in the status queue based on a status of each of the plurality of test stations; as well as The test engine determines the selected test station based on the state information, and calls the agent manager to execute the run test command on the selected test station.

9. The test system of claim 8, wherein each of the plurality of test stations is in an idle state, a running state, or a completed state with respect to the test execution, and The status queue includes an idle queue, a running queue, and a completion queue, which are respectively used to store the idle status, the running status, and the completion status of each of the plurality of test stations.

10. The test system of claim 8, wherein the agent manager is connected to the plurality of test stations via a Remote Procedure Call (RPC) protocol and is connected to the test engine via a Hypertext Transfer Protocol (HTTP) interface.

11. A method of operating a test system, comprising: dividing a plurality of test stations for testing a plurality of firmware products into a plurality of pools, each pool including a plurality of test stations; Each competing test station among the plurality of test stations transmits a competing test initiation event to a corresponding execution entity among a plurality of execution entities, each execution entity executing a test corresponding to an associated pool; receiving the contention test initiation event from the contention test station through the corresponding execution entity; and executing a run test command on a selected test station among the competing test stations through the corresponding execution entity, so that the selected test station performs test execution based on a test sequence; Multiple testing stations in each pool compete for testing. 12 . The method according to claim 11 , wherein the selected test station is a test station among the competing test stations that first transmits a corresponding test initiation event to the corresponding execution entity.

13. The method according to claim 11, wherein the corresponding execution entity is prepared for execution in response to a start test sequence command from a user interface.

14. The method according to claim 13, further comprising: A test completion event and a test result are received from the selected test station by the corresponding execution entity, and the test result is provided to the user interface in response to the test completion event. 15 . The method according to claim 14 , wherein the corresponding execution entity comprises an incoming queue for storing the start test sequence command and an external queue for storing the test results.

16. The method of claim 11, wherein partitioning the plurality of test stations comprises: The plurality of test stations are divided into the plurality of pools based on set characteristics, wherein the test stations in a given pool have at least one set characteristic in common.

17. The method of claim 16, wherein the set characteristics include hardware used in a memory system, an operating system used in the memory system, and the amount of available memory and memory devices of the memory system.

18. The method according to claim 12, wherein the corresponding execution entity comprises: Status queue; an agent manager coupled to the plurality of test stations and configured to store status information about the plurality of test stations in the status queue based on a status of each of the plurality of test stations; as well as The test engine determines the selected test station based on the state information, and calls the agent manager to execute the run test command on the selected test station.

19. The method of claim 18, wherein each of the plurality of test stations is in an idle state, a running state, or a completed state with respect to the test execution, and The status queue includes an idle queue, a running queue, and a completion queue, which are respectively used to store the idle status, the running status, and the completion status of each of the plurality of test stations.

20. The method of claim 18, wherein the agent manager is coupled to the plurality of test stations via a Remote Procedure Call (RPC) protocol and is coupled to the test engine via a Hypertext Transfer Protocol (HTTP) interface.

Images