Hardware automated test self-discovery, self-adaptation, and reporting self-layout system and method

Abstract

The application relates to the technical field of computer automation testing, and discloses a hardware automation testing self-discovery self-adaptation and reporting self-layout system and method. A main control scheduling module automatically scans a test module conforming to a naming convention at startup, self-registration of a test component is completed through dynamic import and reflection extraction, and module plug-and-play type extension is realized. A hardware capability set is defined through an external configuration file, a test case declares a corresponding capability requirement, automatic matching and verification are performed before execution, and if the requirement is not met, the test case is automatically skipped, thereby realizing self-adaptive execution of the test case across hardware platforms. A report layout is defined through an ordered data structure, a general rendering engine generates a basic framework, content updating is completed through key name addressing, and flexible configuration of the report layout is realized. The application effectively improves the expansibility of a test framework, hardware adaptability and report maintenance efficiency, and is suitable for an automation testing scene of network switch type hardware.

Description

Technical Field

[0001] This application relates to the technical field of computer automated testing, and in particular to a hardware automated testing self-discovery, self-adaptation, and self-reporting system and method. Background Technology

[0002] In the field of automated testing of network devices, existing testing frameworks are mostly built based on the software engineering approach of data-driven and layered abstraction. By separating test data such as test parameters and expected results from execution logic code, the reusability and maintainability of test cases are improved. At the same time, by encapsulating independent interaction modules to manage the communication process between the device under test and the test instruments, upper-level test cases can call the underlying capabilities through standardized interfaces without having to worry about the underlying implementation details such as Telnet commands and instrument APIs. To further improve the visualization and archiving capabilities of test results, existing frameworks usually integrate third-party table processing libraries to generate structured test reports and store environment configurations such as device addresses and communication parameters in external configuration files for adjustment.

[0003] However, existing testing frameworks still have several technical limitations when combining the above technologies. At the framework extension level, adding new test function modules requires modifying the core entry code or configuration files for manual registration. The strong coupling between test module extensions and the core framework not only increases maintenance costs and error risks but also requires repackaging and distributing the entire framework. At the hardware adaptation level, test cases are mostly developed for specific hardware platforms. When running on multiple hardware platforms with different function sets, there is a lack of automated adaptation mechanisms. Multiple sets of test case code need to be maintained or complex hardware judgment logic needs to be written within the test cases, resulting in high code redundancy and maintenance difficulty. At the report generation level, the layout structure and content fields of the report are mostly hard-coded in the program logic. Customization operations such as adding or deleting fields and adjusting the order require in-depth modification of the code logic, resulting in low report iteration efficiency and hindering the rapid adaptation and iterative updates of the testing framework. Summary of the Invention

[0004] To address the aforementioned technical issues, this application provides a hardware automated testing self-discovery, adaptive, and self-reporting system and method.

[0005] Firstly, this application provides a hardware automated testing self-discovery, adaptive, and report self-layout system, employing the following technical solution: The hardware automation testing self-discovery, self-adaptation, and self-layout reporting system includes a main control scheduling module, a configuration manager, and several test modules that follow a preset naming convention. The main control scheduling module is configured to automatically scan the preset test case directory to match and obtain the file paths of all test modules that conform to the naming convention when the system starts. It loads each test module file as a module object through a dynamic code import mechanism, and then extracts the corresponding test execution class and test case dataset from each module object according to the naming convention through a reflection mechanism. The extracted components are then registered in the global test suite for test scheduling to call.

[0006] Optionally, the configuration manager is used to load an external configuration file bound to the hardware platform under test, wherein the external configuration file stores the set of hardware capabilities corresponding to the hardware platform under test in the form of key-value pairs. In the test case dataset of each test module, each test case has a capability requirement field, which declares the hardware capabilities required to execute the test case; The main control scheduling module is configured to read the capability requirement field of the current test case before executing each test case, query the actual status of the corresponding hardware capability through the configuration manager, execute the test case when the hardware capability meets the capability requirement, and automatically skip the test case and record the skip status and reason when the hardware capability does not meet the requirement.

[0007] Optionally, it also includes a report generation module, which has a built-in ordered layout data structure, layout rendering engine and key name update interface; The ordered layout data structure is used to define the display text, display order, initial value, and style attributes of all fields within the report page; The layout rendering engine is used to traverse the ordered layout data structure and generate the static base layout of the report in the spreadsheet according to the defined rules. The key name update interface is used to receive the field key name and the content to be updated, automatically calculate the cell coordinates of the corresponding field key name in the spreadsheet, and complete the content filling.

[0008] Optionally, the capability requirement field supports single capability declarations and multi-capability combination declarations. The multi-capability combination declaration includes a logical relationship identifier and a list of capability items. The logical relationship includes AND logic and OR logic, and supports multi-level nested combination. The main control scheduling module matches and verifies multiple hardware capabilities according to the declared logical relationship to generate the final execution decision result.

[0009] Optionally, the ordered layout data structure includes ordinary fields, title fields, table header fields, and placeholder fields, with different types of fields corresponding to a preset number of rows. The key name update interface calculates the row coordinates of the target cell by adding the row occupancy of all fields within the index range based on the index position of the target key name in the ordered layout data structure, and then uses a preset column offset to complete the cell positioning.

[0010] Secondly, this application provides a method for hardware automated testing with self-discovery, self-adaptation, and self-layout of reports, employing the following technical solution: The hardware automation testing self-discovery, self-adaptive, and self-layout report method includes the following steps: S1. Module Scanning and Import: When the system starts, it automatically scans the preset test case directory, matches and obtains all test module files that follow the preset naming convention, and loads each test module file as a module object through a dynamic code import mechanism. S2, Component parsing and registration: Using reflection to extract the corresponding test execution class and test case dataset from each module object according to the preset naming convention, and register the extracted components to the global test suite; S3. Test scheduling and execution: Select the corresponding test component from the global test suite according to the test instructions and execute the corresponding test process.

[0011] Optionally, before executing each test case in step S3, a capability matching step is also included: Read the hardware capability requirements declared in the current test case and compare them with the actual hardware capabilities of the hardware platform under test loaded from the external configuration file. If the actual hardware capabilities meet the capability requirements, execute the test case; otherwise, automatically skip the test case and record the skip information.

[0012] Optionally, it also includes a report generation step, specifically: The complete layout information of the report page is predefined through an ordered data structure, including the key name, display order and display attributes of each field; the ordered data structure is traversed to automatically generate the static basic framework of the report in the spreadsheet; during or after the test, the corresponding cell is automatically located and the test result data is filled in by receiving the update interface of the field key name.

[0013] Optionally, the hardware capability requirements support multi-capability combination declarations, which include logical relationship identifiers and a list of capability items. In the capability matching step, the actual state of the corresponding hardware capability is verified item by item according to the declared AND, OR, and OR logical relationships, and the final matching result is obtained through logical operations.

[0014] Optionally, when locating a cell, first obtain the index position of the target field key name in the ordered data structure, accumulate the preset row occupancy of all layout elements within the index range, obtain the row coordinates of the target cell, and combine them with the preset column offset to complete the precise positioning of the cell.

[0015] In summary, this application includes at least the following beneficial technical effects: This application achieves complete decoupling between test function modules and the core framework through a module self-discovery and self-registration mechanism. Adding a new test module only requires creating the corresponding file according to the naming convention, without modifying the core framework code or configuration. This significantly reduces the maintenance cost and error risk of framework expansion and improves the iteration efficiency of the test framework. Through a hardware adaptive execution mechanism based on capability declarations, the same set of test cases can be directly run on multiple hardware platforms with different functional sets. The framework automatically identifies and skips unsupported test cases, eliminating the need to maintain multiple sets of redundant test case code, thus improving the cross-platform reusability of the test suite and its adaptability to complex product line environments. Through a data-driven dynamic report layout mechanism, the report layout definition and rendering logic are completely separated. Adding, deleting, rearranging, and modifying the style of report fields only requires adjusting the layout data structure, without in-depth modification of the rendering code. This significantly improves the flexibility and maintenance efficiency of report customization, effectively enhancing the scalability, adaptability, and maintainability of the hardware automation testing framework overall. Attached Figure Description

[0016] Figure 1 This is a schematic diagram illustrating the architecture and interaction relationships of the hardware automated testing system described in this application; Figure 2 This is a flowchart illustrating the dynamic discovery and self-registration method of the test module described in this application; Figure 3 This is a schematic diagram of the decision-making process for hardware adaptive execution described in this application; Figure 4 This is a flowchart illustrating the process of generating the dynamic layout of the report described in this application.

[0017] Attached reference numerals: 101, Main control scheduling module; 102, Configuration manager; 103, Test module; 104, Core framework library; 105, Report generation module. Detailed Implementation

[0018] The embodiments of this application are described in detail below, and examples of the embodiments are shown in the accompanying drawings.

[0019] In the description of this specification, the references to "certain embodiments," "one embodiment," "some embodiments," "illustrative embodiment," "example," "specific example," or "some examples" refer to specific features, structures, materials, or characteristics described in connection with the described embodiment or example, which are included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0020] The overall architecture of the automated hardware testing system described in this application is as follows: Figure 1 As shown, the system consists of a main control scheduling module 101, a configuration manager 102, several test modules 103, a core framework library 104, and a report generation module 105.

[0021] The main control scheduling module 101 serves as the system entry point, receiving test start commands and configuration parameters, and coordinating various modules to complete the test process. The configuration manager 102 is responsible for loading and parsing external configuration files, providing interfaces for hardware capability query and environment parameter reading. The test modules 103 are stored in a preset test case directory, with each module corresponding to a hardware function test, including test execution classes, test case datasets, and report configuration information. The core framework library 104 encapsulates the communication interaction interface between the switch under test and the test instruments, providing standardized hardware operation capabilities to the upper layer. The report generation module 105 includes a summary reporter and a detailed reporter, which are responsible for generating test summary reports and test case detail reports, respectively, and finally outputting structured spreadsheet report files.

[0022] The three core mechanisms will be explained in detail below through three examples.

[0023] Example 1: Dynamic discovery and self-registration mechanism for test modules: This embodiment corresponds to Figure 2 The process shown enables plug-and-play expansion of the test module without modifying the core framework code.

[0024] First, a unified naming convention is established as the foundation for module self-discovery. Specifically, test module files are uniformly stored in the `test_cases` directory, with filenames in the format `test_function_identifier.py`, where the function identifier is a string composed of lowercase English letters and underscores. Within each module, test execution classes are named in uppercase `Test + function_identifier` format, test case dataset variables are named in all uppercase `function_identifier + _CASES`, and report configuration variables are named in all uppercase `function_identifier + _REPORT_CONFIG`. For example, a test module with the function identifier "routing" would be named `test_routing.py`, its internal test execution class would be `TestRouting`, its test case dataset would be `ROUTING_CASES`, and its report configuration would be `ROUTING_REPORT_CONFIG`. Those skilled in the art can deduce the naming format for any functional module based on this general rule, achieving standardized module development.

[0025] The complete process of module dynamic discovery and self-registration is as follows: Figure 2 As shown, the specific steps are as follows: Step 201: Develop test module files according to the above naming conventions and store them in the preset test_cases directory.

[0026] Step 202: When the main control scheduling module starts, it calls the directory scanning function to traverse the test_cases directory and matches all test_cases files using filename wildcard rules. A .py file is used to obtain the file paths of all candidate test modules.

[0027] Step 203: Traverse all matched module file paths and determine if there are any unprocessed module files. If so, continue with the subsequent process; otherwise, end the registration process.

[0028] Step 204: For the module file to be processed, deduce its complete module import path and load the module file into a module object in memory through a dynamic code import mechanism. Specifically, the importlib library of the Python language can be used to implement dynamic import.

[0029] Step 205: After the module is loaded, according to the naming convention, the corresponding test execution class, test case dataset, and report configuration information are extracted from the module object using reflection. Specifically, the getattr method can be used to concatenate the corresponding variable name and class name according to the naming convention to obtain the corresponding component from the module object.

[0030] Step 206: Package the extracted test execution class, test case dataset, and report configuration information into a test suite unit, and register it in the global test suite dictionary with the function identifier in all uppercase as the key name for subsequent test scheduling.

[0031] Through the above mechanism, when adding a new hardware function test module, developers only need to create the corresponding file and put it in the specified directory according to the naming convention. The framework can automatically recognize and complete the registration when it starts, without modifying the core code of the main control scheduling module, thus achieving complete decoupling between the test module and the core of the framework.

[0032] Example 2: Hardware Adaptive Execution Mechanism Based on Capability Declaration: This embodiment corresponds to Figure 3 The process shown enables the intelligent adaptation and execution of the same set of test cases on different hardware platforms.

[0033] First, hardware capabilities are defined through an external configuration file bound to the hardware platform. This configuration file uses the INI format and contains a `[capabilities]` field, declaring the various functional capabilities supported by the hardware platform in key-value pairs. The key is a capability identifier string, and a value of 1 indicates support for the capability, while a value of 0 indicates no support. Different hardware platforms correspond to different configuration files. For example, the configuration file for chip model A declares `support_acl_egress=1` and `support_qos_scheduler_wrr=1`, while the configuration file for chip model B declares `support_acl_egress=0` and `support_qos_scheduler_wrr=1`. During system startup, the main control scheduling module passes the path to the configuration file corresponding to the currently tested hardware as a startup parameter. The configuration manager loads and parses this configuration file, providing a hardware capability query interface.

[0034] Secondly, each test case's data structure includes a `required_capability` field to declare the hardware capabilities required to execute the test case. Capability requirements support two declaration formats: single-capability declarations directly pass the capability identifier string; multi-capability combination declarations are dictionary structures containing `logic` and `items` fields. The `logic` field takes the value AND or OR, indicating that all capability items must be satisfied or that any one capability item must be satisfied, respectively. The `items` field is a list of capability items, where each element can be a single capability identifier or a nested combination declaration structure, supporting multiple levels of logical nesting. For example, declaring the requirement to support both outbound and inbound ACLs simultaneously can be represented as `{"logic":"AND","items":["support_acl_egress","support_acl_ingress"]}`.

[0035] The decision-making process for hardware adaptive execution is as follows: Figure 3 As shown, the specific steps are as follows: Step 301: Define the set of hardware capabilities for the corresponding hardware platform in the external configuration file in advance, and load and parse it by the configuration manager when the system starts.

[0036] Step 302: In the data definition of each test case, declare the hardware capabilities required for execution through the capability requirements field.

[0037] Step 303: Before scheduling the execution of a single test case, the execution engine calls the capability verification method to read the capability requirement declaration of the current test case.

[0038] Step 304: Iterate through each capability unit in the capability requirements and verify each one.

[0039] Step 305: For the current capability item, query the actual value of the corresponding hardware capability through the configuration manager.

[0040] Step 306: Based on the logical relationships in the capability requirement declaration, determine whether all capability items meet the requirements.

[0041] If the logic is AND, all capability items must be satisfied; if it's OR, only one capability item needs to be satisfied. Nested structures execute logical judgments recursively according to their hierarchy. If all capability requirements are met, the subsequent test steps for that test case are executed. If the capability requirements are not met, all execution steps for that test case are skipped, the skipped status and the reason for the mismatch are written to the report generation module, and the next test case is processed. After traversing all capability items and completing all verifications, the final execution decision is output.

[0042] Through the above mechanism, the test suite can automatically tailor the execution scope according to the actual capabilities of the current hardware platform. There is no need to maintain multiple sets of test cases for different hardware platforms, nor is there a need to write hardware judgment logic in the test case code, which greatly improves the cross-platform reusability of test cases.

[0043] Example 3: Data-driven dynamic report layout mechanism: This embodiment corresponds to Figure 4 The process shown enables flexible configuration and maintenance of the report layout, and layout adjustments do not require modification of the rendering logic.

[0044] The report generation module includes a summary reporter and a detail reporter, both of which employ a data-driven dynamic layout mechanism. The only difference lies in the definition of the layout data structure; the rendering and update logic is completely interchangeable. This embodiment uses the summary report as an example for illustration; the implementation principle of the detail report is the same, differing only in the layout fields and data sources.

[0045] First, in the summary reportr of the report generation module, an ordered layout data structure is defined. Specifically, an ordered dictionary structure can be used to store the data, ensuring the order of fields. Each entry in the layout data structure uses a key name with a clear business meaning as its key, and the value is a list containing display text, initial values, style attributes, or a special type identifier. Layout elements are divided into four categories: ordinary fields, occupying one row, containing a label column and a content column; title fields, occupying one row, merging cells across columns, and applying title styles; table header fields, occupying one row, applying table header styles; and placeholder fields, occupying one row, used for layout adjustments, without actual content. For example, the layout structure of a summary report includes, in sequence, title fields, placeholder fields, environment header fields, framework version ordinary fields, and product model ordinary fields, etc., with each entry arranged in order to define the complete page structure of the report.

[0046] The process of generating and updating the dynamic layout of the report is as follows: Figure 4As shown, the specific steps are as follows: Step 401: During the reporter initialization phase, a complete ordered layout data structure is predefined, specifying the key name, display text, initial value, style, and type of all fields.

[0047] Step 402: Invoke the layout rendering engine to traverse all entries in the ordered layout data structure and generate the Excel worksheet content row by row in sequence. Perform corresponding rendering operations for different types of entries: merge cells within a specified range for the title field, write the title text and apply the title style; write the display text in the label column and the initial value in the content column and apply the corresponding style for ordinary fields; skip the current row for placeholder fields and leave them blank. After rendering, the static basic framework of the report is formed.

[0048] Step 403, during the test run phase, when the report content needs to be updated, call the key name update interface, passing in the key name of the target field and the content value to be updated.

[0049] Step 404: The interface internally performs coordinate calculations: First, it obtains the index position of the target key name in the ordered layout data structure. Then, it iterates through all layout entries before the index position, accumulating the row occupancy of each entry to obtain the row number corresponding to the target field content column. Next, it combines this with the preset fixed column offset of the content column to obtain the precise coordinates of the target cell. After the calculation is completed, the updated content is written to the corresponding cell, and the preset cell style is applied to complete the content update.

[0050] The formula for calculating the row coordinates is as follows: ; In the formula, R is the row number of the target cell, and i is the index of the target key name in the ordered layout structure. This represents the number of rows occupied by the k-th layout element. The number of rows occupied by regular fields, title fields, header fields, and placeholder fields is preset to 1. If there are elements spanning multiple lines, the value of the row occupancy number is adjusted accordingly. Those skilled in the art can use this formula, combined with the specific layout element type, to achieve accurate row number calculation.

[0051] For detailed reports, each test module has an independent detailed layout configuration. When rendering the detailed page, the report generation module loads the report configuration information of each test module in sequence and generates the corresponding detailed area according to the same rendering logic, realizing the dynamic expansion of the detailed report and forming a complete technical closed loop with the module self-discovery mechanism.

[0052] Through the above mechanism, the structure and content of the report are completely defined by the layout data structure. When the report layout needs to be adjusted, only the order of the items in the layout data structure, the addition or deletion of fields, or the style need to be modified. There is no need to modify the code logic of the rendering engine and the update interface, which greatly reduces the difficulty and maintenance cost of report customization.

[0053] Although embodiments of this application have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting this application. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of this application.

Claims

1. A hardware automated testing self-discovery, adaptive, and report self-layout system, characterized in that: It includes a main control scheduling module, a configuration manager, a report generation module, and several test modules that follow a preset naming convention; The main control scheduling module is configured to automatically scan the preset test case directory to match and obtain all test module file paths that conform to the naming convention when the system starts. It loads each test module file as a module object through a dynamic code import mechanism, and then extracts the corresponding test execution class and test case dataset from each module object according to the naming convention through a reflection mechanism. The extracted components are then registered in the global test suite for test scheduling to call. The configuration manager is used to load external configuration files bound to the hardware platform under test. The external configuration files store the set of hardware capabilities corresponding to the hardware platform under test in key-value pairs. In the test case dataset of each test module, each test case has a capability requirement field, which declares the hardware capabilities required to execute the test case; The main control scheduling module is configured to read the capability requirement field of the current test case before executing each test case, query the actual status of the corresponding hardware capability through the configuration manager, execute the test case when the hardware capability meets the capability requirement, and automatically skip the test case and record the skip status and reason when the hardware capability does not meet the requirement. The report generation module has a built-in ordered layout data structure, layout rendering engine and key name update interface; The ordered layout data structure is used to define the display text, display order, initial value, and style attributes of all fields within the report page; The layout rendering engine is used to traverse the ordered layout data structure and generate the static base layout of the report in the spreadsheet according to the defined rules. The key name update interface is used to receive the field key name and the content to be updated, automatically calculate the cell coordinates of the corresponding field key name in the spreadsheet, and complete the content filling.

2. The hardware automated testing self-discovery, adaptive, and report self-layout system according to claim 1, characterized in that, The capability requirement field supports single capability declarations and multi-capability combination declarations. The multi-capability combination declaration includes a logical relationship identifier and a list of capability items. The logical relationship includes AND and OR logic, and supports multi-level nested combinations. The main control scheduling module matches and verifies multiple hardware capabilities based on the declared logical relationship to generate the final execution decision result.

3. The hardware automated testing self-discovery, adaptive, and report self-layout system according to claim 1, characterized in that, The ordered layout data structure includes ordinary fields, title fields, table header fields, and placeholder fields, with different types of fields corresponding to a preset number of rows. The key name update interface calculates the row coordinates of the target cell by adding the row occupancy of all fields within the index range based on the index position of the target key name in the ordered layout data structure, and then uses a preset column offset to complete the cell positioning.

4. A hardware automated test self-discovery, adaptive, and self-layout reporting method, based on the hardware automated test self-discovery, adaptive, and self-layout reporting system as described in any one of claims 1-3, characterized in that, Includes the following steps: S1. Module Scanning and Import: When the system starts, it automatically scans the preset test case directory, matches and obtains all test module files that follow the preset naming convention, and loads each test module file as a module object through a dynamic code import mechanism. S2, Component parsing and registration: Using reflection to extract the corresponding test execution class and test case dataset from each module object according to the preset naming convention, and register the extracted components to the global test suite; S3. Test scheduling and execution: Select the corresponding test component from the global test suite according to the test instructions and execute the corresponding test process.

5. The hardware automated testing self-discovery, adaptive, and report self-layout method according to claim 4, characterized in that, Before executing each test case in step S3, a capability matching step is also included: Read the hardware capability requirements declared in the current test case and compare them with the actual hardware capabilities of the hardware platform under test loaded from the external configuration file. If the actual hardware capabilities meet the capability requirements, execute the test case; otherwise, automatically skip the test case and record the skip information.

6. The hardware automated testing self-discovery, adaptive, and report self-layout method according to claim 5, characterized in that, It also includes the report generation step, specifically: The complete layout information of the report page is predefined through an ordered data structure, including the key name, display order and display attributes of each field; the ordered data structure is traversed to automatically generate the static basic framework of the report in the spreadsheet; during or after the test, the corresponding cell is automatically located and the test result data is filled in by receiving the update interface of the field key name.

7. The hardware automated testing self-discovery, adaptive, and report self-layout method according to claim 6, characterized in that, The hardware capability requirements support multiple capability combination declarations, which include logical relationship identifiers and a list of capability items. In the capability matching step, the actual state of the corresponding hardware capability is verified item by item according to the declared AND, OR, and OR logical relationships, and the final matching result is obtained through logical operations.

8. The hardware automated testing self-discovery, adaptive, and report self-layout method according to claim 6, characterized in that, When locating a cell, first obtain the index position of the target field key name in the ordered data structure, then add the preset row occupancy of all layout elements within the index range to obtain the row coordinates of the target cell, and finally complete the precise location of the cell by combining the preset column offset.

Images