Benchmarking method and apparatus for software and hardware product base environment

Abstract

This invention discloses a benchmark testing method and apparatus for the basic environment of software and hardware products, relating to the field of automated programming. The method includes: creating a test plan for benchmarking the basic environment; selecting one or more test machines from a resource pool to execute the test plan; selecting test cases to be executed on each test machine from a test case set according to the test plan; controlling each test machine to execute the corresponding test cases and obtaining the test results of each test machine; and generating a test analysis report based on the test results of each test machine. This invention not only enables the sharing of test resources among different test projects and testers but also significantly improves the testing efficiency of the basic environment of software and hardware products.

Description

Technical Field

[0001] This invention relates to the field of automatic programming, and more particularly to a benchmark testing method and apparatus for the basic environment of software and hardware products. Background Technology

[0002] This section is intended to provide background or context for the embodiments of the invention set forth in the claims. The description herein is not an admission that it is prior art simply because it is included in this section.

[0003] Currently, there are almost no benchmark testing software specifically designed for basic environments on the market. How to quickly and efficiently conduct comprehensive evaluations of different software and hardware of the same type is a technical problem that urgently needs to be solved in this field. Summary of the Invention

[0004] This invention provides a benchmark testing method for the basic environment of software and hardware products, which enables rapid and efficient comprehensive evaluation of different software and hardware products of the same type. The method includes: creating a test plan for benchmark testing of the basic environment; selecting one or more test machines from a resource pool to execute the test plan; selecting test cases to be executed on each test machine from a test case set according to the test plan; controlling each test machine to execute the corresponding test cases and obtaining the test results of each test machine; and generating a test analysis report based on the test results of each test machine.

[0005] This invention also provides a benchmark testing device for the basic environment of software and hardware products, which enables rapid and efficient comprehensive evaluation of different software and hardware products of the same type. The device includes: a test plan creation module for creating a test plan for benchmark testing of the basic environment; a test machine determination module for selecting one or more test machines from a resource pool to execute the test plan; a test case determination module for selecting test cases to be executed on each test machine from a case set according to the test plan; a testing module for controlling each test machine to execute the corresponding test cases and obtaining the test results of each test machine; and a test result analysis module for generating test analysis reports based on the test results of each test machine.

[0006] This invention also provides an electronic device for rapidly and efficiently evaluating different software and hardware products of the same type. The electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the benchmark testing method for the basic environment of the aforementioned software and hardware products.

[0007] This invention also provides a computer-readable storage medium for rapidly and efficiently evaluating different software and hardware products of the same type. The computer-readable storage medium stores a computer program that executes a benchmark testing method for the basic environment of the aforementioned software and hardware products.

[0008] The benchmark testing method, apparatus, computer equipment, and computer-readable storage medium for the basic environment of software and hardware products provided in this embodiment of the invention reconstruct the benchmark testing framework, uniformly manage various test resources such as test cases, test machines, test plans, and test results, and establish a resource pool containing multiple test machines and a case set containing multiple test cases in advance. After creating a test plan for benchmark testing of the basic environment, one or more test machines are selected from the resource pool to execute the test plan. Then, according to the test plan, test cases are selected from the case set to be executed on each test machine. Each test machine is then controlled to execute the corresponding test cases, and the test results of each test machine are obtained. Finally, a test analysis report is generated based on the test results of each test machine.

[0009] Through the embodiments of the present invention, not only can test resources be shared among different test projects and testers, but the testing efficiency of the basic environment of software and hardware products can also be greatly improved. Attached Figure Description

[0010] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. In the drawings:

[0011] Figure 1 This is a flowchart of a benchmark testing method for a basic environment of a software and hardware product provided in an embodiment of the present invention;

[0012] Figure 2 This is a flowchart illustrating a test result storage process provided in an embodiment of the present invention;

[0013] Figure 3 This is a flowchart of a test result analysis provided in an embodiment of the present invention;

[0014] Figure 4 This is a flowchart of a synchronous testing process provided in an embodiment of the present invention;

[0015] Figure 5 This is a schematic diagram of a test framework provided in an embodiment of the present invention;

[0016] Figure 6This is a flowchart illustrating the execution of a benchmark test provided in an embodiment of the present invention;

[0017] Figure 7 This is a schematic diagram of a test case data model provided in an embodiment of the present invention;

[0018] Figure 8 This is a schematic diagram illustrating a classification of test indicators provided in an embodiment of the present invention;

[0019] Figure 9 This is a schematic diagram of an XML test result file structure provided in an embodiment of the present invention;

[0020] Figure 10 This is a flowchart illustrating the analysis of benchmark test results provided in an embodiment of the present invention;

[0021] Figure 11 This is a schematic diagram of a benchmark testing device for a basic environment of software and hardware products provided in an embodiment of the present invention;

[0022] Figure 12 This is a schematic diagram of a benchmark testing device for an optional software and hardware product basic environment provided in an embodiment of the present invention.

[0023] Figure 13 This is a schematic diagram of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0024] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the embodiments of the present invention will be further described in detail below with reference to the accompanying drawings. Here, the illustrative embodiments of the present invention and their descriptions are used to explain the present invention, but are not intended to limit the present invention.

[0025] Before introducing the embodiments of the present invention, the terms and concepts involved in the embodiments of the present invention are explained as follows:

[0026] Basic Environment: In the process of domestic self-reliance and controllability substitution of China Construction Bank, the basic environment involves IBM system architecture hardware and software, Intel system architecture hardware, Oracle hardware and software, domestic CPU server hardware, domestic server software, open source software, etc. Here, it refers to all hardware equipment, application software, server systems, big data platforms, cloud computing platforms, etc. in the development environment, testing environment, and production environment of an enterprise or organization.

[0027] Benchmarking: Through the design of scientific testing methods, tools, and systems, quantitative and comparable testing of a specific performance indicator of a class of test objects is achieved. Hardware and software selection testing.

[0028] Test Program: A comprehensive testing and benchmarking platform for the Linux operating system. This test management platform automates the entire testing process, from writing and updating test cases, remotely distributing test cases and test case sets, managing remote systems under test, scheduling test execution, generating test reports, saving test results, searching test results, and benchmarking.

[0029] Test case: A standalone test, with a definition file for the test's installation, dependencies, execution, and results, saved in XML file format.

[0030] Test Case Set: A collection of test cases. When performing tests on the system under test, a test case or a test set can be issued. The test set is executed sequentially on the system under test.

[0031] Case classification: Classification of test cases based on a certain type of hardware or software of the system being tested.

[0032] Local load testing: Test cases are executed on the machine under test. This execution method may be required for some hardware and software resource load tests. The test program and the system resources under test associated with the test are both located on the local machine.

[0033] Elastic stress: The elastic stress generator operates in a container cluster. When the test program needs to increase the load on the system under test, it needs to schedule the container cluster and specify instance replicas or worker threads to the stress machine service working in it.

[0034] Remote Load: The test case is executed on a remote elastic stress generator and is suitable for network-accessible applications.

[0035] This invention provides a benchmark testing method for the basic environment of a software and hardware product. Figure 1 This is a flowchart of a benchmark testing method for a basic environment of a software and hardware product provided in an embodiment of the present invention, such as... Figure 1 As shown, it includes:

[0036] S101, Create a test plan to benchmark the base environment.

[0037] It should be noted that the basic environment in the embodiments of the present invention can be any software or hardware product's development environment, testing environment, or production environment. The test associations in the embodiments of the present invention include test cases, test machines, and test results for testing the basic environment.

[0038] S102, Select one or more test machines from the resource pool to execute the test plan.

[0039] It should be noted that the resource pool in this embodiment of the invention refers to a resource pool composed of multiple computer devices that can serve as test machines. Before implementation, the available computer devices are placed in the resource pool so that when executing a test plan, one or more computer devices can be selected from the resource pool as test machines.

[0040] To achieve the goal of synchronously controlling multiple test machines for testing, this embodiment of the invention adopts a C / S architecture test framework, with a test client installed on each test machine to execute corresponding tests according to the synchronous commands issued by the test server.

[0041] In one embodiment, the above S102 can be implemented by the following steps: checking if there is a test machine in the resource pool that executes the test plan; if there is no test machine in the resource pool that executes the test plan, adding a test machine to the resource pool that executes the test plan.

[0042] S103, based on the test plan, select test cases from the test case set to be executed on each test machine.

[0043] It should be noted that the test case set in this embodiment of the invention refers to a test case set consisting of multiple pre-created standardized test cases. In one embodiment, the above S103 can be implemented through the following steps: checking whether there are test cases with test plans in the test case set; if there are test cases with test plans in the test case set, adding test cases with test plans to the test case set.

[0044] S104 controls each test machine to execute the corresponding test cases and obtains the test results of each test machine.

[0045] After selecting test cases to be executed on each test machine from the test case set, control each test machine to execute the corresponding test cases according to the test plan, and obtain the test results returned by each test machine.

[0046] S105 generates test analysis reports based on the test results of each tester.

[0047] It should be noted that test result analysis often requires testers to spend a lot of time on data processing. In this embodiment of the invention, test analysis reports are generated based on the test results of each test machine, which can realize automated analysis of test results, greatly improve the efficiency of test result analysis, and reduce the error rate of manual analysis.

[0048] In one embodiment, after generating a test analysis report based on the test results of each test machine, the benchmark testing method for the basic environment of the software and hardware product provided in this embodiment of the invention may further include the following steps: visually displaying the generated test analysis report in the form of charts.

[0049] In one embodiment, the benchmark testing method for the basic environment of the software and hardware product provided in this embodiment of the invention may further include the following steps: defining the data model used for the test cases, wherein the data model adopts a three-level tree structure.

[0050] In this embodiment of the invention, a data model that defines benchmark test case rules facilitates test case management and allows testers to quickly generate test cases, which are then integrated into the standard processing flow of the test framework.

[0051] Furthermore, in one embodiment, the benchmark testing method for the basic environment of software and hardware products provided in this embodiment of the invention may further include the following steps: defining test metrics used in test cases, wherein the test metrics include: framework-defined metrics and custom metrics.

[0052] In this embodiment of the invention, by sorting out various benchmark indicators, a technical solution is designed to facilitate the statistics, storage, calculation and display of indicators, that is, to realize a data structure and related processing algorithm suitable for the persistent storage and analysis of benchmark indicators.

[0053] In one embodiment, such as Figure 2 As shown, after controlling each test machine to execute the corresponding test cases and obtaining the test results of each test machine, the benchmark testing method for the basic environment of the software and hardware product provided in this embodiment of the invention may further include the following steps:

[0054] S201 uses an XML format file to store the test results, resulting in an XML format test result file.

[0055] Optionally, such as Figure 2 As shown, after storing the test results in an XML format file and obtaining the test result file in XML format, the benchmark testing method for the basic environment of the software and hardware product provided in this embodiment of the invention may further include the following steps:

[0056] S202, stores the test result file in XML format to a network-attached storage (NAS) device;

[0057] S203 stores the basic information of the test result file and the NAS storage path information into a relational database. The NAS storage path information is the file storage path information of the test result file in the NAS device.

[0058] In one embodiment, such as Figure 3 As shown, the benchmark testing method for the basic environment of software and hardware products provided in this embodiment of the invention can also generate test analysis reports through the following steps:

[0059] S301 reads the test result file obtained from each benchmark test;

[0060] S302, Verify the test result file obtained from each benchmark test;

[0061] S303, extract data from the test result file that has passed verification;

[0062] S304. Based on the data extraction results, calculate the maximum, minimum, and average values ​​of each test indicator.

[0063] S305 generates a test analysis report for each benchmark test based on the maximum, minimum, and average values ​​of each test indicator.

[0064] In one embodiment, such as Figure 4 As shown, the benchmark testing method for the basic environment of software and hardware products provided in this embodiment of the invention can also control each test machine to perform tests through the following steps:

[0065] S401 sends test synchronization commands to each test machine;

[0066] S402 controls each test machine to execute the corresponding test cases synchronously according to the test synchronization command;

[0067] S403 receives the test results returned by each test machine.

[0068] In practical implementation, a C / S testing architecture can be used to allow benchmark tests on multiple test machines to run simultaneously through a server, and the results can be aggregated, compared horizontally, and displayed.

[0069] By reconstructing the benchmark testing framework, we can unify the management of various testing resources, including test cases, test machines, test plans, and test results, thereby increasing the automation and integration of test management and allowing testers to focus their main work on the business itself. By standardizing the data models of test cases, test metrics, and test results, we can enable these testing resources to be shared among different test projects and testers.

[0070] By reconstructing the benchmark testing framework, we can unify the management of various test resources (including test cases, test machines, test plans, and test results), and increase the automation and integration of test management. By standardizing the data models of test cases, test metrics, and test results, we can enable these test resources to be shared among different test projects and testers. Finally, we can improve analysis efficiency by automating the results analysis process.

[0071] This application provides a versatile and flexibly configurable benchmark testing software suitable for various enterprises or organizations to select or evaluate hardware and software for development, testing, and production environments.

[0072] Specifically, it includes:

[0073] This application restructures the benchmark testing framework to unify the management of various testing resources, including test cases, test machines, test plans, and test results, thereby increasing the automation and integration of test management. By standardizing the data models of test cases, test metrics, and test results, these testing resources can be shared among different test projects and testers. Furthermore, by automating the result analysis process, analysis efficiency can be greatly improved, and the error rate of manual analysis can be reduced.

[0074] As can be seen from the above, the benchmark testing method for the basic environment of software and hardware products provided in the embodiments of the present invention reconstructs the benchmark testing framework, unifies the management of various test resources such as test cases, test machines, test plans, and test results, and establishes a resource pool containing multiple test machines and a case set containing multiple test cases in advance. After creating a test plan for benchmark testing of the basic environment, one or more test machines are selected from the resource pool to execute the test plan. Then, according to the test plan, test cases to be executed on each test machine are selected from the case set. Each test machine is then controlled to execute the corresponding test cases, and the test results of each test machine are obtained. Finally, a test analysis report is generated based on the test results of each test machine.

[0075] The benchmark testing method for the basic environment of software and hardware products provided by the embodiments of the present invention not only enables the sharing of testing resources among different testing projects and testers, but also greatly improves the testing efficiency of the basic environment of software and hardware products.

[0076] To ensure the efficiency and versatility of the benchmark testing framework, this embodiment of the invention abandons the traditional single-type testing framework and redesigns the overall architecture, proposing an improved C / S testing architecture, such as... Figure 5 As shown, it mainly includes five aspects:

[0077] 1) Adopting a server / client architecture, the server can simultaneously perform test management on multiple candidate objects.

[0078] 2) By executing the coordinator middleware, the connection of all clients, task execution progress and test results can be controlled and statistically analyzed quickly and efficiently, while reducing the workload of the core server.

[0079] 3) Introduce a key center to provide a secure and fast client access method.

[0080] 4) By using NAS storage to share test cases, test results, and analysis reports between the server and the test machine, the consumption of network bandwidth is reduced, and the execution efficiency of the test cases is greatly improved.

[0081] 5) Decoupling of result management, result analysis and report management facilitates the reuse of test results and reduces redundant testing during the selection process.

[0082] Figure 6 This is a flowchart illustrating the execution of a benchmark test as provided in an embodiment of the present invention, such as... Figure 6 As shown, the benchmark test execution process designed in this embodiment of the invention connects various test elements such as test cases, test machines, and test results through a test plan, including creating a test plan and generating a test analysis report.

[0083] Benchmarking is characterized by a wide range of test objects, a large quantity of test objects, and few commonalities. Therefore, ensuring compatibility between different test objects within the same testing framework is a primary challenge. This solution defines a set of standard test case representation rules to address compatibility issues between the framework and different test objects. This is the basic data model for benchmark test cases, comprising three dimensions: "Basic Information" (TestInfo), "Test Definition" (TestProfile), and "Test Settings" (TestSetting).

[0084] Figure 7 This illustrates a test case data model provided in an embodiment of the present invention, such as... Figure 7 As shown, the three-layer tree structure defines the basic data model for benchmark test cases. This model acts as an adaptation layer, adapting different types of tests to the benchmark testing framework for unified test management. Tables 1, 2, and 3 list detailed information for each type:

[0085] Table 1: Details of Basic Information Fields

[0086] Name Case Name Required Test Case Name Version Version number Required Test case version Description Description information Optional Basic Explanation of the Case Maintainer maintainers Optional To facilitate testers in contacting case designers Status state Optional Mark obsolete test case versions

[0087] Table 2 Test Definition Field Details

[0088]

[0089] Table 3 Test Settings Field Details

[0090] Argument Execution parameters Optional Passing parameters when executing test cases Options Execution Options Optional Different execution scenarios can be selected when executing test cases.

[0091] The tree-structured test case data model defined in this embodiment of the invention not only establishes a universal adaptation layer between the test framework and the test objects, but also its clear hierarchical logic helps test personnel quickly grasp the test cases. Test developers can quickly define test cases (including creating new ones and migrating from test cases in other models), and test execution personnel can also relatively easily grasp the core points of the test cases.

[0092] In this embodiment of the invention, test metrics are divided into two categories: framework-defined metrics and custom metrics. Framework-defined metrics standardize general benchmark test metrics, which can be directly used by test cases. Custom metrics are specific metrics based on test cases, and test cases need to add a metric definition template before using these metrics. Figure 8 The diagram shows the specific types of test metrics. Framework-defined metrics are divided into three categories: system resource metrics, including "CPU Load Line," "Memory Usage," "IO Frequency," and "Network Usage"; process metrics, including "Execution Time" and "Execution Result"; and log / alarm metrics, referring to "Warnings" and "Errors" occurring during test execution. Conversely, custom metrics accept any metric type defined in the metric definition template for test cases, and the test framework automatically loads custom metrics during test case execution initialization.

[0093] Since test comparison analysis can only analyze and compare results for the same type of test cases, and there is a one-to-many relationship between test plans and test results, the way test result data is stored directly affects the overall framework design. To facilitate the sharing and transmission of test result data between the service backend and the client, and to minimize the coupling between the result management module and other functional modules, this solution uses XML files combined with a relational database for storing and managing the result data. The XML test result file structure is as follows: Figure 9 As shown, the XML result file is stored in NAS storage, accessible to both the service backend and the client. A relational database stores basic information about the test results and their NAS storage paths, with corresponding records linked to the test plan, facilitating queries and comparative analysis by testers in the console.

[0094] The benchmark testing method for the basic environment of software and hardware products provided in this embodiment of the invention can support one-click test report generation. The most important thing in the report generation process is to perform reasonable and efficient analysis and calculation of multiple test results, and then present them to the testers in an intuitive chart format.

[0095] Figure 10 This is a flowchart illustrating the analysis of benchmark test results provided in an embodiment of the present invention, such as... Figure 10As shown, the benchmark test result analysis is divided into five stages from input to output. The "Input" stage reads the result files sequentially. The "Preprocessing" stage mainly verifies the files from the "Input" stage and extracts and marks the data. "Data Integration" calculates the maximum, minimum, and average values ​​of each indicator for each test based on the data extracted in the previous stage. "Report Generation" mainly involves creating graphs and integrating explanatory information from the indicator templates to generate reports. Finally, the output is a file in the format expected by the testers.

[0096] The benchmark testing method for the basic environment of software and hardware products provided in this embodiment of the invention can generate a highly versatile, flexibly configurable benchmark testing software suitable for various enterprises or organizations to select or evaluate software and hardware for development testing and production environments. It evaluates and benchmarks all aspects of open-source software on domestically produced ARM chips. It automatically distributes and tests test cases for basic environment software patch updates, patch set updates, operating system upgrades, etc., and generates result analysis reports.

[0097] Based on the same inventive concept, this invention also provides a benchmark testing device for the basic environment of software and hardware products, as described in the following embodiments. Since the principle by which this device solves the problem is similar to the benchmark testing method for the basic environment of software and hardware products, the implementation of this device can refer to the implementation of the benchmark testing method for the basic environment of software and hardware products; repeated details will not be elaborated further.

[0098] Figure 11 This is a schematic diagram of a benchmark testing device for a basic environment of software and hardware products provided in an embodiment of the present invention, as shown below. Figure 11 As shown, the device includes: a test plan creation module 111, a test machine determination module 112, a test case determination module 113, a test module 114, and a test result analysis module 115.

[0099] The test plan creation module 111 is used to create a test plan for benchmark testing of the basic environment; the test machine determination module 112 is used to select one or more test machines from the resource pool to execute the test plan; the test case determination module 113 is used to select test cases to be executed on each test machine from the case set according to the test plan; the test module 114 is used to control each test machine to execute the corresponding test cases and obtain the test results of each test machine; and the test result analysis module 115 is used to generate test analysis reports based on the test results of each test machine.

[0100] It should be noted that the test plan creation module 111, test machine determination module 112, test case determination module 113, test module 114, and test result analysis module 115 mentioned above correspond to S101 to S105 in the method embodiment. The examples and application scenarios implemented by the above modules and corresponding steps are the same, but are not limited to the content disclosed in the above method embodiment. It should be noted that the above modules, as part of the apparatus, can be executed in a computer system such as a set of computer-executable instructions.

[0101] In one embodiment, in the benchmark testing device for the basic environment of software and hardware products provided in this embodiment of the invention, the test machine determination module 112 is further configured to: search whether there is a test machine in the resource pool that executes the test plan; and add a test machine to the resource pool that executes the test plan when there is no test machine in the resource pool that executes the test plan.

[0102] In one embodiment, in the benchmark testing device for the basic environment of software and hardware products provided in this embodiment of the invention, the test case determination module 113 is further configured to: search whether there are test cases with test plans in the case set; and add test cases with test plans to the case set when there are test cases with test plans in the case set.

[0103] In one embodiment, such as Figure 12 As shown, the benchmark testing device for the basic environment of software and hardware products provided in this embodiment of the invention further includes: a test case data model definition module 116, used to define the data model used for test cases, wherein the data model adopts a three-level tree structure.

[0104] In one embodiment, such as Figure 12 As shown, the benchmark testing device for the basic environment of software and hardware products provided in this embodiment of the invention further includes: a test index definition module 117, used to define the test indexes used in test cases, wherein the test indexes include: framework-defined indexes and custom indexes.

[0105] In one embodiment, such as Figure 12 As shown, the benchmark testing device for the basic environment of software and hardware products provided in this embodiment of the invention further includes: a test result storage module 118, which is used to store test results in XML format to obtain test result files in XML format.

[0106] In one embodiment, in the benchmark testing device for the basic environment of software and hardware products provided in this embodiment of the invention, the test result storage module 118 is further configured to: store the test result file in XML format in a network attached storage (NAS) device; and store the basic information of the test result file and the NAS storage path information in a relational database, wherein the NAS storage path information is the file storage path information of the test result file stored in the NAS device.

[0107] In one embodiment, such as Figure 13 As shown, in the benchmark testing device for the basic environment of software and hardware products provided in this embodiment of the invention, the test result analysis module 115 specifically includes: a data reading unit 1151, a verification unit 1152, a data extraction unit 1153, a data statistical extraction unit 1154, and a data output unit 1155.

[0108] The data reading unit 1151 is used to read the test result file obtained from each benchmark test; the verification unit 1152 is used to verify the test result file obtained from each benchmark test; the data extraction unit 1153 is used to extract data from the verified test result file; the data statistics unit 1154 is used to calculate the maximum, minimum and average values ​​of each test indicator based on the data extraction results; and the data output unit 1155 is used to generate a test analysis report for each benchmark test based on the maximum, minimum and average values ​​of each test indicator.

[0109] It should be noted that the data reading unit 1151, verification unit 1152, data extraction unit 1153, data statistics unit 1154, and data output unit 1155 mentioned above correspond to S101 to S105 in the method embodiment. The examples and application scenarios implemented by the above modules and corresponding steps are the same, but are not limited to the content disclosed in the above method embodiment. It should be noted that the above modules, as part of the device, can be executed in a computer system such as a set of computer-executable instructions.

[0110] In one embodiment, such as Figure 12 As shown, the benchmark testing device for the basic environment of software and hardware products provided in this embodiment of the invention further includes: a visualization module 119, used to visualize and display the generated test analysis report in the form of charts.

[0111] In one embodiment, in the benchmark testing device for the basic environment of software and hardware products provided in this embodiment of the invention, the testing module 114 is further configured to: issue test synchronization instructions to each test machine; control each test machine to synchronously execute corresponding test cases according to the test synchronization instructions; and receive test results returned by each test machine.

[0112] As can be seen from the above, the benchmark testing device for the basic environment of software and hardware products provided in the embodiments of the present invention, by reconstructing the benchmark testing framework, uniformly manages various test resources such as test cases, test machines, test plans, and test results. After pre-establishing a resource pool containing multiple test machines and a case set containing multiple test cases, after creating a test plan for benchmark testing of the basic environment, one or more test machines are selected from the resource pool to execute the test plan. Then, according to the test plan, test cases to be executed on each test machine are selected from the case set. Then, each test machine is controlled to execute the corresponding test cases, and the test results of each test machine are obtained. Finally, a test analysis report is generated based on the test results of each test machine.

[0113] The benchmark testing device for the basic environment of software and hardware products provided by the embodiments of the present invention not only enables the sharing of testing resources among different testing projects and testers, but also greatly improves the testing efficiency of the basic environment of software and hardware products.

[0114] Based on the same inventive concept, this invention also provides an embodiment of an electronic device for implementing all or part of the benchmark testing method for the basic environment of the aforementioned software and hardware products. This electronic device specifically includes the following:

[0115] The device comprises a processor, memory, a communications interface, and a bus; wherein the processor, memory, and communications interface communicate with each other via the bus; the communications interface is used to realize information transmission between related devices; the electronic device can be a desktop computer, tablet computer, or mobile terminal, etc., and this embodiment is not limited to these. In this embodiment, the electronic device can be implemented with reference to the embodiments of the benchmark testing method and the benchmark testing device for implementing the basic environment of the above-mentioned software and hardware products, the contents of which are incorporated herein by reference, and repeated details will not be described again.

[0116] Figure 13 This is a schematic diagram of the system composition structure of an electronic device provided in an embodiment of the present invention. Figure 13 As shown, the electronic device 130 may include a processor 1301 and a memory 1302; the memory 1302 is coupled to the processor 1301. It is worth noting that... Figure 13 This is an example; other types of structures can also be used to supplement or replace this structure to achieve telecommunications functions or other functions.

[0117] In one embodiment, the benchmark testing method for the basic environment of the software and hardware product can be integrated into the processor 1301. The processor 1301 can be configured to perform the following controls: create a test plan for benchmarking the basic environment; select one or more test machines from a resource pool to execute the test plan; select test cases to be executed on each test machine from a test case set according to the test plan; control each test machine to execute the corresponding test cases and obtain the test results of each test machine; and generate a test analysis report based on the test results of each test machine.

[0118] As can be seen from the above, the electronic device provided in the embodiments of the present invention, by reconstructing the benchmark testing framework, uniformly manages various test resources such as test cases, test machines, test plans, and test results. After pre-establishing a resource pool containing multiple test machines and a case set containing multiple test cases, after creating a test plan for benchmark testing of the basic environment, it selects one or more test machines from the resource pool to execute the test plan, and then selects test cases to be executed on each test machine from the case set according to the test plan. Then, it controls each test machine to execute the corresponding test cases, obtains the test results of each test machine, and finally generates a test analysis report based on the test results of each test machine.

[0119] The electronic equipment provided by the embodiments of the present invention not only enables the sharing of test resources among different test projects and testers, but also greatly improves the testing efficiency of the basic environment of software and hardware products.

[0120] In another embodiment, the benchmark testing device for the basic environment of the software and hardware product can be configured separately from the processor 1301. For example, the benchmark testing device for the basic environment of the software and hardware product can be configured as a chip connected to the processor 1301, and the function of the benchmark testing method for the basic environment of the software and hardware product can be realized through the control of the processor.

[0121] like Figure 13 As shown, the electronic device 130 may further include: a communication module 1303, an input unit 1304, an audio processing unit 1305, a display 1306, and a power supply 1307. It is worth noting that the electronic device 130 does not necessarily need to include these components. Figure 13 All components shown; in addition, the electronic device 130 may also include Figure 13 For components not shown, please refer to existing technologies.

[0122] like Figure 13 As shown, processor 1301, sometimes also referred to as controller or operation control, may include a microprocessor or other processor device and / or logic device, which receives input and controls the operation of various components of electronic device 130.

[0123] The memory 1302 may be, for example, one or more of a cache, flash memory, hard drive, removable media, volatile memory, non-volatile memory, or other suitable devices. It may store the aforementioned failure-related information, and also store a program for executing that information. The processor 1301 may execute the program stored in the memory 1302 to perform information storage or processing, etc.

[0124] Input unit 1304 provides input to processor 1301. This input unit 1304 may be, for example, a keypad or touch input device. Power supply 1307 provides power to electronic device 130. Display 1306 displays images and text, etc. This display may be, for example, an LCD display, but is not limited thereto.

[0125] The memory 1302 can be a solid-state memory, such as a read-only memory (ROM), random access memory (RAM), SIM card, etc. It can also be a memory that retains information even when power is off, can be selectively erased, and contains more data; examples of this type of memory are sometimes referred to as EPROMs. The memory 1302 can also be some other type of device. The memory 1302 includes a buffer memory 13021 (sometimes referred to as a buffer). The memory 1302 may include an application / function storage unit 13022 for storing application programs and function programs or processes for executing the operation of the electronic device 130 via the processor 1301.

[0126] The memory 1302 may also include a data storage unit 13023 for storing data, such as contacts, digital data, pictures, sounds, and / or any other data used by the electronic device. The driver storage unit 13024 of the memory 1302 may include various drivers for the electronic device for communication functions and / or for performing other functions of the electronic device (such as messaging applications, address book applications, etc.).

[0127] The communication module 1303 is a transmitter / receiver that transmits and receives signals via the antenna 1308. The communication module (transmitter / receiver) 1303 is coupled to the processor 1301 to provide input signals and receive output signals, which can be the same as in a conventional mobile communication terminal.

[0128] Based on different communication technologies, multiple communication modules 1303 can be configured in the same electronic device, such as cellular network modules, Bluetooth modules, and / or wireless LAN modules. The communication module (transmitter / receiver) 1303 is also coupled to a speaker 1309 and a microphone 1310 via an audio processing unit 1305 to provide audio output via the speaker 1309 and receive audio input from the microphone 1310, thereby realizing typical telecommunications functions. The audio processing unit 1305 may include any suitable buffer, decoder, amplifier, etc. Additionally, the audio processing unit 1305 is also coupled to a processor 1301, enabling on-device recording via the microphone 1310 and on-device playback of stored audio via the speaker 1309.

[0129] Based on the same inventive concept, embodiments of the present invention also provide a computer-readable storage medium for implementing all steps of the benchmark testing method for the basic environment of the software and hardware product in the above embodiments. The computer-readable storage medium stores a computer program, which, when executed by a processor, implements all steps of the benchmark testing method for the basic environment of the software and hardware product in the above embodiments. For example, when the processor executes the computer program, it implements the following steps: creating a test plan for benchmark testing of the basic environment; selecting one or more test machines from a resource pool to execute the test plan; selecting test cases to be executed on each test machine from a set of test cases according to the test plan; controlling each test machine to execute the corresponding test cases and obtaining the test results of each test machine; and generating a test analysis report based on the test results of each test machine.

[0130] As can be seen from the above, the computer-readable storage medium provided in the embodiments of the present invention, by reconstructing the benchmark testing framework, uniformly manages various test resources such as test cases, test machines, test plans, and test results. After pre-establishing a resource pool containing multiple test machines and a case set containing multiple test cases, after creating a test plan for benchmark testing of the basic environment, one or more test machines are selected from the resource pool to execute the test plan. Then, according to the test plan, test cases to be executed on each test machine are selected from the case set. Then, each test machine is controlled to execute the corresponding test cases, and the test results of each test machine are obtained. Finally, a test analysis report is generated based on the test results of each test machine.

[0131] The computer-readable storage medium provided by the embodiments of the present invention not only enables test resources to be shared among different test projects and testers, but also greatly improves the testing efficiency of the basic environment of software and hardware products.

[0132] While this invention provides the method operation steps as described in the embodiments or flowcharts, more or fewer operation steps may be included based on conventional or non-inventive labor. The order of steps listed in the embodiments is merely one possible execution order among many and does not represent the only possible execution order. In actual device or client product execution, the methods shown in the embodiments or drawings can be executed sequentially or in parallel (e.g., in a parallel processor or multi-threaded processing environment).

[0133] Those skilled in the art will understand that the embodiments of this specification can be provided as methods, apparatus (systems), or computer program products. Therefore, the embodiments of this specification can take the form of entirely hardware embodiments, entirely software embodiments, or embodiments combining software and hardware aspects. Furthermore, the invention can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0134] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0135] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0136] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0137] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on its differences from other embodiments. In particular, the system embodiments are relatively simple in description because they are fundamentally similar to the method embodiments; relevant parts can be referred to the descriptions in the method embodiments. In this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. The terms "upper," "lower," etc., indicating orientation or positional relationships based on the orientation or positional relationships shown in the accompanying drawings, are only for the convenience of describing the invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be construed as limiting the invention. Unless otherwise expressly specified and limited, the terms "installation," "connection," and "linking" should be interpreted broadly. For example, they can refer to fixed connections, detachable connections, or integral connections; mechanical connections or electrical connections; direct connections or indirect connections through an intermediate medium; and internal connections between two components. Those skilled in the art can understand the specific meaning of these terms in this invention according to the specific circumstances. It should be noted that, without conflict, the embodiments and features in the embodiments of this invention can be combined with each other. This invention is not limited to any single aspect, nor to any single embodiment, nor to any combination and / or substitution of these aspects and / or embodiments. Each aspect and / or embodiment of this invention can be used alone, or in combination with one or more other aspects and / or other embodiments.

[0138] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention, and they should all be covered within the scope of the claims and specification of the present invention.

Claims

1. A method of benchmarking a software and hardware product base environment, characterized by, include: A resource pool containing multiple test machines and a case set containing multiple test cases are pre-established; Create a test plan to benchmark the base environment; Select one or more test machines from the resource pool to execute the test plan; According to the test plan, select test cases from the test case set to be executed on each test machine; Control each test machine to execute the corresponding test cases and obtain the test results of each test machine; Based on the test results of each test machine, generate test analysis reports; The method further includes: defining a data model for test cases, wherein the data model adopts a three-level tree structure; the data model includes three dimensions: basic information, test definition, and test settings; The method further includes: defining test metrics used by test cases, wherein the test metrics include: framework-defined metrics and custom metrics; framework-defined metrics are divided into three categories: system resource metrics, including CPU load, memory utilization, IO read / write frequency, and network utilization; process metrics, including execution time and execution result; and log / alarm metrics, referring to the number of system alarms and system errors that occur during test execution; custom metrics refer to any metric type defined by the test cases in the metric definition template. After controlling each test machine to execute the corresponding test cases and obtaining the test results of each test machine, the method further includes: The test results are stored in XML format, resulting in an XML format test result file. After storing the test results in an XML format file and obtaining the test result file in XML format, the method further includes: The test results file in XML format is stored in a network-attached storage (NAS) device, which can be accessed by both the service backend and the client. The basic information of the test result file and the NAS storage path information are stored in a relational database, wherein the NAS storage path information is the file storage path information of the test result file in the NAS device; Based on the test results from each test machine, a test analysis report is generated, including: Read the test result file obtained from each benchmark test; Verify the test result files obtained from each benchmark test; Extract data from the test result files that have passed verification; Based on the data extraction results, the maximum, minimum, and average values ​​of each test indicator are calculated. Based on the maximum, minimum, and average values ​​of each test metric, a test analysis report is generated for each benchmark test; one-click test report generation is supported. Among them, the C / S testing architecture is used to enable benchmark tests on multiple test machines to run simultaneously through the server, and the results are aggregated, compared horizontally, and displayed.

2. The method of claim 1, wherein, Select one or more test machines from the resource pool to execute the test plan, including: Check if a test machine exists in the resource pool to execute the test plan; If there is no test machine in the resource pool to execute the test plan, add a test machine to the resource pool to execute the test plan.

3. The method of claim 1, wherein, According to the test plan, select test cases from the test case set to be executed on each test machine, including: Check if test cases for the test plan exist in the case set; If test cases for the test plan exist in the test case set, add test cases for the test plan to the test case set.

4. The method of claim 1, wherein, After generating a test analysis report based on the test results of each test machine, the method further includes: The generated test analysis reports are visualized in the form of charts.

5. The method according to any one of claims 1 to 4, characterized in that, Control each test machine to execute the corresponding test cases and obtain the test results of each test machine, including: Send test synchronization instructions to each test machine; According to the test synchronization command, control each test machine to synchronously execute the corresponding test cases; Receive test results returned by each test machine.

6. A benchmarking apparatus for a software and hardware product infrastructure environment, characterized by include: The test plan creation module is used to create test plans for benchmarking the base environment; The test machine determination module is used to select one or more test machines from the resource pool to execute the test plan; The test case determination module is used to select test cases to be executed on each test machine from the test case set according to the test plan; The testing module is used to control each test machine to execute the corresponding test cases and obtain the test results of each test machine; The test result analysis module is used to generate test analysis reports based on the test results of each test machine. The device further includes: a pre-established resource pool containing multiple test machines and a case set containing multiple test cases; a test case data model definition module, used to define the data model used for the test cases, wherein the data model adopts a three-level tree structure; the data model includes three dimensions: basic information, test definition, and test settings; The device further includes: a test metric definition module, used to define test metrics used by test cases, wherein the test metrics include: framework-defined metrics and custom metrics; framework-defined metrics are divided into three categories: system resource metrics, including CPU load, memory usage, IO read / write frequency, and network usage; process metrics, including execution time and execution result; and log / alarm metrics, referring to the number of system alarms and system errors that occur during test execution; custom metrics refer to any metric type defined by test cases in the metric definition template. The device further includes: The test result storage module is used to store test results in XML format, resulting in an XML format test result file. The test result storage module is also used for: The test results file in XML format is stored in a network-attached storage (NAS) device, which can be accessed by both the service backend and the client. The basic information of the test result file and the NAS storage path information are stored in a relational database, wherein the NAS storage path information is the file storage path information of the test result file in the NAS device; The test result analysis module specifically includes: The data reading unit is used to read the test result file obtained from each benchmark test. The verification unit is used to verify the test result file obtained from each benchmark test; The data extraction unit is used to extract data from the test result files that have passed the verification. The data statistics unit is used to calculate the maximum, minimum, and average values ​​of each test indicator based on the data extraction results. The data output unit is used to generate test analysis reports for each benchmark test based on the maximum, minimum, and average values ​​of each test indicator; it supports one-click test report generation. Among them, the C / S testing architecture is used to enable benchmark tests on multiple test machines to run simultaneously through the server, and the results are aggregated, compared horizontally, and displayed.

7. The apparatus of claim 6, wherein, The testing machine determination module is also used for: Check if a test machine exists in the resource pool to execute the test plan; If there is no test machine in the resource pool to execute the test plan, add a test machine to the resource pool to execute the test plan.

8. The apparatus as claimed in claim 6, characterized in that, The test case determination module is also used for: Check if test cases for the test plan exist in the case set; If test cases for the test plan exist in the test case set, add test cases for the test plan to the test case set.

9. The apparatus as claimed in claim 6, characterized in that, The device further includes: The visualization module is used to visually display the generated test analysis reports in the form of charts.

10. The apparatus according to any one of claims 6 to 9, characterized in that, The testing module is also used for: Send test synchronization instructions to each test machine; According to the test synchronization command, control each test machine to synchronously execute the corresponding test cases; Receive test results returned by each test machine.

11. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the benchmark testing method for the basic environment of the software and hardware product as described in any one of claims 1 to 5.

12. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that performs a benchmark testing method for the underlying environment of the software and hardware product according to any one of claims 1 to 5.

Images