Automatic testing method and system

Abstract

The invention relates to the technical field of testing, and provides an automatic testing method and system. The automatic control server determines a corresponding target test bed according to the to-be-tested task, and divides test cases with the same resource requirements in the to-be-tested task into one group to obtain a plurality of test case groups; the automatic control server generates first test information and sends the first test information to the automatic execution server when the test resources of the target test bed meet the resource requirements of the test case group, and the automatic execution server performs testing according to the first test information; when the test resources of the target test bed do not meet the resource requirements of the test case group, the automatic control server obtains a target resource bed and each target auxiliary test resource, establishes an automatic communication tunnel between the target test bed and the target resource bed, generates second test information and sends the second test information to an automatic execution server; and the automatic execution server performs testing according to the second test information. And the resource utilization rate and the test efficiency are improved.

Description

Technical Field

[0001] The present invention relates to the field of testing technology, and in particular to an automated testing method and system. Background Art

[0002] In the automated test system of communication equipment, each automated test environment is independent. Such a test environment usually consists of a topology switch and several resources, which is called a test bed. The test bed usually contains two types of resources. One is the test resource, that is, the "test equipment" also known as the "tested equipment", which is a product of project research and development and is mobile, that is, it is applied for in the early stage of the project and released after the project is completed; the other is auxiliary test resources, such as "auxiliary test equipment", "automated test server", "automated test instrument" and "automated test terminal".

[0003] "Auxiliary test equipment" is a published standard product of the same level as the project R&D product, which is used for docking testing with the "equipment under test". After the project is completed, if it meets the test requirements of subsequent projects, it can continue to be used in the next project. "Automated test server", "automated test instrument" and "automated test terminal" are universal resources and can be used in various projects. When there are multiple projects at the same time, each project has different test requirements. It is necessary to build a test bed for each project to achieve physical isolation and avoid mutual influence between multiple test beds. However, since auxiliary test resources need to be configured in each test bed, there is a shortage of auxiliary test resources, which reduces resource utilization and the efficiency of automated testing. Summary of the invention

[0004] In view of this, an object of the present invention is to provide an automated testing method and system.

[0005] In order to achieve the above object, the technical solution adopted by the present invention is as follows:

[0006] In a first aspect, the present invention provides an automated testing method, which is applied to an automated testing system, wherein the automated testing system includes an automated control server, an automated execution server, multiple test beds, and multiple resource beds, wherein the resource bed includes multiple auxiliary test resources, and the method includes:

[0007] The automation control server obtains multiple test cases in the task to be tested and a target test bed corresponding to the task to be tested, and groups the test cases according to the resource requirements of each test case to obtain multiple test case groups, wherein all test cases in the test case groups have the same resource requirements;

[0008] When the test resources of the target test bed meet the resource requirements of the test case group, the automation control server generates first test information based on the test case group and the target test bed and sends the first test information to the automation execution server;

[0009] The automation execution server performs testing according to the first test information;

[0010] When the test resources of the target test bed do not meet the resource requirements of the test case group, the automation control server determines the resources to be supplemented of the test case group, determines the target resource bed based on the resources to be supplemented, obtains each target auxiliary test resource in the target resource bed, and establishes an automated communication tunnel between the target test bed and the target resource bed, then generates second test information based on the test case group, the target test bed and each of the target auxiliary test resources and sends it to the automation execution server;

[0011] The automation execution server performs testing according to the second test information.

[0012] In an optional implementation manner, the automation control server stores the location of each of the test beds and the location and bandwidth of each of the resource beds;

[0013] The automation control server determines a target resource bed based on the resources to be supplemented, including:

[0014] The automation control server obtains each candidate resource bed from all resource beds, and all idle auxiliary test resources of the candidate resource bed include the resource to be supplemented;

[0015] The automation control server calculates the distance between each candidate resource bed and the target test bed based on the location of each candidate resource bed and the location of the target test bed, and uses the candidate resource bed with the shortest distance and the largest bandwidth as the target resource bed.

[0016] In an optional implementation, the resources to be supplemented include multiple types of resources to be allocated, and the automation control server stores the version, capability set and performance parameters of each auxiliary test resource in the resource bed;

[0017] The automation control server obtains each target auxiliary test resource in the target resource bed, including:

[0018] The automation control server obtains all candidate auxiliary test resources corresponding to each resource to be allocated in the target resource bed, and obtains an auxiliary test resource list, wherein the versions and capability sets of the candidate auxiliary test resources match the corresponding resources to be allocated;

[0019] The automation control server calculates a performance evaluation value of each of the candidate auxiliary test resources based on the performance parameters of each of the candidate auxiliary test resources;

[0020] The automation control server combines resources based on the auxiliary test resource list to obtain multiple auxiliary test resource groups, and calculates a total performance evaluation value of each auxiliary test resource group, wherein the auxiliary test resource group includes a candidate auxiliary test resource corresponding to each resource to be allocated;

[0021] The automation control server acquires the auxiliary test resource group with the largest total performance evaluation value, obtains a target auxiliary test resource group, and uses each candidate auxiliary test resource in the target auxiliary test resource group as each of the target auxiliary test resources.

[0022] In an optional implementation, the performance parameters include a test success rate, a test health, a test success list, a distance parameter, a load parameter, and a bandwidth parameter, wherein the test success rate represents the success rate of all test cases executed by the auxiliary test resource, the test health represents the success rate of all test cases executed by the auxiliary test resource within a preset historical period, the test success list includes all test cases successfully executed by the auxiliary test resource within the preset historical period, the distance parameter represents the relative distance between the auxiliary test resource and the resource bed to which it belongs, and the load parameter represents the total number of test cases executed by the auxiliary test resource within the preset historical period;

[0023] The automatic control server calculates a performance evaluation value of each of the candidate auxiliary test resources based on the performance parameter of each of the candidate auxiliary test resources, including:

[0024] The automation control server performs weighted summation for each candidate auxiliary test resource based on the test success rate, test health, test success list, distance parameter, load parameter and bandwidth parameter of the candidate auxiliary test resource to obtain the performance evaluation value of each candidate auxiliary test resource. In an optional embodiment, each of the test beds and each of the resource beds includes a topology switch;

[0025] The automation control server establishes an automation communication tunnel between the target test bed and the target resource bed, including:

[0026] The automation control server establishes an automation communication tunnel between the target test bed and the target resource bed by connecting the topology switch of the target test bed and the topology switch of the target resource bed.

[0027] In an optional implementation manner, the automation control server stores performance parameters of each auxiliary test resource in the resource bed, and the method further includes:

[0028] After the automation execution server finishes the test according to the second test information, reporting the test log to the automation control server;

[0029] The automation control server updates the performance parameters of each of the target auxiliary test resources according to the test log, releases each of the target auxiliary test resources, and deletes the automation communication tunnel between the target test bed and the target resource bed.

[0030] In a second aspect, the present invention provides an automated testing system, the automated testing system comprising an automated control server, an automated execution server, a plurality of test beds and a plurality of resource beds, the resource bed comprising a plurality of auxiliary testing resources;

[0031] The automation control server is used to obtain multiple test cases in the task to be tested and the target test bed corresponding to the task to be tested, and group the test cases according to the resource requirements of each test case to obtain multiple test case groups, and the resource requirements of all test cases in the test case group are the same;

[0032] The automation control server is used for generating first test information based on the test case group and the target test bed and sending the first test information to the automation execution server when the test resources of the target test bed meet the resource requirements of the test case group;

[0033] The automation execution server is used to perform testing according to the first test information;

[0034] The automation control server is used to determine the resources to be supplemented of the test case group when the test resources of the target test bed do not meet the resource requirements of the test case group, and after determining the target resource bed based on the resources to be supplemented and obtaining each target auxiliary test resource in the target resource bed and establishing an automated communication tunnel between the target test bed and the target resource bed, generate second test information based on the test case group, the target test bed and each of the target auxiliary test resources and send it to the automation execution server;

[0035] The automation execution server is used to perform testing according to the second test information.

[0036] In an optional implementation manner, the automation control server stores the location of each of the test beds and the location and bandwidth of each of the resource beds, and the automation control server is further used to:

[0037] Acquire each candidate resource bed from all resource beds, wherein all idle auxiliary test resources of the candidate resource bed include the resource to be supplemented;

[0038] Based on the position of each candidate resource bed and the position of the target test bed, the distance between each candidate resource bed and the target test bed is calculated, and the candidate resource bed with the shortest distance and the largest bandwidth is used as the target resource bed.

[0039] In an optional implementation, the resources to be supplemented include multiple types of resources to be allocated, the automation control server stores the version, capability set and performance parameters of each auxiliary test resource in the resource bed, and the automation control server is further used to:

[0040] In the target resource bed, all candidate auxiliary test resources corresponding to each resource to be allocated are obtained to obtain an auxiliary test resource list, wherein the versions and capability sets of the candidate auxiliary test resources match the corresponding resources to be allocated;

[0041] Calculating a performance evaluation value of each of the candidate auxiliary test resources based on the performance parameters of each of the candidate auxiliary test resources;

[0042] Combining resources based on the auxiliary test resource list to obtain multiple auxiliary test resource groups, and calculating a total performance evaluation value of each auxiliary test resource group, wherein the auxiliary test resource group includes a candidate auxiliary test resource corresponding to each resource to be allocated;

[0043] An auxiliary test resource group with a maximum total performance evaluation value is obtained to obtain a target auxiliary test resource group, and each candidate auxiliary test resource in the target auxiliary test resource group is used as each of the target auxiliary test resources.

[0044] In an optional implementation, the performance parameters include a test success rate, a test health, a test success list, a distance parameter, a load parameter, and a bandwidth parameter. The test success rate represents the success rate of all test cases executed by the auxiliary test resource. The test health represents the success rate of all test cases executed by the auxiliary test resource within a preset historical period. The test success list includes all test cases successfully executed by the auxiliary test resource within a preset historical period. The distance parameter represents the relative distance between the auxiliary test resource and the resource bed to which it belongs. The load parameter represents the total number of test cases executed by the auxiliary test resource within the preset historical period. The automation control server is also used for:

[0045] For each of the candidate auxiliary test resources, a weighted sum is performed based on the test success rate, test health, test success list, distance parameter, load parameter and bandwidth parameter of the candidate auxiliary test resource to obtain the performance evaluation value of the candidate auxiliary test resource.

[0046] The present invention provides an automated testing method and system, wherein an automated control server determines a corresponding target test bed according to a task to be tested, and groups test cases with the same resource requirements in the task to be tested into a group to obtain multiple test case groups; when the test resources of the target test bed meet the resource requirements of the test case group, the automated control server generates first test information and sends it to an automated execution server, and the automated execution server performs a test according to the first test information; when the test resources of the target test bed do not meet the resource requirements of the test case group, the automated control server obtains the target resource bed and each target auxiliary test resource and establishes an automated communication tunnel between the target test bed and the target resource bed, then generates second test information and sends it to the automated execution server, and the automated execution server performs a test according to the second test information. By dividing the common auxiliary test resources into multiple resource beds, and dynamically allocating auxiliary test resources from the resource bed to assist in the test when the test resources of the test bed cannot meet the resource requirements of the test case, the utilization rate of resources is improved, and by establishing an automated communication tunnel, there is no need to repeatedly build a test environment, thereby improving the efficiency of automated testing.

[0047] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, preferred embodiments are given below and described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments are briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without creative work.

[0049] Figure 1 A schematic diagram of an existing automated testing system is shown;

[0050] Figure 2 A schematic diagram of an automated testing system provided by an embodiment of the present invention is shown;

[0051] Figure 3 A schematic diagram of a process flow of an automated testing method provided by an embodiment of the present invention is shown;

[0052] Figure 4Another schematic flow chart of the automated testing method provided by an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0053] The following will be combined with the accompanying drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings here can be arranged and designed in various different configurations.

[0054] Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the invention claimed for protection, but merely represents selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without making creative work are within the scope of protection of the present invention.

[0055] It should be noted that relational terms such as "first" and "second" are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0056] In the automated test system of communication equipment, each automated test environment is independent. Such a test environment usually consists of a topology switch and several resources, which is called a test bed. The test bed usually contains two types of resources. One is the test resource, that is, the "test equipment" also known as the "tested equipment", which is a product of project research and development and is mobile, that is, it is applied for in the early stage of the project and released after the project is completed; the other is auxiliary test resources, such as "auxiliary test equipment", "automated test server", "automated test instrument" and "automated test terminal".

[0057] "Auxiliary test equipment" is a published standard product of the same level as the project R&D product, which is used for docking test with the "device under test". After the project is completed, if it meets the test requirements of subsequent projects, it can continue to be used in the next project. "Automated test server", "automated test instrument" and "automated test terminal" are universal resources and can be used in various projects. When there are multiple projects at the same time, each project has different test requirements. It is necessary to build a test bed for each project to achieve physical isolation and avoid mutual influence between multiple test beds.

[0058] See also Figure 1 , is a schematic diagram of an existing automated test system, which includes an automated server and multiple independent test beds, each of which includes a topology switch and several test devices and auxiliary test resources. It can be seen that the auxiliary test resources need to be configured in each test bed, resulting in a shortage of auxiliary test resources, which reduces resource utilization and the efficiency of automated testing. Although there are ways to improve resource utilization by improving a single test bed, the existing automated test system still has the following problems:

[0059] (1) Repeated construction and repeated execution: Each project requires the construction of a test bed, which requires the configuration of test equipment, auxiliary test resources, and topology switches, that is, the completion of a complete test bed network construction. Each time the auxiliary test resources are built and their maintenance work needs to be repeated.

[0060] (2) Auxiliary test resources are scattered, cannot be reused, and have low utilization: Because a test bed needs to be built for each project, and each test bed requires a certain number of auxiliary test resources, under the premise that the total number of auxiliary test resources is limited and in short supply, the auxiliary test resources allocated to each test bed are uneven. Some resources may not be available, and some resources may be in short supply, resulting in incomplete coverage of the project's automation execution strategy, requiring additional manpower for manual testing and verification, and lengthening the automation execution cycle, affecting the automation test rhythm and test plan, and thus affecting the project progress. In addition, because the execution strategies in the early, mid-term, and late stages of the project are different, and the demand for resources in each automation execution cycle is also different, it is impossible for the resources planned in the early stage to be continuously used all the time, and they are idle most of the time. This results in the consumption of human resources if resources are not invested, and the investment of resources may result in the inability to fully utilize resources, resulting in a low input-output ratio.

[0061] (3) Resource configuration is relatively fixed, with poor flexibility and difficulty in migration: If resources are not used for a long time, they need to be manually physically migrated to other environments. Or in a large Layer 2 network, resources can be temporarily migrated and used in the same Layer 2 network by manually or automatically configuring VLAN (Virtual Local Area Network) and Trunk (port aggregation) on the interconnected large Layer 2 switches. However, this method can only be used in a Layer 2 environment and cannot cross a Layer 3 network environment. In addition, the maintenance cost is very high when the network environment is complex, and network storms and other problems often occur.

[0062] (4) Poor maintainability and high maintenance cost: With the continuous iteration and update of products and services, resources such as automated test servers, automated test instruments, and automated test terminals will also be iteratively updated or added. However, because each auxiliary test resource is distributed in each test environment and is the responsibility of the environment maintainer of each project, it is difficult to achieve a unified upgrade or addition, and it will consume a lot of human resources. This will lead to the failure or inability to execute subsequent new automated test cases, and human analysis or manual testing will be required, which consumes human resources.

[0063] Therefore, the embodiment of the present invention provides an automated testing system to solve the above problems. Figure 2 , is a schematic diagram of an automated test system provided by an embodiment of the present invention, which includes an automated control server, an automated execution server, multiple test beds, and multiple resource beds. The test bed includes a topology switch and multiple test devices; the resource bed includes a topology switch and multiple auxiliary test resources. The number of automated execution servers can be the same as the number of test beds, or can be set according to actual conditions, which is not limited in the embodiment of the present invention.

[0064] The automation control server, or CS server, is used to manage and maintain all test beds and all resource beds, and to allocate resources. The automation execution server, or ES server, is used to register with the CS server, and after registration, perform tests according to the test information sent by the CS server and report test logs.

[0065] Each test bed corresponds to a project, which includes the test equipment required for the corresponding project and may also include a small amount of auxiliary test equipment.

[0066] The auxiliary test resources in the resource bed can be auxiliary test equipment, automated test servers, automated test instruments and automated test terminals. Among them, the automated test server can be understood as a general term for servers such as AAA (Authentication, Authorization, Accounting) authentication servers, certificate servers, IPv6 (Internet Protocol Version 6, Internet Communication Network Protocol Version 6) Ready servers, NETCONF (Network Configuration Protocol, Network Configuration Protocol) servers, and Telemetry (remote data acquisition) servers. Automated test instruments can be understood as a general term for test instruments such as TestCenter testers, IXIA testers, and Xinertai testers. Automated test terminals can be understood as a general term for test terminals of various versions such as Linux systems and Windows systems.

[0067] The topology switch can be regarded as a tunnel node, which supports the establishment of Layer 2 and Layer 3 tunnels such as VXLAN (Virtual Extensible Local Area Network) tunnels and L3VPN (Layer 3 Virtual Private Network) tunnels. It can be understood that the automated test system establishes tunnels through the topology switches in each test bed and each resource bed, and all tunnels are logically isolated and will not affect each other, which can ensure the stability of the automated test environment.

[0068] It can be seen that compared with the existing automated testing system, the automated testing system provided by the embodiment of the present invention divides public auxiliary testing resources into multiple resource beds, and uses an automated control server to uniformly manage and maintain all resource beds, saving labor costs while also providing environmental support for subsequent dynamic resource allocation.

[0069] The following will be Figure 2 The automated testing system shown serves as an execution entity, executes each step in each method provided in the embodiments of the present invention, and achieves corresponding technical effects.

[0070] See also Figure 3 , Figure 3 It is a flowchart of an automated testing method provided by an embodiment of the present invention.

[0071] Step S202, the automation control server obtains multiple test cases in the task to be tested and the target test bed corresponding to the task to be tested, and groups the test cases according to the resource requirements of each test case to obtain multiple test case groups, and all the test cases in the test case group have the same resource requirements;

[0072] In this embodiment, each project has a corresponding test environment, namely a test bed. The tester can submit the test task of a project, such as Project A, to the automation control server. The automation control server receives the test task of Project A, namely, obtains the task to be tested, and then obtains the test bed corresponding to Project A from all test beds, namely, obtains the test bed corresponding to the task to be tested, and obtains the target test bed.

[0073] In addition, the task to be tested includes multiple test cases, each of which records resource requirements, which can be understood as the type and quantity of resources required to execute the test case. The automation control server groups the multiple test cases in the task to be tested according to the resource requirements, that is, groups the test cases with the same resource requirements into one group, and obtains multiple test case groups.

[0074] It can be understood that the automation control server groups test cases by resource requirements to facilitate subsequent resource allocation by group, so that test cases with the same resource requirements can be executed in the same test environment, thereby improving resource utilization and the efficiency of automated testing.

[0075] Step S204: When the test resources of the target test bed meet the resource requirements of the test case group, the automation control server generates first test information based on the test case group and the target test bed and sends the first test information to the automation execution server;

[0076] Step S206, the automation execution server performs a test according to the first test information;

[0077] Step S208, when the test resources of the target test bed do not meet the resource requirements of the test case group, the automation control server determines the resources to be supplemented of the test case group, determines the target resource bed based on the resources to be supplemented, obtains each target auxiliary test resource in the target resource bed, and establishes an automated communication tunnel between the target test bed and the target resource bed, then generates second test information based on the test case group, the target test bed and each target auxiliary test resource and sends it to the automation execution server;

[0078] Step S210, the automation execution server performs a test according to the second test information;

[0079] It can be understood that the manner of performing the test based on each test case group is similar. For the sake of brevity, one test case group is used as an example for explanation below.

[0080] The automated execution server determines whether the test resources of the target test bed meet the resource requirements of the test case group based on the test resources of the target test bed and the resource requirements of the test case group.

[0081] If satisfied, it means that the type and quantity of the test resources in the target test bed can support the execution of the test case group, and the test resources of the target test bed are used to execute the test case group, that is, first test information is generated based on the test case group and the target test bed and sent to the automation execution server. The automation execution server receives the first test information and performs a test according to the first test information, that is, performs a test operation on the test case group using the test resources of the target test bed.

[0082] If it is not satisfied, it means that the type and quantity of test resources in the target test bed are insufficient to support the execution of the test case group. Then the resources that need to be supplemented are calculated, that is, the resources to be supplemented are determined, and then the resource bed with the resources to be supplemented is selected from all resource beds, that is, the target resource bed is obtained, and the auxiliary test resources matching the resources to be supplemented are selected from the target resource bed, that is, each target auxiliary test resource is obtained and allocated, so as to utilize these target auxiliary test resources to assist the target test bed in testing.

[0083] It is understandable that in order to facilitate these target auxiliary test resources to assist the target test bed in testing, it is also necessary to establish an automated communication tunnel between the target test bed and the target resource bed to complete the topological network mapping. Then, based on the test case group, the target test bed and each target auxiliary test resource, the second test information is generated and sent to the automated execution server. The automated execution server receives the second test information and performs testing according to the second test information, that is, through the automated communication tunnel, using the test resources of the target test bed and each target auxiliary test resource, the test operation is performed on the test case group.

[0084] It can be seen that based on the above steps, the automation control server determines the corresponding target test bed according to the task to be tested, and groups the test cases with the same resource requirements in the task to be tested into a group to obtain multiple test case groups; when the test resources of the target test bed meet the resource requirements of the test case group, the automation control server generates a first test information and sends it to the automation execution server, and the automation execution server performs the test according to the first test information; when the test resources of the target test bed do not meet the resource requirements of the test case group, the automation control server obtains the target resource bed and each target auxiliary test resource and establishes an automated communication tunnel between the target test bed and the target resource bed, and then generates a second test information and sends it to the automation execution server, and the automation execution server performs the test according to the second test information. By dividing the public auxiliary test resources into multiple resource beds, and dynamically allocating auxiliary test resources from the resource bed to assist in the test when the test resources of the test bed cannot meet the resource requirements of the test case, the utilization rate of resources is improved, and by establishing an automated communication tunnel, there is no need to repeatedly build a test environment, which improves the efficiency of automated testing.

[0085] Optionally, for the process in the above step S208 in which the automation control server determines the target resource bed based on the resources to be supplemented, an embodiment of the present invention provides a possible implementation method.

[0086] Step S208-1, the automation control server obtains each candidate resource bed from all resource beds, and all idle auxiliary test resources of the candidate resource bed include resources to be supplemented;

[0087] Step S208 - 3 : The automation control server calculates the distance between each candidate resource bed and the target test bed based on the location of each candidate resource bed and the location of the target test bed, and uses the candidate resource bed with the shortest distance and the largest bandwidth as the target resource bed.

[0088] It can be understood that the automation control server stores the location of each test bed and the location and bandwidth of each resource bed.

[0089] For ease of understanding, an embodiment of the present invention provides an example. For example, if the test resource group is test resource group A, and if the test resources of the target test bed do not meet the resource requirements of test case group A, the automation control server will calculate the resources that need to be supplemented based on the resource requirements of test case group A and the test resources of the target test bed, that is, determine the resources to be supplemented; then obtain each resource bed containing the resources to be supplemented from all resource beds, that is, obtain each candidate resource bed.

[0090] It can be understood that in order to ensure that auxiliary test resources matching the resources to be supplemented can be selected, it is necessary to refer to the idle auxiliary test resources in the resource bed when selecting the resource bed so that all the idle auxiliary test resources of the selected resource bed include the resources to be supplemented.

[0091] If the number of candidate resource beds is one, the candidate resource bed is used as the target resource bed.

[0092] If there are multiple candidate resource beds, the automation control server will calculate the distance between each candidate resource bed and the target test bed based on the location of each candidate resource bed and the location of the target test bed, and select the candidate resource bed with the shortest distance, that is, obtain the pending resource bed.

[0093] If the number of pending resource beds is one, the pending resource bed is used as the target resource bed. If the number of pending resource beds is multiple, the bandwidths of the multiple pending resource beds are compared, and the pending resource bed with the largest bandwidth is used as the target resource bed.

[0094] After determining the target resource bed, the automation control server selects auxiliary test resources from the target resource bed to assist the target test bed in executing test case group A.

[0095] Optionally, for the process in the above step S208 in which the automation control server obtains each target auxiliary test resource in the target resource bed, an embodiment of the present invention provides a possible implementation method.

[0096] Step S208-5: the automation control server obtains all candidate auxiliary test resources corresponding to each resource to be allocated in the target resource bed, obtains an auxiliary test resource list, and the versions and capability sets of the candidate auxiliary test resources match the corresponding resources to be allocated;

[0097] Step S208-7, the automation control server calculates a performance evaluation value of each candidate auxiliary test resource based on the performance parameters of each candidate auxiliary test resource;

[0098] Step S208-9: the automation control server combines resources based on the auxiliary test resource list to obtain multiple auxiliary test resource groups, and calculates the total performance evaluation value of each auxiliary test resource group, where the auxiliary test resource group includes a candidate auxiliary test resource corresponding to each resource to be allocated;

[0099] Step S208-11: the automation control server obtains the auxiliary test resource group with the largest total performance evaluation value, obtains the target auxiliary test resource group, and uses each candidate auxiliary test resource in the target auxiliary test resource group as each target auxiliary test resource.

[0100] It is understandable that the resources to be supplemented include various types of resources to be allocated. The automation control server also stores the version, capability set and performance parameters of each auxiliary test resource in the resource bed.

[0101] The version can be understood as the version model of the auxiliary test resource. Different auxiliary test resources may have different version descriptions. When selecting auxiliary test resources, you need to consider whether their version meets the requirements to ensure that the test case can be executed normally.

[0102] A capability set can be understood as a collection of computing capabilities or functions that auxiliary test resources can provide. For example, if a TestCenter tester supports emulation of the BFD (Bidirectional Forwarding Detection) protocol, its capability set includes emulation of the BFD protocol; if a switch supports the L3VPN function, its capability set includes the L3VPN function. The capability set of auxiliary test resources can be configured according to actual needs.

[0103] The performance parameter may be understood as a parameter used to reflect the performance of the auxiliary test resource, which may be a dynamic parameter. For example, the performance parameter may be updated based on the execution of the test case by the auxiliary test resource.

[0104] For ease of understanding, the above example is used for further explanation. Assume that the automation control server determines that the resources to be supplemented are two types of resources to be allocated based on the resource requirements of test case group A and the test resources of the target test bed. The first type of resource to be allocated is an automation test terminal with a Windows system and 12GB of memory; the second type of resource to be allocated is an automation test server with a Windows system and integrated Telemetry function.

[0105] First, the automation control server obtains each auxiliary test resource whose version and capability set match the first resource to be allocated in the target resource bed, and obtains each first candidate auxiliary test resource, that is, the first candidate auxiliary test resource is an automation test terminal with a Windows system version and a memory of not less than 12GB in the capability set. In addition, obtain each auxiliary test resource whose version and capability set match the second resource to be allocated, and obtain each second candidate auxiliary test resource, that is, the second candidate auxiliary test resource is an automation test server with a Windows system version and a Telemetry function in the capability set. Obtain an auxiliary test resource list including each first candidate auxiliary test resource and each second candidate auxiliary test resource.

[0106] It should be understood that the first candidate auxiliary test resource and the second candidate auxiliary test resource are both candidate auxiliary test resources, and the descriptions of "first" and "second" are used to distinguish whether the candidate auxiliary test resource corresponds to the first type of resource to be allocated or the second type of resource to be allocated.

[0107] Then, the automation control server calculates the performance evaluation value of each candidate auxiliary test resource based on the performance parameter of each candidate auxiliary test resource. It can be understood that the performance evaluation value of the candidate auxiliary test resource is calculated based on the performance parameter of the candidate auxiliary test resource, so as to make a selection based on the performance evaluation value of the candidate auxiliary test resource in the future.

[0108] Next, the automation control server combines resources based on the auxiliary test resource list to obtain multiple auxiliary test resource groups, and calculates the sum of the performance evaluation values ​​of all candidate auxiliary test resources in the same auxiliary test resource group to obtain the total performance evaluation value of each auxiliary test resource group. The auxiliary test resource group includes a selected auxiliary test resource corresponding to the first type of resource to be allocated, that is, a first candidate auxiliary test resource, and a selected auxiliary test resource corresponding to the second type of resource to be allocated, that is, a second candidate auxiliary test resource.

[0109] It can be understood that multiple auxiliary test resource groups are obtained through resource combination and their total performance evaluation value is calculated, so that the optimal auxiliary test resource combination can be selected by referring to the total performance evaluation value to assist the target test bed in testing, thereby improving the success rate and efficiency of automated testing.

[0110] Finally, the automation control server obtains the auxiliary test resource group with the largest total performance evaluation value to obtain the target auxiliary test resource group. It can be understood that the target auxiliary test resource group is the optimal auxiliary test resource combination, and each candidate auxiliary test resource in the target auxiliary test resource group is each target auxiliary test resource.

[0111] It is understandable that the automation control server will mark each selected target auxiliary test resource to indicate that the target auxiliary test resource is locked, that is, before the test is completed, the target auxiliary test resource will always be occupied, and the target auxiliary test resource cannot be allocated. After the test is completed and the target auxiliary test resource is released, the target auxiliary test resource becomes an idle auxiliary test resource, and then it can be allocated.

[0112] Optionally, for the above-mentioned step S208-7, an embodiment of the present invention provides a possible implementation method, namely: the automated control server performs weighted summation on each candidate auxiliary test resource based on the test success rate, test health, test success list, distance parameter, load parameter and bandwidth parameter of the candidate auxiliary test resource to obtain the performance evaluation value of the candidate auxiliary test resource, thereby obtaining the performance evaluation value of each candidate auxiliary test resource.

[0113] It is understandable that the performance parameters may include test success rate, test health, test success list, distance parameter, load parameter and bandwidth parameter. It should be understood that the performance parameters may also be set according to actual conditions, and the embodiment of the present invention does not limit this.

[0114] Among them, the test success rate represents the success rate of all test cases executed by the auxiliary test resources; the test health represents the success rate of all test cases executed by the auxiliary test resources within the preset historical period; the test success list includes all test cases successfully executed by the auxiliary test resources within the preset historical period; the distance parameter represents the relative distance between the auxiliary test resource and the resource bed to which it belongs; the load parameter represents the total number of test cases executed by the auxiliary test resources within the preset historical period.

[0115] In this embodiment, corresponding weight values ​​can be set in advance for the test success rate, test health, test success list, distance parameter, load parameter and bandwidth parameter. It should be understood that the weight value corresponding to each performance parameter can be dynamically adjusted according to the specified rules, and the embodiment of the present invention does not limit it.

[0116] The automated control server may perform weighted summation based on the test success rate, test health, test success list, distance parameter, load parameter and bandwidth parameter of the candidate auxiliary test resources and their corresponding weight values, thereby obtaining a performance evaluation value of the candidate auxiliary test resources.

[0117] By calculating each candidate auxiliary test resource in a similar manner, the performance evaluation value of each candidate auxiliary test resource can be obtained.

[0118] Optionally, for the process in the above step S208 in which the automation control server establishes an automation communication tunnel between the target test bed and the target resource bed, an embodiment of the present invention provides a possible implementation method.

[0119] Step S208 - 13 : The automation control server establishes an automation communication tunnel between the target test bed and the target resource bed by connecting the topology switch of the target test bed and the topology switch of the target resource bed.

[0120] In this embodiment, each test bed and each resource bed includes a topology switch, which can be regarded as a tunnel node. After determining the target resource bed, the automation control server can establish a dedicated tunnel for testing between the target test bed and the target resource bed, that is, an automation communication tunnel, by connecting the topology switch of the target test bed with the topology switch of the target resource bed.

[0121] It can be understood that the automation control server connects the tunnel nodes by connecting the topology switch to complete the networking mapping from logical topology to physical topology. In addition, there is logical isolation between multiple tunnels in the automated test system, which will not affect each other, thus ensuring the stability of the automated test environment, providing a communication basis for dynamic resource allocation, and improving the efficiency of automated testing.

[0122] Optionally, after the above step S210, the embodiment of the present invention further provides a possible implementation method, see Figure 4 .

[0123] Step S212, after the automation execution server finishes the test according to the second test information, it reports the test log to the automation control server;

[0124] Step S214: the automation control server updates the performance parameters of each target auxiliary test resource according to the test log, releases each target auxiliary test resource, and deletes the automation communication tunnel between the target test bed and the target resource bed.

[0125] In this embodiment, after the automation execution server finishes the test according to the second test information, it will report a test log to the automation control server. The test log includes information such as test results and data.

[0126] The automated control server receives the test log and updates the performance parameters of each target auxiliary test resource, such as test success rate, test health, test success list, and load parameters, based on the test results, data, and other information in the test log. It then releases each target auxiliary test resource, that is, the target auxiliary test resource is restored to a public auxiliary test resource for reuse.

[0127] In addition, the automation control server will also delete the automation communication tunnel between the target test bed and the target resource bed to restore the default connection between the topology switch of the target test bed and the topology switch of the target resource bed.

[0128] The embodiment of the present invention also provides an automated testing system. It should be noted that the basic principle and technical effects of the automated testing system provided in this embodiment are the same as those of the above embodiments. For the sake of brief description, the parts not mentioned in this embodiment can refer to the corresponding contents in the above embodiments. The automated testing system provided in this embodiment includes an automated control server, an automated execution server, multiple test beds, and multiple resource beds, and the resource bed includes multiple auxiliary test resources.

[0129] The automation control server is used to obtain multiple test cases in the task to be tested and the target test bed corresponding to the task to be tested, and group them according to the resource requirements of each test case to obtain multiple test case groups, and all test cases in the test case group have the same resource requirements;

[0130] The automation control server is used to generate first test information based on the test case group and the target test bed and send it to the automation execution server when the test resources of the target test bed meet the resource requirements of the test case group;

[0131] The automated execution server is used to perform a test according to the first test information;

[0132] The automation control server is used to determine the resources to be supplemented of the test case group when the test resources of the target test bed do not meet the resource requirements of the test case group, and after determining the target resource bed based on the resources to be supplemented and obtaining each target auxiliary test resource in the target resource bed and establishing an automated communication tunnel between the target test bed and the target resource bed, generate second test information based on the test case group, the target test bed and each target auxiliary test resource and send it to the automation execution server;

[0133] The automation execution server is used for performing a test according to the second test information.

[0134] Optionally, the automation control server is also used to: obtain each candidate resource bed from all resource beds, where all idle auxiliary test resources of the candidate resource bed include resources to be supplemented; based on the location of each candidate resource bed and the location of the target test bed, calculate the distance between each candidate resource bed and the target test bed, and use the candidate resource bed with the shortest distance and the largest bandwidth as the target resource bed.

[0135] Optionally, the automation control server is also used to: obtain all candidate auxiliary test resources corresponding to each resource to be allocated in the target resource bed, obtain an auxiliary test resource list, and match the version and capability set of the candidate auxiliary test resources with the corresponding resources to be allocated; calculate the performance evaluation value of each candidate auxiliary test resource based on the performance parameters of each candidate auxiliary test resource; combine resources based on the auxiliary test resource list to obtain multiple auxiliary test resource groups, and calculate the total performance evaluation value of each auxiliary test resource group, the auxiliary test resource group includes a candidate auxiliary test resource corresponding to each resource to be allocated; obtain the auxiliary test resource group with the largest total performance evaluation value, obtain the target auxiliary test resource group, and use each candidate auxiliary test resource in the target auxiliary test resource group as each target auxiliary test resource.

[0136] Optionally, the automation control server is also used to: for each candidate auxiliary test resource, perform weighted summation based on the test success rate, test health, test success list, distance parameter, load parameter and bandwidth parameter of the candidate auxiliary test resource to obtain a performance evaluation value of the candidate auxiliary test resource, thereby obtaining a performance evaluation value of each candidate auxiliary test resource.

[0137] Optionally, the automation control server is further used to establish an automation communication tunnel between the target test bed and the target resource bed by connecting the topology switch of the target test bed and the topology switch of the target resource bed.

[0138] Optionally, the automation execution server is further used to: report a test log to the automation control server after finishing the test according to the second test information;

[0139] The automation control server is also used to: update the performance parameters of each target auxiliary test resource according to the test log and release each target auxiliary test resource, and delete the automation communication tunnel between the target test bed and the target resource bed.

[0140] In summary, the automated testing method and system provided by the embodiment of the present invention has the following beneficial effects compared with the prior art: (1) The embodiment of the present invention improves the architecture of multiple independent test beds in the existing automated testing system into a distributed and centralized hybrid architecture, and improves resource utilization by uniformly managing all auxiliary test resources and dynamically allocating them according to actual resource requirements. (2) The automated testing system provided by the embodiment of the present invention breaks the physical area restrictions of the original automated environment, allowing resources in the automated environment to be directly used across the three-layer network. When building an automated environment for each project, it is no longer necessary to consider the configuration of various resources. It only needs to configure test equipment, and a small amount of auxiliary test equipment can also be selected to be retained. It can avoid repeating the construction and allocation process of auxiliary test resources for each project, reducing labor costs. (3) The embodiment of the present invention provides a way to fully integrate resources and optimize the use of resources, so that idle resources that could only be allocated to each test bed before can be fully utilized, reducing the resource maintenance of each test bed and reducing the resource consumption of each test bed, greatly improving resource utilization. At the same time, all resources are uniformly managed and maintained, reducing management and maintenance costs and the labor testing costs of execution failure or inability to execute due to various resource versions and resource missing.

[0141] In several embodiments provided by the present invention, it should be understood that each box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, a program segment or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a sequence different from that marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs the specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions.

[0142] In addition, the functional modules in the various embodiments of the present invention may be integrated together to form an independent part, or each module may exist independently, or two or more modules may be integrated to form an independent part.

[0143] If the functions are implemented in the form of software function modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc., various media that can store program codes.

[0144] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. An automated testing method, characterized in that: Applied to an automated testing system, the automated testing system includes an automated control server, an automated execution server, multiple test beds and multiple resource beds, the resource bed includes multiple auxiliary testing resources, the method includes: The automation control server obtains multiple test cases in the task to be tested and a target test bed corresponding to the task to be tested, and groups the test cases according to the resource requirements of each test case to obtain multiple test case groups, wherein all test cases in the test case groups have the same resource requirements; When the test resources of the target test bed meet the resource requirements of the test case group, the automation control server generates first test information based on the test case group and the target test bed and sends the first test information to the automation execution server; The automation execution server performs testing according to the first test information; When the test resources of the target test bed do not meet the resource requirements of the test case group, the automation control server determines the resources to be supplemented of the test case group, determines the target resource bed based on the resources to be supplemented, obtains each target auxiliary test resource in the target resource bed, and establishes an automated communication tunnel between the target test bed and the target resource bed, then generates second test information based on the test case group, the target test bed and each of the target auxiliary test resources and sends it to the automation execution server; The automation execution server performs testing according to the second test information.

2. The method according to claim 1, characterized in that The automation control server stores the location of each of the test beds and the location and bandwidth of each of the resource beds; The automation control server determines a target resource bed based on the resources to be supplemented, including: The automation control server obtains each candidate resource bed from all resource beds, and all idle auxiliary test resources of the candidate resource bed include the resource to be supplemented; The automation control server calculates the distance between each candidate resource bed and the target test bed based on the location of each candidate resource bed and the location of the target test bed, and uses the candidate resource bed with the shortest distance and the largest bandwidth as the target resource bed.

3. The method according to claim 1, characterized in that The resources to be supplemented include multiple types of resources to be allocated, and the automation control server stores the version, capability set and performance parameters of each auxiliary test resource in the resource bed; The automation control server obtains each target auxiliary test resource in the target resource bed, including: The automation control server obtains all candidate auxiliary test resources corresponding to each resource to be allocated in the target resource bed, and obtains an auxiliary test resource list, wherein the versions and capability sets of the candidate auxiliary test resources match the corresponding resources to be allocated; The automation control server calculates a performance evaluation value of each of the candidate auxiliary test resources based on the performance parameters of each of the candidate auxiliary test resources; The automation control server combines resources based on the auxiliary test resource list to obtain multiple auxiliary test resource groups, and calculates a total performance evaluation value of each auxiliary test resource group, wherein the auxiliary test resource group includes a candidate auxiliary test resource corresponding to each resource to be allocated; The automation control server acquires the auxiliary test resource group with the largest total performance evaluation value, obtains a target auxiliary test resource group, and uses each candidate auxiliary test resource in the target auxiliary test resource group as each of the target auxiliary test resources.

4. The method according to claim 3, characterized in that The performance parameters include test success rate, test health, test success list, distance parameter, load parameter and bandwidth parameter. The test success rate represents the success rate of all test cases executed by the auxiliary test resources. The test health represents the success rate of all test cases executed by the auxiliary test resources within a preset historical period. The test success list includes all test cases successfully executed by the auxiliary test resources within the preset historical period. The distance parameter represents the relative distance between the auxiliary test resource and the resource bed to which it belongs. The load parameter represents the total number of test cases executed by the auxiliary test resources within the preset historical period. The automatic control server calculates a performance evaluation value of each of the candidate auxiliary test resources based on the performance parameter of each of the candidate auxiliary test resources, including: The automated control server performs weighted summation on each of the candidate auxiliary test resources based on the test success rate, test health, test success list, distance parameter, load parameter and bandwidth parameter of the candidate auxiliary test resource to obtain a performance evaluation value of the candidate auxiliary test resource.

5. The method according to claim 1, characterized in that Each of the test beds and each of the resource beds includes a topology switch; The automation control server establishes an automation communication tunnel between the target test bed and the target resource bed, including: The automation control server establishes an automation communication tunnel between the target test bed and the target resource bed by connecting the topology switch of the target test bed and the topology switch of the target resource bed.

6. The method according to claim 5, characterized in that The automation control server stores the performance parameters of each auxiliary test resource in the resource bed, and the method further includes: After the automation execution server finishes the test according to the second test information, reporting the test log to the automation control server; The automation control server updates the performance parameters of each of the target auxiliary test resources according to the test log, releases each of the target auxiliary test resources, and deletes the automation communication tunnel between the target test bed and the target resource bed.

7. An automated testing system, characterized in that: The automated testing system includes an automated control server, an automated execution server, a plurality of test beds and a plurality of resource beds, wherein the resource bed includes a plurality of auxiliary testing resources; The automation control server is used to obtain multiple test cases in the task to be tested and the target test bed corresponding to the task to be tested, and group the test cases according to the resource requirements of each test case to obtain multiple test case groups, and the resource requirements of all test cases in the test case group are the same; The automation control server is used for generating first test information based on the test case group and the target test bed and sending the first test information to the automation execution server when the test resources of the target test bed meet the resource requirements of the test case group; The automation execution server is used to perform testing according to the first test information; The automation control server is used to determine the resources to be supplemented of the test case group when the test resources of the target test bed do not meet the resource requirements of the test case group, and after determining the target resource bed based on the resources to be supplemented and obtaining each target auxiliary test resource in the target resource bed and establishing an automated communication tunnel between the target test bed and the target resource bed, generate second test information based on the test case group, the target test bed and each of the target auxiliary test resources and send it to the automation execution server; The automation execution server is used to perform testing according to the second test information.

8. The system according to claim 7, characterized in that The automation control server stores the location of each of the test beds and the location and bandwidth of each of the resource beds, and is further used for: Acquire each candidate resource bed from all resource beds, wherein all idle auxiliary test resources of the candidate resource bed include the resource to be supplemented; Based on the position of each candidate resource bed and the position of the target test bed, the distance between each candidate resource bed and the target test bed is calculated, and the candidate resource bed with the shortest distance and the largest bandwidth is used as the target resource bed.

9. The system according to claim 7, characterized in that The resources to be supplemented include multiple types of resources to be allocated. The automation control server stores the version, capability set and performance parameters of each auxiliary test resource in the resource bed. The automation control server is also used for: In the target resource bed, all candidate auxiliary test resources corresponding to each resource to be allocated are obtained to obtain an auxiliary test resource list, wherein the versions and capability sets of the candidate auxiliary test resources match the corresponding resources to be allocated; Calculating a performance evaluation value of each of the candidate auxiliary test resources based on the performance parameters of each of the candidate auxiliary test resources; Combining resources based on the auxiliary test resource list to obtain multiple auxiliary test resource groups, and calculating a total performance evaluation value of each auxiliary test resource group, wherein the auxiliary test resource group includes a candidate auxiliary test resource corresponding to each resource to be allocated; An auxiliary test resource group with a maximum total performance evaluation value is obtained to obtain a target auxiliary test resource group, and each candidate auxiliary test resource in the target auxiliary test resource group is used as each of the target auxiliary test resources.

10. The system according to claim 9, characterized in that The performance parameters include test success rate, test health, test success list, distance parameter, load parameter and bandwidth parameter. The test success rate represents the success rate of all test cases executed by the auxiliary test resource. The test health represents the success rate of all test cases executed by the auxiliary test resource within a preset historical period. The test success list includes all test cases successfully executed by the auxiliary test resource within a preset historical period. The distance parameter represents the relative distance between the auxiliary test resource and the resource bed to which it belongs. The load parameter represents the total number of test cases executed by the auxiliary test resource within the preset historical period. The automation control server is also used for: For each of the candidate auxiliary test resources, a weighted sum is performed based on the test success rate, test health, test success list, distance parameter, load parameter and bandwidth parameter of the candidate auxiliary test resource to obtain the performance evaluation value of the candidate auxiliary test resource.

Images