Automatic testing method and device, computer program product and electronic equipment

The automated testing method for NFV systems optimizes test case execution by evaluating test case values based on previous results, enhancing efficiency and reliability by prioritizing tests that are likely to uncover new issues.

CN119088690APending Publication Date: 2024-12-06CHINA TELECOM CORP LTD TECHNOLOGY INNOVATION CENTER +1
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411178489.1
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-26
Publication Date
2024-12-06

AI Technical Summary

Technical Problem

Existing automated testing methods for Network Function Virtualization (NFV) systems are inefficient and unsuitable, leading to low test efficiency and high risk, especially when dealing with complex virtual networks.

Method used

An automated testing method that dynamically determines the execution order of test cases based on their labels and similarities with previous test cases, using a combination of first and second value coefficients to prioritize test cases that are likely to discover new problems, thereby reducing the number of tests required to identify issues.

Benefits of technology

This approach enhances test efficiency and reliability by identifying more problems with fewer test cases, minimizing the impact on the tested environment and improving the applicability to NFV scenarios.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119088690A_ABST
    Figure CN119088690A_ABST
Patent Text Reader

Abstract

The embodiment of the invention relates to an automatic testing method and device, a computer program product and electronic equipment, and relates to the technical field of automatic testing, and the method comprises the following steps: determining a plurality of test cases according to a test range of a network function virtualization environment; determining a current test case from the plurality of test cases, and testing based on the current test case to obtain a test result; according to the test result, determining execution values of the remaining test cases except the current test case in the plurality of test cases; and determining a next test case from the remaining test cases based on the execution value, and performing an iterative test according to the next test case until a value condition is not met so as to perform an automatic test on the network function virtualization environment. The automatic testing efficiency can be improved.
Need to check novelty before this filing date? Find Prior Art

Description

Background Art

[0002] NFV, or Network Function Virtualization, is a combination of cloud computing and network technology. It is a method of virtualizing network services (such as routers, firewalls, and load balancers) that traditionally run on proprietary hardware so that they can be hosted on general-purpose hardware. NFV (Network Function Virtualization) involves many component levels and requires a lot of testing to ensure the normal operation of the system. Based on this, automated testing technology can be introduced to accelerate the testing and verification of NFV systems.

[0003] In the related art, test cases are grouped based on their characteristics, and several test cases are selected in each group for group testing, thereby reducing the probability of multiple test cases finding the same problem. However, the above method is not suitable for NFV automated testing scenarios, and the test efficiency is low. Summary of the invention

[0004] The purpose of the present disclosure is to provide an automated testing method, an automated testing device, a computer program product, and an electronic device, thereby overcoming the problem of low automated testing efficiency caused by limitations and defects of related technologies to at least a certain extent.

[0005] According to one aspect of the present disclosure, there is provided an automated testing method, comprising: determining a plurality of test cases according to a test scope of a network function virtualization environment; determining a current test case from the plurality of test cases, and performing testing based on the current test case to obtain a test result; determining, based on the test result, execution values ​​of remaining test cases in the plurality of test cases except the current test case; determining a next test case from the remaining test cases based on the execution value, and performing iterative testing based on the next test case until a value condition is not met, so as to perform automated testing on the network function virtualization environment.

[0006] In an exemplary embodiment of the present disclosure, determining the current test case from the multiple test cases includes: randomly selecting a test case from the multiple test cases as the current test case according to initial weights of the multiple test cases.

[0007] In an exemplary embodiment of the present disclosure, determining the execution value of the remaining test cases other than the current test case in the multiple test cases based on the test result includes: if the test result is a first type of result, determining the execution value of each remaining test case based on a label of each remaining test case and a similarity between each remaining test case and an executed test case; the executed test cases include at least the current test case; if the test result is a second type of result, determining the initial value of the remaining test cases as the execution value.

[0008] In an exemplary embodiment of the present disclosure, the value of each remaining test case is determined based on the label of each remaining test case and the similarity between each remaining test case and the executed test case, including: determining a first value coefficient of each remaining test case based on the label of each remaining test case and the label of the current test case; determining a second value coefficient of each remaining test case based on the similarity between each remaining test case and the executed test case; and determining the value of the remaining test cases for discovering new problems based on the first value coefficient and the second value coefficient.

[0009] In an exemplary embodiment of the present disclosure, the first value coefficient of each remaining test case is determined based on the label of each remaining test case and the label of the current test case, including: determining the first value coefficient of each remaining test case based on the number of labels of the current test case, the number of test cases that have the target label and the test results of the first type of results with the current test case, and the label ratio of the target test case; wherein the target test case is a test case that has the target label and the test result is the first type of result, and the label ratio is the ratio of the target label to the number of labels of the target test case.

[0010] In an exemplary embodiment of the present disclosure, the second value coefficient of each remaining test case is determined according to the similarity between each remaining test case and the executed test case, including: based on the number of test cases executed before the current test case and whose historical test results are all first type results, the second value coefficient is determined according to the similarity between the remaining test cases and the executed test cases; wherein the similarity is determined according to the historical test results of the remaining test cases and the executed test cases.

[0011] In an exemplary embodiment of the present disclosure, determining the execution value based on the first value coefficient and the second value coefficient includes: determining the execution value based on the product of the first value coefficient and the second value coefficient; if the execution value is greater than or equal to a value threshold, keeping the execution value unchanged; if the execution value is less than the value threshold, configuring the execution value as a preset parameter.

[0012] In an exemplary embodiment of the present disclosure, the step of determining a next test case from the remaining test cases based on the execution value, and performing iterative testing according to the next test case until the value condition is not satisfied, includes: if the execution values ​​of multiple remaining test cases satisfy the value condition, using the execution value of each test case as a weight, determining the next test case from the remaining test cases, and performing testing according to the next test case to update the test results; determining the execution values ​​of the remaining test cases except the next test case based on the updated test results; and continuing to select test cases for iterative testing according to the execution value until the execution values ​​of multiple remaining test cases fail to satisfy the value condition.

[0013] According to one aspect of the present disclosure, an automated testing device is provided, including: a test case range determination module, used to determine multiple test cases according to the test range of a network function virtualization environment; a current test case selection module, used to determine a current test case from multiple test cases, and perform testing based on the current test case to obtain a test result; an execution value determination module, used to determine the execution value of the remaining test cases in the multiple test cases except the current test case according to the test result; an iterative testing module, used to determine the next test case from the remaining test cases based on the execution value, and perform iterative testing based on the next test case until the value condition is not met, so as to perform automated testing on the network function virtualization environment.

[0014] According to one aspect of the present disclosure, a computer program product is provided, including a computer program, wherein when the computer program is executed by a processor, the computer program implements any one of the above-mentioned automated testing methods.

[0015] According to one aspect of the present disclosure, an electronic device is provided, comprising: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to perform any one of the above-mentioned automated testing methods by executing the executable instructions.

[0016] In the technical solution provided in the embodiment of the present disclosure, on the one hand, based on the test results of the previous test cases represented by the current test case, the value of the remaining test cases is evaluated, and the execution order of the remaining test cases can be dynamically determined, thereby effectively improving the test efficiency and test reliability, and improving the applicability to the test scenario. On the other hand, the execution value of the remaining test cases can be determined, and then the test cases that need to be executed can be selected according to the execution value to ignore the test cases with lower execution value, so that more problems can be discovered through as few test cases as possible, reducing the impact on the tested environment.

[0017] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The accompanying drawings herein are incorporated into the specification and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the specification are used to explain the principles of the present disclosure. Obviously, the accompanying drawings described below are only some embodiments of the present disclosure, and for ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without creative work.

[0019] Figure 1 A flowchart of an automated testing method in an embodiment of the present disclosure is schematically shown.

[0020] Figure 2 The framework diagram of the automated testing platform in the embodiment of the present disclosure is schematically shown.

[0021] Figure 3 A schematic diagram for determining a test case in an embodiment of the present disclosure is schematically shown.

[0022] Figure 4 The specific process flow diagram of the automated test in the embodiment of the present disclosure is schematically shown.

[0023] Figure 5 The block diagram of the automated testing device in the embodiment of the present disclosure is schematically shown.

[0024] Figure 6 The block diagram of the electronic device according to the embodiment of the present disclosure is schematically shown. DETAILED DESCRIPTION

[0025] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in a variety of forms and should not be construed as being limited to the examples set forth herein; on the contrary, these embodiments are provided so that the present disclosure will be more comprehensive and complete, and the concepts of the example embodiments are fully conveyed to those skilled in the art. The described features, structures, or characteristics may be combined in one or more embodiments in any suitable manner. In the following description, many specific details are provided to provide a full understanding of the embodiments of the present disclosure. However, those skilled in the art will appreciate that the technical solutions of the present disclosure may be practiced while omitting one or more of the specific details, or other methods, components, devices, steps, etc. may be adopted. In other cases, known technical solutions are not shown or described in detail to avoid obscuring various aspects of the present disclosure.

[0026] In addition, the accompanying drawings are only schematic illustrations of the present disclosure and are not necessarily drawn to scale. The same reference numerals in the figures represent the same or similar parts, and their repeated description will be omitted. Some of the block diagrams shown in the accompanying drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities can be implemented in software form, or implemented in one or more hardware modules or integrated circuits, or implemented in different networks and / or processor devices and / or microcontroller devices.

[0027] In some embodiments, traditional NFV automated testing executes all test cases in sequence, and then analyzes the test cases that fail the test to find problems in the tested environment. Its disadvantages are low efficiency and high risk. On the one hand, the number of test cases executed is not proportional to the number of problems found, because several test cases may point to the same problem. Especially when time is tight and problems need to be found and fixed as soon as possible, the effect of traditional methods is very limited. On the other hand, if there are problems in the tested environment and the automated test script does not fully consider abnormal situations, there is a higher probability of bugs when executing all test cases, and even damage to the environment itself.

[0028] Based on this, the related technology groups test cases based on their characteristics, and selects several test cases in each group for group testing, thereby reducing the probability of multiple test cases finding the same problem. The problem is that it is difficult to decouple NFV test cases. For example, if a virtual network is not accessible, the computing, storage, and network use cases using the network may fail. Therefore, it is not suitable for NFV automated testing scenarios and has low automated testing efficiency.

[0029] In order to solve the above technical problems, an automated testing method is provided in the embodiment of the present disclosure, which can be applied to NFV automated testing scenarios and automated testing platforms. Figure 1As shown in , it mainly includes the following steps:

[0030] In step S110, a plurality of test cases are determined according to the test scope of the network function virtualization environment;

[0031] In step S120, a current test case is determined from a plurality of test cases, and a test is performed based on the current test case to obtain a test result;

[0032] In step S130, according to the test result, the execution value of the remaining test cases in the multiple test cases except the current test case is determined;

[0033] In step S140, a next test case is determined from the remaining test cases based on the execution value, and iterative testing is performed according to the next test case until a test end condition is met, so as to perform automated testing on the network function virtualization environment.

[0034] The technical solution provided in the embodiments of the present disclosure, on the one hand, evaluates the value of the remaining test cases based on the test results of the previous test cases represented by the current test case, and can dynamically determine the execution order of the remaining test cases, thereby effectively improving the test efficiency and test reliability, and improving the applicability to the test scenario. On the other hand, it can determine the execution value of the remaining test cases, and then select the test cases that need to be executed according to the execution value, so as to ignore the test cases with lower execution value, and can discover more problems through as few test cases as possible, reducing the impact on the tested environment.

[0035] Next, refer to Figure 1 The automated testing method in the embodiment of the present disclosure is specifically described in detail.

[0036] In step S110, a plurality of test cases are determined according to a test scope of the network function virtualization environment.

[0037] In the embodiments of the present disclosure, network function virtualization NFV is a combination of cloud computing technology and network technology. It is a method of virtualizing network services (such as routers, firewalls, and load balancers) that traditionally run on proprietary hardware so that they can be hosted on general-purpose hardware.

[0038] The network function virtualization environment refers to the environment under test, i.e., the NFV environment, which can be the execution environment of the test case. Automated testing can be performed on the automated testing platform. The automated testing platform can interact with the network function virtualization environment to perform automated testing on the network function virtualization environment.

[0039] When performing automated testing on a network function virtualization environment, multiple test cases may be determined according to the test scope of the network function virtualization environment. The test scope is used to indicate the coverage of the automated test. The test cases may be the same or different depending on the test scope. The number of test cases may be determined according to actual needs. Multiple test cases may include test case labels, and different test cases may have the same or different test case labels.

[0040] For example, if you want to perform automated testing on an NFV environment, five test cases are pre-selected based on its test scope, namely test case A (label: NFVI-Virtual Computing, NFVI-Virtual Storage), test case B (label: NFVI-Virtual Computing), test case C (label: MANO-VNF Manager), test case D (label: MANO-NFV Orchestrator), and test case E (label: NFVI-Virtual Storage, NFVI-Virtual Network).

[0041] Next, in step S120, a current test case is determined from a plurality of test cases, and a test is performed based on the current test case to obtain a test result.

[0042] In an embodiment of the present disclosure, after determining multiple test cases, a current test case can be determined from the multiple test cases. The current test case refers to a test case that can be currently used to perform automated testing. Exemplarily, based on the initial weight of each test case, a test case can be randomly selected as the current test case to initiate the test. The initial weight is used to represent the initial value of each test case, that is, the value it has before the automated test is performed. It should be noted that when the automated test is performed for the first time, the initial weights of the multiple test cases are equal, and the initial weight can be, for example, 1. For example, test case A can be randomly selected as the current test case based on the initial value.

[0043] Further, a test may be performed based on the selected current test case to determine a test result. Exemplarily, the automated test platform may run the current test case to obtain a test result corresponding to the current test case. The test result may be a first type result or a second type result, the first type result may be a failure, and the second type result may be a pass. The test result may be recorded in a test data management module of the automated test platform.

[0044] Continue to refer Figure 1As shown in, in step S130, the execution values ​​of the remaining test cases in the multiple test cases except the current test case are determined according to the test results.

[0045] In an embodiment of the present disclosure, among multiple test cases, except for the selected current test case, other test cases can be used as remaining test cases. After determining that the test result of the current test case is a first type of result, that is, when the test result is failed, the execution value of the remaining test cases in the multiple test cases except the current test case can be calculated based on the test result. The execution value is used to represent the value of the remaining test cases for discovering new problems in the network function virtualization environment. The execution value here may be different from the initial value. When it is determined that the test result is a second type of result, that is, when the test result is passed, the initial value of the remaining test cases will not be affected by the test result, so the initial value of the remaining test cases can be kept unchanged, and the initial value of the remaining test cases can be directly used as the execution value of the remaining test cases.

[0046] In some embodiments, for test results of the first type, the execution value of each remaining test case can be determined from two dimensions, label and similarity, based on the label of each remaining test case and the similarity between each remaining test case and the executed test case. Exemplarily, a first value coefficient and a second value coefficient can be determined, and the value of each remaining test case can be further determined based on the first value coefficient and the second value coefficient. First, the first value coefficient of each remaining test case can be determined based on the label of each remaining test case and the label of the current test case. The first value coefficient is used to indicate the possibility of discovering new problems based on the test case obtained by label analysis.

[0047] Exemplarily, the first value coefficient of each remaining test case can be determined according to the number of labels of the current test case, the number of test cases with the same target label as the current test case and the test results of the first type of result, and the label ratio of the target test case. Among them, the number of labels of the current test case refers to the total number of labels of the current test case. In some embodiments, candidate test cases with the same target label as the current test case can be determined, and further according to the candidate test cases with the test results of the first type of result, the number of test cases with the same target label as the current test case and the test results of the first type of result can be determined, and the number of test cases can be specifically represented by m. Among them, the target label can be the i-th label, and specifically can be each label of the current test case. The target test case refers to the j-th test case among the test cases with the target label and the test results of the first type of label, and the label ratio of the target test case refers to the ratio of the target label of the j-th test case to the number of labels of the j-th test case. Among them, the i-th label refers to each label of the current test case. j is greater than or equal to 1, and j is less than the number of test cases.

[0048] Exemplarily, the first value coefficient can be determined based on the degree of overlap between the label of the current test case and the labels of the remaining test cases. Exemplarily, the first difference between the preset parameter and the label ratio of the target test case can be determined, and the first difference can be multiplied based on the number of test cases to obtain a first result; the second difference between the preset parameter and the first result can be further determined, and the second difference can be multiplied based on the number of labels of the current test case to obtain a second result; the preset parameter is subtracted from the second result to obtain the first value coefficient. The preset parameter can be determined according to actual needs, for example, it can be 1 or other values. Here, the preset parameter of 1 is used as an example for explanation.

[0049] In some embodiments, the test case label analysis module may apply to the test data management to obtain the labels of all executed test cases with a test result of "failed", and apply to the test case collection to obtain the labels of all test cases. Based on the above data, the test case label analysis module may analyze the first value coefficient of the test case from the label through formula (1), which can be specifically expressed as:

[0050]

[0051] Where n is the number of labels of the current test case; m is the number of test cases with the same target label and a test result of "failed"; P ij It is the ratio of the i-th label of the j-th test case among the test cases with the i-th label and the test result of “fail” to the total number of labels of the j-th test case.

[0052] Furthermore, the second value coefficient of each remaining test case may be determined according to the similarity between each remaining test case and the executed test case. The executed test case may include at least the current test case. In the first round of testing, the executed test case may include the current test case; in the testing process after the first round, the executed test case may also include a test case that has obtained a test result and is located before the current test case.

[0053] Exemplarily, the second value coefficient is determined based on the number of test cases whose historical test results are all first-type results before the current test case is executed, and the similarity between the remaining test cases and the executed test cases. The historical test results refer to the test results of each test case in the same test environment. Specifically, the third difference between the preset parameter and the similarity can be determined, and the third difference can be multiplied based on the number of test cases to obtain the second value coefficient. Among them, the similarity can be determined based on the historical test results of the remaining test cases and each executed test case.

[0054] The second value coefficient can be expressed according to formula (2):

[0055]

[0056] Where q is the number of test cases whose historical test results are all failed before the current test case is executed, and Sk is the similarity between the current test case and the kth test case in the executed test cases. The similarity can be calculated by formula (3):

[0057]

[0058] Among them, H is used to represent the historical test results of the test case. When the historical test result is passed, it can be recorded as 1, and when the historical test result is failed, it can be recorded as -1. H1 represents the historical test results of test case A, and H2 represents the historical test results of test case B. For example, the results of test case A in the past five test environments are H1 = [1, 1, -1, -1, 1].

[0059] After obtaining the first value coefficient and the second value coefficient, the first value coefficient and the second value coefficient can be pushed to the test case value evaluation module. The test case value evaluation module is used to evaluate the execution value of the test case. Exemplarily, based on the comparison result of the calculated execution value and the value threshold, a comprehensive evaluation can be performed using the first value coefficient and the second value coefficient to calculate the execution value. Further, the execution value can be kept unchanged, or the execution value can be assigned a fixed value, which can be 0, for example. Exemplarily, when performing a comprehensive evaluation, the first value coefficient and the second value coefficient can be directly multiplied to obtain the execution value. Further, when the comparison result of the execution value and the value threshold is that the execution value is greater than or equal to the value threshold, the execution value can be kept unchanged. When the comparison result of the execution value and the value threshold is that the execution value is less than the value threshold, the execution value can be reassigned to determine it as a fixed value of 0, as shown in formula (4):

[0060]

[0061] In the disclosed embodiment, based on the test results of the preceding test cases, the execution values ​​of the remaining test cases can be accurately determined through the labels and similarities of the test cases.

[0062] Next, continue to refer to Figure 1 As shown in , in step S140, a next test case is determined from the remaining test cases based on the execution value, and iterative testing is performed according to the next test case until the value condition is not met, so as to perform automated testing on the network function virtualization environment.

[0063] In the disclosed embodiment, after calculating the execution value, the test case value assessment module can push the execution value to the test case orchestration module. When the execution values ​​of multiple remaining test cases meet the value condition, the test case orchestration module can use the execution value as a weight, determine the probability of being selected according to the weight represented by the execution value, and randomly select the next test case from the remaining test cases based on the probability. The value condition may be that the execution values ​​of multiple remaining test cases are not all 0, that is, there are remaining test cases whose execution values ​​are not 0. Specifically, the execution value of the remaining test cases is used as a weight, the probability of the remaining test case being selected is determined according to the weight, and one of the remaining test cases is randomly selected as the next test case from the remaining test cases according to the probability of being selected, and the test is performed according to the next test case to update the test results; further, the execution value of the remaining test cases except the next test case can be updated according to the updated test results. The method of updating the execution value can still be: if the updated test result obtained according to the next test case is the first type of result, determine the first value coefficient of each remaining test case according to the label of each remaining test case and the degree of repetition of the label of the next test case; determine the second value coefficient of each remaining test case according to the similarity between each remaining test case and the next test case, and according to the similarity between each remaining test case and the current test case; determine the execution value of the remaining test cases for discovering new problems according to the first value coefficient and the second value coefficient. If the updated test result is the second type of result, the execution value calculated according to the current test case can be directly used as the updated execution value. Among them, the greater the execution value, the greater the probability of the test case being selected.

[0064] After determining the execution value of each remaining test case, the execution value can be used as a weight to continue selecting test cases from the remaining test cases except the next test case, to perform iterative testing based on the selected test cases to obtain test results, and to continue updating the execution value of the remaining test cases when the test results are first-type results. Repeat the steps of selecting test cases according to the execution value, performing iterative testing according to the test cases to obtain test results, and updating the execution value according to the test results until the execution value of the remaining test cases does not meet the value condition, that is, when the execution values ​​of the remaining test cases are all 0 or there is no execution value that is not 0, stop the iterative testing process to complete the automated testing of the network function virtualization environment.

[0065] After determining the test cases and the execution order of the test cases, automated testing can be performed in sequence according to the execution order to obtain the test results of the network function virtualization environment. The test result of the network function virtualization environment can be a test pass or a test fail. When the test result of the network function virtualization environment is a test fail, the problems existing in the network function virtualization environment can be displayed to facilitate subsequent adjustments to the network function virtualization environment.

[0066] In some embodiments, assuming that an NFV environment is to be automated tested, five test cases are pre-selected based on its test scope, namely, test case A (label: NFVI-Virtual Computing, NFVI-VirtualStorage), test case B (label: NFVI-Virtual Computing), test case C (label: MANO-VNFManager), test case D (label: MANO-NFV Orchestrator), and test case E (label: NFVI-VirtualStorage, NFVI-Virtual Network).

[0067] Step 1: First, the automated test is initiated. The test case arrangement module randomly selects a test case as the current test case based on the weight represented by the initial value of each test case to initiate the test. When the test cases are executed for the first time, the weights of all test cases are equal. For example, test case A is randomly selected. In this round, the executed test cases include the current test case, that is, test case A.

[0068] Step 2: The automated testing platform executes the selected test case, and the test result of test case A is a first type result of "failed", and the test result is recorded in the test data management module.

[0069] Step 3: Test case label analysis module calculates the first value coefficient W of each remaining test case based on the test results t . W of test case B t2 =1-[1-(1-0.5)]=0.5, W of test case E t5 =1-[1-(1-0.5)]=0.5, test cases C and D do not have the same labels as test case A, that is, there is no label overlap, so W of test case C t3 =1, W of test case D t4 =1.

[0070] Step 4: Test case similarity analysis module calculates the second value coefficient W of each remaining test case based on the test results sAssume that the historical test results of test case A in the past five test environments are H1 = [1, 1, -1, -1, 1], the historical test results of test case B in the same five environments are H2 = [-1, -1, 1, 1, -1], the historical test results of test case C are H3 = [1, 1, -1, -1, 1], the historical test results of test case D are H4 = [1, -1, 1, 1, 1], and the historical test results of test case E are H5 = [-1, 1, 1, 1, -1]. Then the similarity between test case B and test case A is The corresponding second value coefficient The same method is used to calculate the similarity between test case C, test case D, test case E and test case A, which are 1, -0.2, and -0.6 respectively. The corresponding second value coefficient W s3 =0, W s4 =0.6, W s5 =0.8.

[0071] Step 5: The value evaluation module calculates the execution value of the remaining test cases based on the first value coefficient and the second value coefficient. Assume that the value threshold T set based on experience is 0.4. The value of test case B W2 = W t2 ×W s2 =0.5, similarly, the execution value of test case C is W3=0, the execution value of test case D is W4=0.6, and the execution value of test case E is W5=0.4.

[0072] At this point, there are test cases whose execution value is not 0 among the remaining test cases, which meet the value condition. Therefore, steps one to five can be repeated to continue screening the remaining test cases for iterative testing.

[0073] Step 1: The test case scheduling module randomly selects one of the remaining test cases based on the execution value of each remaining test case to initiate the test. At this time, the execution values ​​of test cases B, C, D, and E are 0.5, 0, 0.6, and 0.4 respectively. Based on the execution value, the remaining test case B is randomly selected.

[0074] Step 2: The automated testing platform executes the remaining selected test cases, and the test results of the remaining test cases B are the second type results "passed", and the test results are recorded in the test data management module.

[0075] Steps 3 to 5: Since the test result of the remaining test case B is "passed", according to the value evaluation method, the execution value of each remaining test case is not affected, so the execution value of the remaining test cases can be kept unchanged, that is, the execution value of test case C is W3=0, the execution value of test case D is W4=0.6, and the execution value of test case E is W5=0.4.

[0076] At this point, there are test cases whose execution value is not 0 among the remaining test cases, which meet the value condition. Therefore, steps one to five can be repeated to continue screening the remaining test cases for iterative testing.

[0077] Step 1: The test case arrangement module randomly selects a test case based on the execution value of each remaining test case to initiate the test. At this time, the execution values ​​of the remaining test cases C, D, and E are 0, 0.6, and 0.4 respectively. Test case E is randomly selected based on the execution value. In this round, the executed test cases include test case A and test case E.

[0078] Step 2: The automated testing platform executes the selected test case, and the test result of test case E is a first type result of "failed", and the test result is recorded in the test data management module.

[0079] Step 3: Test case label analysis module calculates the first value coefficient W of each remaining test case based on the test results t Since test cases C and D have no label overlap with the executed test cases A and E, W t3 =1, W t4 =1.

[0080] Step 4: Test case similarity analysis module calculates the second value coefficient W of each remaining test case based on the test results s The similarity between test case D and test case A is -0.2, and the similarity between test case D and test case E is Therefore, the second value coefficient corresponding to test case D is Similarly, the second value coefficient W of test case C s3 =0.

[0081] Step 5: The value evaluation module calculates the execution value of the remaining test cases based on the first value coefficient and the second value coefficient. The execution value of test case C is W3 = W t3 ×W s3 =1×0=0. The execution value of test case D is W4=W t4 ×W s4 =1×0.36=0.36. Since the execution value of the test case is less than the value threshold T, the execution value can be set to 0.

[0082] Based on this, the execution values ​​of the remaining test cases are all 0, which does not meet the value condition. Therefore, the iterative testing process can be ended and the automated testing of the network function virtualization environment can be completed.

[0083] Figure 2 The schematic diagram of the automated test is shown in Figure 2As shown in , the automated testing platform may include a test case set and a test data management module, in addition to which a test case scheduler may be included. The test case set refers to a group of test cases selected manually to verify that the functions and performance of the tested environment meet the requirements. The test data management module is used to manage the test results (pass / fail), logs, etc. of the test cases. The test case scheduler is used to analyze the value of test cases for discovering new problems, and then schedule test cases according to the value of the test cases.

[0084] Exemplarily, the test case orchestrator may include a use case similarity analysis module, a use case label analysis module, a use case value assessment module, and a test case orchestration module. Among them, the use case similarity analysis module is used to evaluate the similarity between the test case and the test case that failed the test, so as to analyze the possibility of discovering new problems in the use case. The use case label analysis module: analyzes the possibility of discovering new problems in the test case based on the label, and the label is used to mark the relationship between the test case and the NFV component. The use case value assessment module is used to evaluate the value of the use case based on the results of the use case similarity analysis module and the use case label analysis module. The test case orchestration module: arranges the test cases based on the value of the use cases.

[0085] refer to Figure 2 As shown in , for example, through the test case scheduler, test case A, test case E, test case G, and test case N can be selected in sequence. Through the selected test cases, the tested environment can be automatically tested.

[0086] In the embodiment of the present disclosure, based on the test result of the current test case, the execution value of the remaining test cases is evaluated to dynamically determine the order of the test cases to be executed in the subsequent order according to the execution value. The test case label analysis module evaluates the execution value of the test case based on the relationship between the test case and the NFV component. If many test cases that fail the test involve a certain NFV component label, and the execution value of the remaining test cases that also involve the label is low. The test case similarity analysis module evaluates the execution value of the test case based on the similarity between the test cases. If two test cases often pass together and fail together, the problems they find may be the same, so when one test case fails, the execution value of other test cases with high similarity is low. After determining the execution value, it can be identified whether the found problem is a new problem based on the execution value, thereby improving the test efficiency. The test case scheduler in the embodiment of the present disclosure adopts a dynamic test case scheduling method. Based on the test results of the previous test cases, the execution value of the remaining test cases is judged, and the test case scheduling is driven by the execution value, which can effectively improve the test efficiency. In addition, the similarity between test cases, the relationship between test cases and NFV components are comprehensively considered, and the overall evaluation and judgment of the execution value of test cases are more effective in improving test efficiency. NFV components refer to the modules of the system under test.

[0087] Figure 3 A schematic diagram of arranging test cases is shown in FIG. Figure 3 As shown in , suppose that test case A, test case B, test case C, test case D, and test case E are included. In the first test, test case A can be selected and the test result of test case A can be determined. When the test result is not passed, the execution value of test case B, test case C, test case D, and test case E is calculated. If the execution value is not all 0, continue to select test cases from test case B, test case C, test case D, and test case E for the second test, for example, select test case B, and determine the test result of test case B. When the test result is passed, keep the execution value of test case B, test case C, test case D, and test case E unchanged. If the execution value is not all 0, continue to select test cases from test case C, test case D, and test case E for the third test, for example, select test case E, and determine the test result of test case E. When the test result is not passed, calculate the execution value of test case C and test case D. If the execution value is all 0, stop the entire test process.

[0088] Figure 4 A specific flow chart for performing the test is schematically shown in FIG. Figure 4 As shown in , it mainly includes the following steps:

[0089] In step S400, a plurality of test cases are obtained;

[0090] In step S402, the test case arrangement module randomly selects a test case based on the execution weight;

[0091] In step S404, the test case is executed and the test result is recorded;

[0092] In step S406, the test case analysis module calculates the first value coefficient of the remaining test cases;

[0093] In step S408, the test case similarity analysis module calculates the second value coefficients of the remaining test cases;

[0094] In step S410, the execution value of the remaining test cases is calculated according to the first value coefficient and the second value coefficient;

[0095] In step S412, determine whether there is a test case whose execution value is not 0; if so, go to step S402 to continue execution; if not, end.

[0096] The technical solution provided in the embodiments of the present disclosure proposes a test case scheduler for NFV automated test scenarios, which adopts a dynamic test case scheduling method, and judges the execution value of the remaining test cases from the two aspects of test case similarity and test case labels based on the execution status of the preceding test cases, so as to dynamically adjust the subsequent test cases and the execution order of the test cases according to the execution value. Through the above method, as few test cases as possible can be executed, and as many problems as possible can be found in the shortest possible time, thereby improving the test efficiency and reducing the impact on the tested environment.

[0097] The present disclosure also provides an automated testing device. Figure 5 As shown, the automated testing device 500 mainly includes the following modules:

[0098] A test case scope determination module 501 is used to determine a plurality of test cases according to a test scope of a network function virtualization environment;

[0099] A current test case selection module 502 is used to determine a current test case from multiple test cases, and perform a test based on the current test case to obtain a test result;

[0100] An execution value determination module 503, configured to determine the execution values ​​of the remaining test cases in the plurality of test cases except the current test case according to the test result;

[0101] The iterative testing module 504 is used to determine the next test case from the remaining test cases based on the execution value, and perform iterative testing according to the next test case until the test end condition is met, so as to perform automated testing on the network function virtualization environment.

[0102] In an exemplary embodiment of the present disclosure, determining the current test case from the multiple test cases includes: randomly selecting a test case from the multiple test cases as the current test case according to initial weights of the multiple test cases.

[0103] In an exemplary embodiment of the present disclosure, determining the execution value of the remaining test cases other than the current test case in the multiple test cases based on the test result includes: if the test result is a first type of result, determining the execution value of each remaining test case based on a label of each remaining test case and a similarity between each remaining test case and an executed test case; the executed test cases include at least the current test case; if the test result is a second type of result, determining the initial value of the remaining test cases as the execution value.

[0104] In an exemplary embodiment of the present disclosure, the value of each remaining test case is determined based on the label of each remaining test case and the similarity between each remaining test case and the executed test case, including: determining a first value coefficient of each remaining test case based on the label of each remaining test case and the label of the current test case; determining a second value coefficient of each remaining test case based on the similarity between each remaining test case and the executed test case; and determining the value of the remaining test cases for discovering new problems based on the first value coefficient and the second value coefficient.

[0105] In an exemplary embodiment of the present disclosure, the first value coefficient of each remaining test case is determined based on the label of each remaining test case and the label of the current test case, including: determining the first value coefficient of each remaining test case based on the number of labels of the current test case, the number of test cases that have the target label and the test results of the first type of results with the current test case, and the label ratio of the target test case; wherein the target test case is a test case that has the target label and the test result is the first type of result, and the label ratio is the ratio of the target label to the number of labels of the target test case.

[0106] In an exemplary embodiment of the present disclosure, the second value coefficient of each remaining test case is determined according to the similarity between each remaining test case and the executed test case, including: based on the number of test cases executed before the current test case and whose historical test results are all first type results, the second value coefficient is determined according to the similarity between the remaining test cases and the executed test cases; wherein the similarity is determined according to the historical test results of the remaining test cases and the executed test cases.

[0107] In an exemplary embodiment of the present disclosure, determining the execution value based on the first value coefficient and the second value coefficient includes: determining the execution value based on the product of the first value coefficient and the second value coefficient; if the execution value is greater than or equal to a value threshold, keeping the execution value unchanged; if the execution value is less than the value threshold, configuring the execution value as a preset parameter.

[0108] In an exemplary embodiment of the present disclosure, the step of determining a next test case from the remaining test cases based on the execution value, and performing iterative testing according to the next test case until the value condition is not satisfied, includes: if the execution values ​​of multiple remaining test cases satisfy the value condition, using the execution value of each test case as a weight, determining the next test case from the remaining test cases, and performing testing according to the next test case to update the test results; determining the execution values ​​of the remaining test cases except the next test case based on the updated test results; and continuing to select test cases for iterative testing according to the execution value until the execution values ​​of multiple remaining test cases fail to satisfy the value condition.

[0109] It should be noted that the specific details of each module in the above-mentioned automated testing device have been described in detail in the corresponding automated testing method, so they will not be repeated here.

[0110] It should be noted that, although several modules or units of the device for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiments of the present disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. On the contrary, the features and functions of one module or unit described above can be further divided into multiple modules or units to be embodied.

[0111] In addition, although the steps of the method in the present disclosure are described in a specific order in the drawings, this does not require or imply that the steps must be performed in this specific order, or that all the steps shown must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps, etc.

[0112] In an exemplary embodiment of the present disclosure, an electronic device capable of implementing the above method is also provided.

[0113] Those skilled in the art will appreciate that various aspects of the present disclosure may be implemented as systems, methods or program products. Therefore, various aspects of the present disclosure may be specifically implemented in the following forms, namely: complete hardware implementation, complete software implementation (including firmware, microcode, etc.), or a combination of hardware and software, which may be collectively referred to herein as "circuits", "modules" or "systems".

[0114] Refer to the following Figure 6 The electronic device 600 according to this embodiment of the present disclosure is described. Figure 6 The electronic device 600 shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0115] like Figure 6 As shown, the electronic device 600 is in the form of a general computing device. The components of the electronic device 600 may include, but are not limited to: the at least one processing unit 610, the at least one storage unit 620, a bus 630 connecting different system components (including the storage unit 620 and the processing unit 610), and a display unit 640.

[0116] The storage unit stores program codes, which can be executed by the processing unit 610, so that the processing unit 610 performs the steps according to various exemplary embodiments of the present disclosure described in the above “Exemplary Method” section of this specification. For example, the processing unit 610 can perform the following steps: Figure 1 Follow the steps shown in .

[0117] The storage unit 620 may include a readable medium in the form of a volatile storage unit, such as a random access storage unit (RAM) 6201 and / or a cache storage unit 6202 , and may further include a read-only storage unit (ROM) 6203 .

[0118] The storage unit 620 may also include a program / utility 6204 having a set (at least one) of program modules 6205, such program modules 6205 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0119] Bus 630 may represent one or more of several types of bus structures, including a memory unit bus or memory unit controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.

[0120] The electronic device 600 may also communicate with one or more external devices 700 (e.g., keyboards, pointing devices, Bluetooth devices, etc.), may also communicate with one or more devices that enable a user to interact with the electronic device 600, and / or communicate with any device that enables the electronic device 600 to communicate with one or more other computing devices (e.g., routers, modems, etc.). Such communication may be performed via an input / output (I / O) interface 650. Furthermore, the electronic device 600 may also communicate with one or more networks (e.g., local area networks (LANs), wide area networks (WANs), and / or public networks, such as the Internet) via a network adapter 660. As shown, the network adapter 660 communicates with other modules of the electronic device 600 via a bus 630. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device 600, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems, etc.

[0121] It should be noted that, in some embodiments of the present disclosure, a computer program product is also provided. The computer program product includes a computer program. When the computer program is executed by a processor, the above method is implemented.

[0122] In one embodiment, a computer program product may be a tangible product containing a computer program, such as a computer-readable storage medium storing a computer program. The readable storage medium may be a storage medium based on electrical, magnetic, optical, electromagnetic, infrared, and other signals, including but not limited to: random access memory (RAM), read-only memory (ROM), magnetic tape, floppy disk, flash memory (Flash), mechanical hard disk (HDD), solid-state drive (SSD), and the like. Exemplarily, a computer program product may be implemented as a non-volatile storage medium storing a computer program, such as a read-only memory, a NAND flash memory (Nand Flash), and the like. In one embodiment, a computer program product may be an intangible product containing a computer program. Exemplarily, a computer program product may be implemented as a virtual digital product, such as a digital file such as an executable file, an installation package, etc. storing a computer program.

[0123] The code of the computer program can be written in one or more programming languages. Programming languages ​​such as C language, Java, C++, etc. The program code can be executed entirely on the user computing device, or partially on the user computing device, or as a separate software package, or partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In the case of a remote computing device, the remote computing device can be connected to the user computing device through any type of network, such as a local area network (LAN), a wide area network (WAN), etc., or can be connected to an external computing device (e.g., an Internet connection provided by an operator).

[0124] The computer program may be carried or transmitted via electrical, magnetic, optical, electromagnetic, infrared, or other signals. The electronic device may convert the signal carrying the computer program into a digital signal, thereby running the computer program. When the computer program is run on an electronic device, its code is used to enable the electronic device to execute (more specifically, the processor of the electronic device may execute) the method steps of various exemplary embodiments of the present disclosure.

[0125] Through the description of the above implementation, it is easy for those skilled in the art to understand that the example implementation described here can be implemented by software, or by software combined with necessary hardware. Therefore, the technical solution according to the implementation of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, including several instructions to enable a computing device (which can be a personal computer, a server, a terminal device, or a network device, etc.) to execute the method according to the implementation of the present disclosure.

[0126] In addition, the above-mentioned figures are only schematic illustrations of the processes included in the method according to the exemplary embodiments of the present disclosure, and are not intended to be limiting. It is easy to understand that the processes shown in the above-mentioned figures do not indicate or limit the time sequence of these processes. In addition, it is also easy to understand that these processes can be performed synchronously or asynchronously, for example, in multiple modules.

[0127] It should be noted that, although several modules or units of the device for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiments of the present disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. On the contrary, the features and functions of one module or unit described above can be further divided into multiple modules or units to be embodied.

[0128] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing what is disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary technical means in the art that are not disclosed in the present disclosure. The specification and embodiments are to be considered merely as exemplary, and the true scope and spirit of the present disclosure are indicated by the claims.

[0129] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. An automated testing method, characterized in that: include: Determine multiple test cases based on the test scope of the network function virtualization environment; Determine a current test case from multiple test cases, and perform a test based on the current test case to obtain a test result; Determining, according to the test result, the execution values ​​of the remaining test cases in the plurality of test cases except the current test case; A next test case is determined from the remaining test cases based on the execution value, and iterative testing is performed according to the next test case until the value condition is not met, so as to perform automated testing on the network function virtualization environment.

2. The automated testing method according to claim 1, characterized in that: The determining of the current test case from the multiple test cases includes: According to the initial weights of the multiple test cases, a test case is randomly selected from the multiple test cases as the current test case.

3. The automated testing method according to claim 1, characterized in that: Determining the execution value of remaining test cases in the multiple test cases except the current test case according to the test result includes: If the test result is a first type result, determining the execution value of each remaining test case according to the label of each remaining test case and the similarity between each remaining test case and the executed test case; the executed test case includes at least the current test case; If the test result is a second type result, the initial value of the remaining test cases is determined as the execution value.

4. The automated testing method according to claim 3, characterized in that: Determining the value of each remaining test case according to the label of each remaining test case and the similarity between each remaining test case and the executed test case includes: Determine a first value coefficient of each remaining test case according to a label of each remaining test case and a label of the current test case; Determining a second value coefficient of each remaining test case according to the similarity between each remaining test case and the executed test case; The value of the remaining test cases for discovering new problems is determined based on the first value coefficient and the second value coefficient.

5. The automated testing method according to claim 4, characterized in that: The step of determining the first value coefficient of each remaining test case according to the label of each remaining test case and the label of the current test case includes: Determine a first value coefficient of each of the remaining test cases according to the number of labels of the current test case, the number of test cases whose current test case has a target label and whose test results are the first type of results, and the label ratio of the target test case; The target test case is a test case having a target tag and a test result of the first type of result, and the tag ratio is the ratio of the target tag to the number of tags of the target test case.

6. The automated testing method according to claim 4, characterized in that: Determining the second value coefficient of each remaining test case according to the similarity between each remaining test case and the executed test case includes: Based on the number of test cases executed before the current test case and whose historical test results are all first type results, determine the second value coefficient according to the similarity between the remaining test cases and the executed test cases; The similarity is determined based on historical test results of the remaining test cases and the executed test cases.

7. The automated testing method according to claim 4, characterized in that: The determining the execution value according to the first value coefficient and the second value coefficient includes: determining the execution value according to the product of the first value coefficient and the second value coefficient; If the execution value is greater than or equal to the value threshold, keep the execution value unchanged; If the execution value is less than the value threshold, the execution value is configured as a preset parameter.

8. The automated testing method according to claim 1, characterized in that: The step of determining a next test case from the remaining test cases based on the execution value, and performing iterative testing according to the next test case until the value condition is not met, includes: If the execution values ​​of multiple remaining test cases meet the value condition, the execution value of each test case is used as a weight, the next test case is determined from the remaining test cases, and the test is performed according to the next test case to update the test result; Based on the updated test results, determine the execution value of the remaining test cases except the next test case; Test cases are continuously selected for iterative testing according to the execution value until the execution values ​​of the remaining test cases do not satisfy the value condition.

9. An automated testing device, characterized in that: include: A test case scope determination module, used to determine a plurality of test cases according to a test scope of a network function virtualization environment; A current test case selection module, used to determine a current test case from multiple test cases, and perform a test based on the current test case to obtain a test result; An execution value determination module, used to determine the execution value of the remaining test cases in the multiple test cases except the current test case according to the test result; The iterative testing module is used to determine the next test case from the remaining test cases based on the execution value, and perform iterative testing according to the next test case until the value condition is not met, so as to perform automated testing on the network function virtualization environment.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the automated testing method according to any one of claims 1 to 8 is implemented.

11. An electronic device, characterized in that: include: processor; as well as A memory, configured to store executable instructions of the processor; The processor is configured to perform the automated testing method according to any one of claims 1 to 8 by executing the executable instructions.