Test method and device, computing equipment and computer readable storage medium

Abstract

The invention relates to a testing method, computing equipment and a computer readable storage medium in the technical field of automatic testing. The method comprises the following steps: representing competitive resources of a test case when the test case is used for testing by utilizing a competitive resource indication in annotation information corresponding to the test case, determining a target test case during testing, determining the competitive resources according to the competitive resource indication in the annotation information of the target test case, and determining a set of competitive resources of all test cases which are currently in a running state, and then determining the relationship between the competitive resource of the target test case and the competitive resource set, if the competitive resource of the target test case is not included in the competitive resource set at present, indicating that the target test case can occupy the competitive resource exclusively after the target test case is operated, so that the condition of test failure caused by resource competition does not occur, and the test efficiency is improved. Therefore, the target test case can run, the test efficiency can be effectively improved, and the test period can be shortened.

Description

Technical Field

[0001] The present application relates to the field of testing technology, and in particular to a testing method, apparatus, computing device and computer-readable storage medium. Background Art

[0002] In the field of automated testing, test cases are the basic unit for verifying the correctness of software functions. As the complexity of software systems increases, the number of test cases used in testing increases dramatically, which naturally puts higher demands on automated testing frameworks.

[0003] In related technologies, in order to avoid interference between test cases, only one test case is executed on a single test machine at a time. Although this can reduce test case failures caused by factors such as environment settings, data pollution or resource competition, it also greatly reduces test efficiency, especially in large-scale automated testing scenarios, resulting in low resource utilization and long test cycles. Summary of the invention

[0004] In order to improve test efficiency and shorten test cycle, the embodiments of the present application provide a test method, apparatus, computing device and computer-readable storage medium.

[0005] In a first aspect, an embodiment of the present application provides a testing method, which includes: determining a target test case, which is a test case to be tested; determining the competitive resources of the target test case based on the annotation information of the target test case, which is a resource that the test case requires exclusive use of during testing; if the competitive resources of the target test case are not included in a competitive resource set, running the target test case for testing, the competitive resource set is a collection of competitive resources of all test cases that are currently in operation.

[0006] By adopting the above technical solution, the test case is marked in advance, the marking information is set for the test case, and the competitive resource indication in the marking information is used to characterize the competitive resource of the test case when it is used for testing, that is, the resource required to be exclusively occupied by the test case when it is running. In the process of running at least one other test case, if the target test case is to be further run, the competitive resource of the target test case is first determined according to the competitive resource indication in the marking information of the target test case, and the set of competitive resources of all test cases currently in operation, that is, the competitive resource set, is determined, and then the relationship between the competitive resource of the target test case and the competitive resource set is determined. If the competitive resource of the target test case is not currently included in the competitive resource set, it means that after running the target test case, the target test case can monopolize the competitive resource, will not cause competition conflicts with other running test cases, and will not fail due to resource competition. Therefore, the target test case can be executed in parallel with other running test cases, so that stable and reliable parallel testing can be achieved, and the test efficiency can be effectively improved and the test cycle can be shortened without increasing the test resources and ensuring the test success rate.

[0007] Optionally, the competitive resource table includes competitive resources required by each test case that is currently in running state. Before running the target test case for testing, it also includes: obtaining the competitive resource table; determining the competitive resource set according to the competitive resource table; after running the target test case for testing, it also includes: adding the competitive resources of the target test case to the competitive resource table; after the target test case is finished running, it also includes: deleting the competitive resources of the target test case from the competitive resource table.

[0008] By adopting the above technical solution, the competitive resources of each test case that is currently in operation are managed using the competitive resource table: when a test case starts to run, its competitive resources are added to the competitive resource table; when the test case ends, its competitive resources are deleted from the competitive resource table. In this way, at any time, you only need to query the competitive resource table to determine the current competitive resource set based on the current records in the competitive resource table. This facilitates management, speeds up the determination of competitive resource sets, and improves test efficiency.

[0009] Optionally, recording the target test case's contention resources in the contention resource table may be performed by recording the target test case's contention resource indication in the contention resource table, and deleting the target test case's contention resources from the contention resource table may be performed by deleting the target test case's contention resource indication from the contention resource table.

[0010] Optionally, the competitive resources of the target test case include at least one of the following: processor resources, storage resources, network communication resources, service resources, file resources, and an opportunity to execute a preset operation.

[0011] Optionally, the testing method also includes: determining the full set of competitive resources based on the competitive resources corresponding to each test case, the full set of competitive resources being the collection of competitive resources of all test cases; before determining the target test case, it also includes: comparing the current set of competitive resources with the full set of competitive resources; and determining that the competitive resource set is not equal to the full set of competitive resources.

[0012] By adopting the above technical solution, before determining the target test case, the relationship between the current competitive resource set and the complete set of competitive resources is first determined. If the competitive resource set is equal to the complete set of competitive resources, it means that the competitive resources of all test cases are currently in an occupied state. Therefore, if any test case is further run in parallel, resource competition will occur, that is, no test case is suitable for testing at this time; if the current competitive resource set is not equal to the complete set of competitive resources, it means that there is at least one competitive resource that has not been occupied. Therefore, there is still an opportunity to run other test cases in parallel, so the target test case can be determined and whether the target test case can be run. Through this solution, it is possible to directly identify the situation where all competitive resources are in an occupied state, prevent the computing device from continuously selecting the target test case and judging whether the target test case can be run when all competitive resources are in an occupied state, prevent "useless work", and avoid wasting computing device processing resources.

[0013] Optionally, the annotation information also includes a parallel test indication, which is used to indicate whether the test case supports parallel execution with other test cases; the parallel test indication includes one of a support indication and a prohibition indication.

[0014] By adopting the above technical solution, the support of the corresponding test case for parallel testing is directly characterized by the parallel test indication in the annotation information. The computing device can screen out the test cases that support parallel testing and those that do not support parallel testing based on the annotation information of the test case. In this way, the computing device can run these two types of test cases in different ways, avoiding the situation where test cases that are not suitable for parallel testing are used to run in parallel with other test cases, thereby improving the test efficiency and reducing the probability of test failure.

[0015] Optionally, before determining the competing resources of the target test case according to the annotation information of the target test case, the testing method further includes: determining that the annotation information of the target test case includes a support indication.

[0016] Optionally, before determining the target test case, it also includes: classifying all test cases according to the parallel testing indication of the test case, and forming a parallel test case set that supports parallel testing and a separate test case set that does not support parallel testing; determining the target test case includes: selecting the current target test case from the parallel test case set; after running the target test case for testing, it also includes: deleting the target test case from the parallel test case set.

[0017] Optionally, the testing method further includes: if the contention resource of the target test case is included in the contention resource set, reselecting a test case from the parallel test case set as the target test case.

[0018] In a second aspect, an embodiment of the present application provides a testing device, including:

[0019] A test case determination module is used to determine a target test case, where the target test case is a test case to be tested;

[0020] A resource determination module, used to determine the competitive resources of the target test case according to the annotation information of the target test case, where the competitive resources are the resources that the test case requires exclusive use during the test;

[0021] The test running module is used to run the target test case for testing when the contention resources of the target test case are not included in the contention resource set, where the contention resource set is a collection of contention resources of all test cases currently in operation.

[0022] In a third aspect, an embodiment of the present application provides a computing device, including a processor, a memory, and a communication bus, wherein the communication bus is used to realize a communication connection between the processor and the memory, and the processor is used to execute a computer program stored in the memory to implement any one of the test methods of the first aspect.

[0023] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium, which stores a computer program; when the computer program is executed by a processor, it can implement any test method in the aforementioned first aspect.

[0024] In a fifth aspect, an embodiment of the present application further provides a computer program product, which includes a computer program / instructions, and when the computer program / instructions are executed by a processor, the test method of any one of the aforementioned first aspects can be implemented.

[0025] In summary, the embodiments of the present application include at least the following beneficial technical effects:

[0026] By controlling the test cases running in parallel, the test efficiency is improved and the test cycle is shortened while ensuring the test success rate. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 A schematic diagram of a flow chart of the test method provided in the embodiments of the present application;

[0028] Figure 2 Another schematic diagram of the test method provided in the embodiments of the present application;

[0029] Figure 3 A schematic diagram of a management queue for managing test cases that have not yet participated in testing in a computing device shown in an embodiment of the present application;

[0030] Figure 4 A schematic diagram of the time consumption of executing three test cases under a test case management scheme;

[0031] Figure 5 A schematic diagram of the time consumption of executing three test cases under another test case management scheme;

[0032] Figure 6 It is another flow chart of the testing method provided in the embodiments of the present application;

[0033] Figure 7 A schematic diagram of a hardware structure of a computing device provided in an embodiment of the present application;

[0034] Figure 8 A schematic diagram of the structure of the testing device provided in an embodiment of the present application.

[0035] Description of reference numerals:

[0036] 70 - computing device; 710 - processor; 720 - memory; 730 - communication bus; 80 - test device; 802 - use case determination module; 804 - resource determination module; 806 - test execution module. DETAILED DESCRIPTION

[0037] The terms used in the following embodiments of the present application are only for the purpose of describing specific embodiments, and are not intended to be used as limitations to the present application. As used in the specification and appended claims of the present application, the singular expressions "one", "a kind of", "said", "above", "the" and "this" are intended to also include plural expressions, unless there is a clear indication to the contrary in the context. It should also be understood that the term "and / or" used in the present application refers to and includes any or all possible combinations of one or more listed items.

[0038] The terms "first" and "second" are used for descriptive purposes only and should not be understood to imply or suggest relative importance or implicitly indicate the number of technical features indicated. Thus, features defined as "first" and "second" may explicitly or implicitly include one or more of the features. In the description of the embodiments of the present application, unless otherwise specified, "plurality" means two or more.

[0039] The suffixes such as "module", "component" or "unit" used to represent elements are only for the purpose of facilitating the description of the embodiments of the present application, and have no specific meanings by themselves. Therefore, "module", "component" or "unit" can be used in a mixed manner.

[0040] In order to enable those skilled in the art to clearly understand the solutions of the embodiments of the present application, some terminology concepts involved in the embodiments of the present application are explained below.

[0041] Parallel testing: Execute at least two test cases in parallel on the same device at the same time. It should be noted that parallel testing does not require the running time of the two test cases to completely overlap. As long as the running time of the two test cases overlaps at least partially, it is considered parallel execution of test cases.

[0042] Test case: refers to the description of the test task for a specific software product, reflecting the test plan, method, technology and strategy. Its content includes test objectives, test environment, input data, test steps, expected results, test scripts, etc., and finally forms a document. Simply put, a test case is a set of test inputs, execution conditions and expected results compiled for a special goal, used to verify whether a specific software requirement is met. Automated test cases can trigger the automated execution of test cases through some specific instructions / scripts.

[0043] Competing resources: If more than one test case may fail due to the use of a common resource when executed in parallel, then this resource is called a competing resource. This resource can be a specific physical resource, a service, a file or other resource (such as the opportunity to perform preset operations, etc.). Competing resources of the physical resource class include processor resources (such as CPU, GPU, etc.), storage resources (such as memory, disk, etc.), network communication resources (such as network card, etc.); competing resources of the service class include selinux (such as Security-Enhanced Linux, Security-Enhanced Linux, etc.) service, firewall service, kdump (a tool and service used to dump memory operating parameters when the system crashes, deadlocks or freezes) service, httpd (HyperText Transfer Protocol Daemon, Hypertext Transfer Protocol) service, etc.; competing resources of the opportunity to perform preset operations include reboot operations, configuration modification (conf) operations, etc. It is understandable that competing resources are actually resources that are required to be exclusive when test cases are running, so in some cases competing resources can also be referred to as "exclusive resources" or "exclusive resources".

[0044] This embodiment provides a test method, see Figure 1 The flow chart of the test method is shown as follows:

[0045] S102: Determine target test cases.

[0046] When testing, the computing device first determines the target test case, which is the test case that is currently to be used for testing, that is, the test case that is currently to be used for testing. Usually, a computing device only needs to execute one round of test cases, that is, on a computing device, each test case is usually executed once, so when the computing device determines the current target test case, it generally selects from the test cases that are not currently involved in the test.

[0047] S104: Determine the competing resources of the target test case according to the annotation information of the target test case.

[0048] After determining the target test case, the computing device may obtain the annotation information of the target test case, and by parsing the annotation information, the computing device may determine the competing resources of the target test case.

[0049] In this embodiment, the test case will be annotated in advance. The information of the annotated test case is referred to as "annotation information" in this embodiment. The annotation information includes a competitive resource indication. The competitive resource indication of a test case is used to indicate the competitive resource of the test case. For example, if the competitive resource indication of the test case test1 is "cpu", it indicates that the competitive resource of the test case is the CPU processing resource. If the competitive resource of another test case test2 also includes the CPU processing resource, for example, its competitive resource indication is "cpu, memory", then the parallel execution of test1 and test2 may cause the two test cases to compete for the CPU processing resources, resulting in the failure of test1 and / or test2.

[0050] In addition to the above examples, "cpu" is used to characterize the competing resource as a CPU processing resource, and "memory" is used to characterize the competing resource as a memory storage resource. In this embodiment, "selinux" is used to characterize the competing resource as a selinux service, "network" is used to characterize the competing resource as a communication network resource, "reboot" is used to characterize the competing resource as a restart operation, "httpd" is used to characterize the competing resource as an httpd service, and "pam_conf" is used to characterize the competing resource as modifying the configuration under / etc / pam.d / . However, it can be understood by those skilled in the art that in some other examples of this embodiment, the competing resource can also be characterized by other types of identification information, for example, different IDs are set for different competing resources: for example, the ID of the CPU processing resource is "1", the ID of the storage resource is "2", the ID of the restart operation is "3", and the ID of the httpd service is "4"... If the competing resource indications of the two test cases are "1" and "3" respectively, then it means that the competing resources of the two test cases are CPU processing resources and restart operations respectively. The specific form of the competing resource indication, that is, how to identify the competing resource specifically, is not limited in this embodiment, as long as the corresponding identification information can uniquely distinguish each resource.

[0051] In addition, those skilled in the art will appreciate that some test cases may have more than one competing resource. For example, in one example, the competing resource indication for test2 is "1, 3", which means that the competing resources for the test case test2 include both CPU processing resources and restart operations.

[0052] S106: If the contention resource of the target test case is not included in the contention resource set, the target test case is run for testing.

[0053] According to the above description of competitive resources, those skilled in the art should understand that the premise of ensuring the successful operation of a test case is to ensure that during the operation of the test case, no other test cases will compete with the test case for its competitive resources. Therefore, before the computing device officially puts the target test case into operation, it is also necessary to determine whether the target test case will compete for resources with other currently running test cases if it starts running, and whether the currently running test case will fail to run due to the addition of the target test case. In this embodiment, the test case currently running is referred to as a "running state test case". It can be understood that the number of running state test cases can be one or more. Based on the competitive resource indications of these running state test cases, the computing device can determine the set of competitive resources of these running state test cases. For the sake of ease of introduction, this set is referred to as a "competitive resource set" in this embodiment.

[0054] If the contention resource set does not contain the contention resource of the target test case, it means that there is no conflict of contention resources between the running test case and the target test case, so the target test case can be run and executed in parallel with the current running test case. After the target test case starts running, it becomes a member of the running test case.

[0055] If the competitive resources of the target test case are included in the competitive resource set, it means that there are test cases in the running test case that require the exclusive use of the same resources as the target test case. In order to ensure the successful operation of the running test case, it is not appropriate to put the target test case into operation. In this case, the computing device can re-determine a new target test case from other test cases that have not yet participated in the test, and then execute the above process for the new target test case to determine whether to put the new target test case into operation. Without considering the performance, the computing device can continue to execute the above process until all test cases are completed. However, in some examples of this embodiment, the number of test cases executed in parallel can also be controlled. For example, in an example, the number of test cases required to be executed in parallel does not exceed 5, that is, the maximum number of test cases set is 5. Then, before running the new target test case, the computing device can first determine the number of current running test cases. If the number of running test cases has reached 5, the new target test case will not be run temporarily until the number of running test cases is less than 5. Or before selecting a new target test case, first determine whether the number of currently running test cases has reached 5. If so, do not select a new target test case first.

[0056] Those skilled in the art will appreciate that there are no strict timing restrictions between the two processes of the computing device determining the set of competing resources and determining the competing resources for the target test case. The computing device may first determine the set of competing resources, or may first determine the competing resources for the target test case, or in some examples, these two processes may also be performed synchronously.

[0057] In some examples of this embodiment, the computing device can record and manage the competitive resource set of the running test case through the competitive resource table: after running the target test case, the computing device records the competitive resources (such as identification information such as competitive resource indication) of the target test case in the competitive resource table. When the target test case is finished running, the computing device deletes its competitive resources (such as identification information such as competitive resource indication) from the competitive resource table. In this way, at any time, the computing device can understand which competitive resources are included in the current competitive resource set as long as it obtains the competitive resource table.

[0058] Assume that the contention resource indication of test case test1 is "cpu", the contention resource indication of test case test2 is "memory, httpd", and the contention resource indication of test case test3 is "reboot". The computing device selects target test cases in the order of test1, test2, and test3. These three test cases are put into operation at time t1, time t2, and time t3 respectively, and the time when the three end their operation is time t4, time t5, and time t6 respectively. Then:

[0059] At time t1, the contention resource indication recorded in the contention resource table is: cpu;

[0060] At time t2, the competing resource indications recorded in the competing resource table are: cpu, memory, httpd;

[0061] At time t3, the competing resource indications recorded in the competing resource table are: cpu, memory, httpd, reboot;

[0062] At time t4, the competing resource indications recorded in the competing resource table are: memory, httpd, reboot;

[0063] At time t5, the contention resource indication recorded in the contention resource table is: reboot;

[0064] At time t6, the contention resource indication recorded in the contention resource table is empty.

[0065] It should be understood that if all resources are already occupied at a certain moment, then no matter which test case is selected as the target test case at this time, the result obtained after evaluation is the same, that is, the competitive resources of the target test case have been included in the competitive resource set, and the target test case is not currently suitable for running. In this case, no matter how many target test cases the computing device reselects, it is doing "useless work", and it will instead cause a waste of processing resources due to the selection of target test cases and the evaluation of target test cases. In order to avoid this situation, in some examples of the present embodiment, the computing device can determine whether to select and evaluate the target test case at present based on the full set of competitive resources. The full set of competitive resources refers to the set of competitive resources of all test cases. The concept of the full set of competitive resources is explained below in conjunction with Table 1, and in conjunction with Figure 2 This test method is explained:

[0066] Table 1

[0067] Test case identification Competitive Resource Identification test1 cpu test2 cpu,memory,network test3 pam_conf test4 httpd,memory test5 reboot test6 cpu

[0068] Assuming that the six test cases test1 to test6 in Table 1 are all the test cases involved in a test task, then the full set of competing resources is "cpu, memory, network, pam_conf, httpd, reboot".

[0069] S202: The computing device determines a complete set of competing resources.

[0070] Since the test cases involved in a test task are usually predetermined and will not change arbitrarily during the test process, in a test, the computing device can determine the full set of contention resources in advance according to the contention resource indication in the annotation information of each test case.

[0071] S204: The computing device determines whether there is a test case that has not yet been tested.

[0072] If the judgment result is yes, then execute S206, otherwise the process ends.

[0073] S206: The computing device determines a current set of competing resources.

[0074] In some examples, the computing device determines the contention resource set according to the contention resource table. When the test case has not been run, both the contention resource table and the contention resource set are empty.

[0075] S208: The computing device determines whether the contention resource set is equal to the entire contention resource set.

[0076] If the judgment result is yes, execute S210, otherwise execute S212.

[0077] S210: The computer device waits for a preset time period.

[0078] If the competitive resource set is equal to the complete set of competitive resources, it means that all the competitive resources of all current test cases are in an occupied state (it must be explained that the occupied state mentioned here does not mean that the resources are currently being used, but means that these resources are needed during the operation of the current running test cases. These resources may have been occupied or "reserved"). Therefore, the computing device does not need to select the target test case and determine whether the competitive resources of the target test case are included in the competitive resource set, but can wait for a period of time, such as waiting for a preset time, and then re-execute S206. The value of the preset time can be determined by the engineering personnel based on the experience value device. In one example, the maximum value of the preset time does not exceed the maximum time required for the test case to run. In other examples, the value of the preset time can be set according to the average running time of each test case.

[0079] In some other examples, the computing device may also monitor the contention resource table when the contention resource set is equal to the entire contention resource set, and then execute S206 after detecting that the contention resource table is modified.

[0080] S212: The computing device determines a target test case from test cases that have not yet participated in the test.

[0081] S214: The computing device determines the competing resources of the target test case according to the annotation information of the target test case.

[0082] S216: The computing device determines whether the contention resource of the target test case is included in the current contention resource set.

[0083] If the judgment result is yes, execute S212, otherwise execute S218.

[0084] S218: The computing device runs the target test case.

[0085] It is understandable that in some examples, when the computing device determines that the target test case's contention resources are included in the contention resource set, it can directly select a new target test case from other test cases that have not yet participated in the test, or wait for a period of time and then re-determine whether the contention resources of the target test case are "released" based on the contention resource set. In comparison, the former solution (i.e., the solution of re-determining the target test case) generally has higher testing efficiency and a shorter overall testing cycle.

[0086] While executing S218, the computing device records the identification information of the competitive resources of the target test case into the competitive resource table. The computing device will subsequently monitor the moment when the test case ends. When the test case ends, the computing device will delete the identification information of the competitive resources from the competitive resource table.

[0087] After executing S218, the computing device may directly continue to execute S204, such as Figure 2 As shown, the computing device may also decide whether to execute S204 immediately according to the maximum number of test cases, or to wait until the number of running test cases is less than the maximum number of test cases before executing S204.

[0088] It is understandable that there are some test cases that are not suitable for parallel execution with any other test cases. For example, if the competitive resources of a test case include all resources, then when this test case is running, no other test case is suitable for running, and when any other test case is running, this test case is also not suitable for running. In some examples, such a test case can be prevented from running in parallel with other test cases by setting a competitive resource indication containing all competitive resource identification information for the test case.

[0089] In some other examples of this embodiment, it is also possible to directly indicate that the test case does not support parallel testing through the parallel test indication in the annotation information. Therefore, in these embodiments, the annotation information includes both parallel test indications and contention resource indications, wherein the parallel test indications are divided into support indications and prohibition indications, the support indications are used to characterize that the test case supports parallel testing, and the prohibition indications characterize that the test case does not support parallel testing. In some examples, "1" is used as a support indication and "0" is used as a prohibition indication. It is undoubted that in some other examples, prohibition indications and support indications can also be characterized by other methods. For test cases (such as performance test cases, long-term stable test cases, etc.) containing prohibition indications in the annotation information, their contention resource indications can be set to empty.

[0090] In some examples, the annotation information includes a unique identifier of the test case (such as the name of the test case or the test case number) in addition to the parallel test indication and the contention resource indication. Table 2 shows some annotation information and its corresponding meaning description, which is convenient for those skilled in the art to understand the annotation information in combination:

[0091] Table 2

[0092]

[0093] Compared to the solution of enumerating all contention resource identification information in the contention resource indication of a test case to prevent the test case from being executed in parallel with other test cases, in this embodiment, a parallel test indication is set in the annotation information, so that the computing device can directly and simply determine that the test case cannot be tested in parallel based on the prohibition indication, without having to compare each contention resource identification of the test case with the current contention resource set one by one. This can simplify the computing device's recognition of test cases that do not support parallel testing, improve testing efficiency, and reduce the occupancy and consumption of computing device processing resources.

[0094] To determine whether the test case is in Table 2, among the three fields in the annotation information, the first field is the unique identifier of the test case, the second field is the parallel test indication, and the third field is the competitive resource indication. However, in the annotation information of some other examples of this embodiment, the arrangement order of the unique identifier of the test case, the parallel test indication, and the competitive resource indication may be other orders. In other examples, the annotation information may also include other contents.

[0095] In some examples of this embodiment, for test cases that have not yet participated in the test, the computing device may manage them uniformly without distinguishing whether the test cases support parallel testing, for example, using the same stack or the same queue to manage the test cases that have not yet participated in the test, and not restricting the arrangement of different types of test cases in the stack or queue. In this scheme, since the test cases that support parallel testing and the test cases that do not support parallel testing are randomly arranged, the order of the target test cases determined by the computing device is very likely to be a random intersperse of the test cases that support parallel testing and the test cases that do not support parallel testing, for example, see Figures 3 to 5 As shown, the computing device uses a queue to manage all test cases that have not yet participated in the test. In this queue, both test11 and test12 support parallel testing, and there is no competition for the same resources between the two. Therefore, the two can be executed in parallel. However, test13, which does not support parallel testing, is arranged between test11 and test12. Then, in the actual test process, after the computing device runs test11, test13 is extracted from the queue. Because test13 does not support parallel testing, test13 can only be run after test11 and the test cases that were previously put into operation have finished running. At the same time, during the process of running test13, test12 cannot be run synchronously. Test12 must also wait until test13 is finished running before it can run. Compared with the test process in which test11 and test12 are executed in parallel and test13 is executed separately, this will significantly limit the improvement of test efficiency. Assuming that the time required to execute the three test cases test11, test12 and test13 separately is t11, t12 and t13 respectively, then according to Figure 3 The management scheme shown in the figure, the time required to complete these three test cases is t11+t12+t13, as shown in Figure 4 As shown; however, if test13, which does not support parallel testing, is not interspersed between test11 and test12 (for example, test11 and test12 are executed first, and then test13 is executed), the time to complete these three test cases must be less than t11+t12+t13, as shown in Figure 5 shown.

[0096] In this regard, in some examples of the present embodiment, the computing device may pre-divide all test cases into two sets, a parallel test case set and a separate test case set, according to the parallel test indications of the test cases, wherein the parallel test indications of all test cases in the parallel test case set are support indications, all supporting parallel testing, and the parallel test indications of all test cases in the separate test case set are prohibition indications, none supporting parallel testing. In these examples, a test task will be divided into two phases, namely, a phase for testing using test cases in the parallel test case set (for ease of introduction, referred to herein as the "parallel test phase"), and a phase for testing using test cases in the separate test case set (referred to herein as the "separate test phase"), and either phase may be executed first.

[0097] In the parallel testing phase, the computing device determines the target test case from the parallel test case set and performs the test according to the test method provided in any of the aforementioned examples. After the target test case is put into operation, the target test case can be deleted from the corresponding management queue or stack to ensure that only the test cases that are not currently involved in the test are retained in the management queue or stack to avoid the computing device from repeatedly running the same test case. In the individual testing phase, the computing device can run each test case in the individual test case set individually for testing. According to this scheme, although each test case in the individual test case set can still only be run individually, two adjacent test cases in the parallel test case set can be run in parallel as long as there are no identical competing resources. The parallel testing process will not be interrupted due to the insertion of test cases that do not support parallel testing. It can effectively improve the testing efficiency and shorten the testing cycle without increasing the testing resources.

[0098] This embodiment also does not exclude the scheme of intentionally interspersing test cases that support parallel testing with test cases that do not support parallel testing. For example, in one example, the computing device divides the parallel test case set into two subsets, namely subset a and subset 2. The computing device first tests the test cases in subset a, then tests the test cases in the separate test case set, and finally tests the test cases in subset 2. Although the testing time of this test scheme is increased compared with the testing time of the aforementioned scheme that is divided into only two stages (i.e., the parallel testing stage and the separate testing stage), it is still significantly shorter and more efficient than the scheme in which all test cases that support parallel testing and test cases that do not support parallel testing are randomly arranged, and the scheme in which all test cases are run separately.

[0099] In order to make the advantages and details of the above test scheme more clear to those skilled in the art, the above test method is introduced below with a more complete example. Figure 6 :

[0100] S602: The computing device classifies the test cases to form a parallel test case set and a separate test case set.

[0101] In this embodiment, the computing device can obtain the annotation information of each test case respectively, and then divide the corresponding test case into a parallel test case set or a separate test case set according to the parallel test indication in the annotation information. In this example, the computing device uses ConcurrentList to manage the test cases that are not currently participating in the test in the parallel test case set, and uses SequentialList to manage the test cases that are not currently participating in the test in the separate test case set.

[0102] S604: The computing device determines a complete set of competing resources based on the parallel test case set.

[0103] In this example, ResourceAll is used to represent the entire set of competing resources. According to the competing resource indications of each test case in the parallel test case set, the computing device can determine ResourceAll.

[0104] S606: The computing device determines whether ConcurrentList is an empty set.

[0105] If the judgment result is no, The computing device executes S608; if the judgment result is yes, This indicates that all test cases supporting parallel testing have completed testing, so the computing device executes S618.

[0106] S608: The computing device determines whether the current set of contention resources is equal to the entire set of contention resources.

[0107] Here, ResouceRunning is used to represent the competing resource set. Therefore, the computing device actually determines whether ResouceRunning is equal to ResourceAll. If ResouceRunning == ResourceAll, S610 is executed; if ResouceRunning≠ResourceAll, the computing device executes S612.

[0108] S610: The computing device waits for a preset time period.

[0109] After the computing device waits for a preset period of time, S608 is executed again.

[0110] S612: The computing device determines a target test case from the ConcurrentList and determines competing resources of the target test case.

[0111] In some examples, the computing device may randomly select a test case from the ConcurrentList as a target test case. In other examples, the computing device may pre-set a test order for each test case in the ConcurrentList, and during the test process, extract the corresponding test cases as target test cases in the established order.

[0112] After the target test case is determined, the competing resource indication is extracted from the annotation information of the target test case, and the computing device can determine what the competing resource of the target test case is.

[0113] S614: The computing device determines whether the current contention resource set includes contention resources of the target test case.

[0114] If the judgment result is no, the computing device executes S616, and if the judgment result is yes, the computing device executes S612 again. If the test cases in the ConcurrentList are pre-set with a test order, the computing device may adjust the test order of the current target test case (e.g., test case a) to the end before re-determining the target test case, or insert the test case a to any position in the middle of the test order queue to ensure that the test case a is not immediately selected when the target test case is re-selected.

[0115] S616: The computing device runs the target test case and deletes the target test case from the ConcurrentList.

[0116] If the computing device determines that ResouceRunning does not contain the competing resources of the target test case, it means that the target test case can be executed at present, so the computing device runs the target test case. After the target test case is run, it becomes a running test case, and it can be deleted from ConcurrentList at this time. In order to facilitate the computing device to determine ResouceRunning, after running the target test case, the computing device will also add the identification information of its competing resources to ResouceRunning. When a running test case ends, the computing device also needs to delete its competing resources from ResouceRunning.

[0117] After executing S616 , the computing device continues to execute S606 .

[0118] S618: The computing device determines whether SequentialList is an empty set.

[0119] If the judgment result is no, The computing device executes S620; if the judgment result is yes, This means that all test cases that do not support parallel testing have been tested, so the process ends.

[0120] S620: The computing device determines a target test case from the SequentialList.

[0121] S622: The computing device runs the target test case.

[0122] After the target test case is run, the target test case becomes a running test case, so the computing device deletes the target test case from the SequentialList.

[0123] S624: The computing device determines whether the current target test case has been tested.

[0124] Since none of the test cases in SequentialList support parallel testing, the computing device must wait for a test case to be completed before running the next test case in SequentialList. Therefore, before selecting the next target test case, the computing device needs to first determine whether the current target test case has been tested. If the determination result is yes, the computing device executes S618, otherwise it continues to execute S624.

[0125] It can be understood by those skilled in the art that Figure 6The timing of certain processes in the test method shown can be adjusted. For example, during the test process, the test cases in the SequentialList can be tested first, and then the target test cases can be determined from the ConcurrentList after the SequentialList is an empty set. For another example, the computing device can first determine whether the current set of competing resources is equal to the full set of competing resources, and then determine whether the ConcurrentList is an empty set if the judgment result is no.

[0126] This embodiment also provides a computing device 70, which can be implemented in various forms. For example, the computing device 70 described in the embodiment of the present application can include mobile terminals such as mobile phones, tablet computers, laptop computers, PDAs, portable media players (PMPs), navigation devices, wearable devices, smart bracelets, pedometers, etc., as well as fixed terminals such as digital TVs, desktop computers, and large computers.

[0127] See also Figure 7 As shown, the computing device 70 includes a processor 710 , a memory 720 , and a communication bus 730 , and the processor 710 and the memory 720 are communicatively connected via the communication bus 730 .

[0128] The memory 720 may be used to store instructions, programs, codes, code sets or instruction sets. The memory 720 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for at least one function, and instructions for implementing the test method provided in the above embodiment, etc.; the data storage area may store data (such as a contention resource table) involved in the test method provided in the above embodiment, etc.

[0129] The processor 710 may include one or more processing cores. The processor 710 calls the data stored in the memory 720 by running or executing the instructions, programs, code sets or instruction sets stored in the memory 720, and executes various functions and processes data of the embodiments of the present application. The processor 710 may be at least one of an application-specific integrated circuit (ASIC), a digital signal processor (DSP), a digital signal processing device (DSPD), a programmable logic device (PLD), a field programmable gate array (FPGA), a central processing unit (CPU), a controller, a microcontroller and a microprocessor. It is understandable that for different devices, the electronic device used to implement the functions of the processor 710 may also be other, and the embodiments of the present application are not specifically limited.

[0130] It is obvious that, in addition to the processor 710, the memory 720 and the communication bus 730, the computing device 70 may also include other components, such as an input unit (such as at least one of a keyboard, a touch panel, and an audio input unit) and an output unit (such as at least one of a display screen and a speaker).

[0131] This embodiment also provides a testing device 80, see Figure 8 As shown, the testing device 80 includes a use case determination module 802, a resource determination module 804 and a test running module 806, wherein the use case determination module 802 is used to determine the target test case; the competing resource determination module 804 is used to determine the competing resources of the target test case according to the annotation information of the target test case; the target use case running module 806 is used to run the target test case for testing when the competing resources of the target test case are not included in the competing resource set.

[0132] The various variations and specific examples of the methods provided in the above embodiments are also applicable to the testing device 80 of this embodiment. Through the above detailed description of the testing method, those skilled in the art can clearly understand the implementation scheme of the testing device 80 in this embodiment. For the sake of brevity of the specification, they will not be described in detail here.

[0133] The embodiment of the present application also provides a computer-readable storage medium, for example, including: a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc. Various media that can store program codes. The computer-readable storage medium stores a computer program that can be read, compiled and executed by the processor 710 to implement the test method in any of the aforementioned examples.

[0134] This embodiment also provides a computer program product, which includes a computer program / instructions. When the computer program / instructions are executed by a processor, the testing method in any of the aforementioned examples can be implemented.

[0135] The above embodiments are only used to introduce the technical solution of the present application in detail, but the description of the above embodiments is only used to help understand the method and core idea of ​​the present application and should not be understood as limiting the present application. Any changes or substitutions that can be easily thought of by technicians in this technical field within the technical scope disclosed in the present application should be included in the protection scope of the present application.

Claims

1. A testing method, characterized in that: The test method includes: Determine a target test case, wherein the target test case is a test case to be tested; Determine the contention resource of the target test case according to the annotation information of the target test case, wherein the contention resource is a resource that the test case requires to be exclusively occupied during the test; If the contention resource of the target test case is not included in a contention resource set, the target test case is run for testing, and the contention resource set is a collection of contention resources of all test cases currently in operation.

2. The testing method according to claim 1, characterized in that: The competitive resource table includes the competitive resources required by each of the test cases currently in operation. Before the target test case is run for testing, the following steps are also included: Obtaining the competition resource table; Determine the contention resource set according to the contention resource table; After the target test case is run for testing, the method further includes: adding the competitive resources of the target test case to the competitive resource table; After the target test case is finished running, the method further includes: deleting the competing resources of the target test case from the competing resource table.

3. The testing method according to claim 1 or 2, characterized in that: The competing resources of the target test case include at least one of the following: processor resources, storage resources, network communication resources, service resources, file resources, and an opportunity to execute a preset operation.

4. The testing method according to any one of claims 1 to 3, characterized in that: The test method also includes: Determine a complete set of competitive resources, where the complete set of competitive resources is a collection of the competitive resources of all the test cases; Before determining the target test case, the method further includes: comparing the set of contention resources with the entire set of contention resources; It is determined that the set of contended resources is not equal to the entire set of contended resources.

5. The testing method according to any one of claims 1 to 4, characterized in that: The annotation information also includes a parallel test indication, which is used to indicate whether the test case supports parallel execution with other test cases; the parallel test indication includes one of a support indication and a prohibition indication.

6. The testing method according to claim 5, characterized in that: Before determining the competing resources of the target test case according to the annotation information of the target test case, the testing method further includes: Determine that the annotation information of the target test case includes the support indication.

7. The testing method according to claim 5, characterized in that: Before determining the target test case, the method further includes: Classifying all the test cases according to the parallel testing indications of the test cases to form a parallel test case set supporting parallel testing and a separate test case set not supporting parallel testing; Determining the target test case includes: selecting the current target test case from the parallel test case set; After running the target test case for testing, the method further includes: deleting the target test case from the parallel test case set.

8. A testing device, characterized in that: include: A test case determination module, used to determine a target test case, wherein the target test case is a test case to be tested; A resource determination module, used to determine the competing resources of the target test case according to the annotation information of the target test case, wherein the competing resources are resources that the test case requires exclusive use during testing; The test running module is used to run the target test case for testing when the competitive resource of the target test case is not included in the competitive resource set, and the competitive resource set is a collection of competitive resources of all test cases currently in operation.

9. A computing device, characterized in that The system comprises a processor, a memory and a communication bus, wherein the communication bus is used to realize the communication connection between the processor and the memory, and the processor is used to execute the computer program stored in the memory to realize the test method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program; the computer program can be executed by a processor to implement the testing method according to any one of claims 1 to 7.

Images