Processing method and device of unstable test case, equipment and storage medium

Abstract

The disclosure provides a processing method and device of unstable test case, equipment and storage medium, relates to the technical field of computers, and in particular to the technical field of automatic testing. The processing method of unstable test case comprises the following steps: obtaining at least one group of test running data of a test case, each group of test running data in the at least one group of test running data comprising test results and retry information; determining an unstable probability value based on the number of groups of the at least one group of test running data, the test results and the retry information, the unstable probability value being used to determine whether the test case is an unstable test case; and if the unstable probability value is greater than a preset threshold, determining that the test case is an unstable test case. The disclosure can improve the processing effect of unstable test cases.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of computer, in particular to the technical field of automation testing, and more particularly to a flaky test case processing method and device, equipment and storage medium. BACKGROUND

[0002] In the field of software engineering, test cases can be used to automatically test software to obtain test results, so that the quality of the software can be detected through the test results. Generally speaking, a test result of fail indicates that the software has a bug, and a test result of pass indicates that the software has no bug.

[0003] A flaky test case refers to a test case whose test result is sometimes successful and sometimes failed when the test object and test conditions remain unchanged. Due to the existence of flaky test cases, the certainty of detecting software quality based on test results is destroyed, and the purpose of automatic testing cannot be achieved. Therefore, flaky test cases are generally considered to be harmful. SUMMARY

[0004] The present disclosure provides a flaky test case processing method, device, equipment and storage medium.

[0005] According to an aspect of the present disclosure, a flaky test case processing method is provided, comprising: obtaining at least one set of test run data of a test case, each set of test run data in the at least one set of test run data comprising: a test result and retry information; determining a flakiness probability value based on the number of sets of test run data, the test result and the retry information, the flakiness probability value being used to determine whether the test case is a flaky test case; and determining that the test case is a flaky test case if the flakiness probability value is greater than a preset threshold.

[0006] According to another aspect of the present disclosure, a flaky test case processing device is provided, comprising: an obtaining module configured to obtain at least one set of test run data of a test case, each set of test run data in the at least one set of test run data comprising: a test result and retry information; a first determining module configured to determine a flakiness probability value based on the number of sets of test run data, the test result and the retry information, the flakiness probability value being used to determine whether the test case is a flaky test case; and a second determining module configured to determine that the test case is a flaky test case if the flakiness probability value is greater than a preset threshold.

[0007] According to another aspect of the present disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected with the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of the above aspects.

[0008] According to another aspect of the present disclosure, a non-transitory computer readable storage medium storing computer instructions is provided, wherein the computer instructions are used to cause the computer to perform the method according to any one of the above aspects.

[0009] According to another aspect of the present disclosure, a computer program product is provided, comprising a computer program which, when executed by a processor, implements the method according to any one of the above aspects.

[0010] According to another aspect of the present disclosure, an automated testing platform is provided, comprising the electronic device described above.

[0011] According to the technical solution of the present disclosure, the processing effect of unstable test cases can be improved.

[0012] It should be understood that the content described in this part is not intended to identify key or important features of the embodiments of the present disclosure, nor to limit the scope of the present disclosure. Other features of the present disclosure will become apparent from the following description. BRIEF DESCRIPTION OF DRAWINGS

[0013] The accompanying drawings are used to better understand the present solution and do not limit the present disclosure. Among them:

[0014] Figure 1 is a schematic diagram according to a first embodiment of the present disclosure;

[0015] Figure 2 is a schematic diagram according to a second embodiment of the present disclosure;

[0016] Figure 3 is a schematic diagram according to a third embodiment of the present disclosure;

[0017] Figure 4 is a schematic diagram according to a fourth embodiment of the present disclosure;

[0018] Figure 5 is a schematic diagram according to a fifth embodiment of the present disclosure;

[0019] Figure 6 is a schematic diagram according to a sixth embodiment of the present disclosure;

[0020] Figure 7 is a schematic diagram according to a seventh embodiment of the present disclosure;

[0021] Figure 8 is a schematic diagram according to an eighth embodiment of the present disclosure;

[0022] Figure 9 is a schematic diagram according to a ninth embodiment of the present disclosure;

[0023] Figure 10 is a schematic diagram according to a tenth embodiment of the present disclosure;

[0024] Figure 11 is a schematic diagram of an electronic device for implementing a processing method of an unstable test case according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0025] Exemplary embodiments of the present disclosure are described below with reference to the accompanying drawings, which include various details of the embodiments of the present disclosure to assist in understanding, which should be considered in a descriptive sense only. Thus, it will be apparent to one of ordinary skill in the art that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Also, for the sake of brevity and clarity, descriptions of well-known functions and constructions are omitted from the following description.

[0026] Since unstable test cases can cause unstable and unreliable test results, unstable test cases are generally identified first for subsequent processing of unstable test cases.

[0027] In related art, a set of test cases can be run multiple times in a defect-free baseline software environment, and test results of at least one failed test case are marked as unstable test cases, i.e., marked as flaky.

[0028] However, this processing method has problems of too large labeling magnitude and large accuracy fluctuation in actual environment.

[0029] To improve the processing effect of unstable test cases, the present disclosure can provide the following embodiments.

[0030] Figure 1 is a schematic diagram according to a first embodiment of the present disclosure, which provides a processing method of unstable test, comprising:

[0031] 101、obtain at least one set of test running data of a test case, each set of test running data in the at least one set of test running data comprising: test results and retry information.

[0032] 102、determine an unstable probability value based on the number of sets of the at least one set of test running data, the test results and the retry information, the unstable probability value being used to determine whether the test case is an unstable test case.

[0033] 103. If the instability probability value is greater than a preset threshold value, determining that the test case is an unstable test case.

[0034] The execution subject of the embodiment can be a processing device of an unstable test case, and the specific form of the device is not limited and can be hardware, software, or a combination of software and hardware. The device can be located in an electronic device, which can be a user terminal or a server. The server can be a local server or a cloud server, and the user terminal can include a mobile device (such as a mobile phone or a tablet computer), a wearable device (such as a smart watch or a smart bracelet), a vehicle-mounted device (such as a car machine), and the like.

[0035] Generally, the method of the embodiment can be applied to a server.

[0036] The application scenario of the embodiment can be an automatic testing platform or framework of software.

[0037] As shown in Figure 2 , a tester or a developer can transmit a tested software to a server 202 through a user terminal 201. The server includes an automatic testing platform, and the tested software (which can be specifically a source code of software) is input to the automatic testing platform. The automatic testing platform can store or obtain test cases from other components. The automatic testing platform uses the test cases to perform automatic testing on the tested software to obtain a test result.

[0038] The test result can be success (pass) or failure (fail).

[0039] It can be understood that the meaning of failure (fail) is broad, and a non-successful state can be classified as failure. That is, failure can also include a case where the test result is an error.

[0040] One test case can be used to test different versions of a tested software. For example, the tested software includes a first version, a second version, a third version, and the like. For each version, a set of test run data can be obtained. For example, for a test case A, which is used to test the above three versions of the tested software, three sets of test run data can be obtained for the test case A.

[0041] For a test case, assuming that the test case includes N (N is a positive integer) sets of test run data, at least one set of test run data of the test case can be as shown in Figure 3 .

[0042] For each set of test run data, the test result (success or failure) and retry information can be included.

[0043] The test result refers to a final test result. In the automated test platform, if a failure occurs, a default automatic retry is performed. For example, if the first test result is a failure, a new test process can be automatically initiated until a preset maximum number of retries is reached. The maximum number of retries is generally 3. If the maximum number of retries is reached and the test result is still a failure, the final test result is a failure. Conversely, if the first test result is a success, no retry process is needed, and the final test result is a success.

[0044] The retry information can be a retry number or identification information of whether a retry occurs. For example, if the maximum number of retries is 3, the retry number of a test case can be any one of 0, 1, 2, and 3. The identification information of whether a retry occurs can be represented by 0 or 1, where 0 indicates that no retry occurs, and 1 indicates that a retry occurs.

[0045] Therefore, based on the retry information, it can be determined whether a retry occurs. For example, if the retry information is a retry number, if the retry number is a non-zero value, it indicates that a retry occurs. Alternatively, if the retry information is identification information of whether a retry occurs, it can be directly determined whether a retry occurs based on the identification information.

[0046] Taking the identification information of whether a retry occurs as an example, for a test case A, N (N is a positive integer) sets of test running data as shown in Table 1 can be obtained. The test result (success or failure) and the identification information of whether a retry occurs (0 or 1) in each set of test running data. Figure 3

[0047] Then, based on the number of sets of test running data, the test result, and the retry information, an unstable probability value can be determined.

[0048] If the unstable probability value is greater than a preset threshold, the test case can be determined to be an unstable test case.

[0049] In related technologies, a test case with a test result in failure is considered to be an unstable case. However, since the tested software itself can have a problem, at this time, the test result should be a failure. That is, after a test case tests a tested software with a problem, the test result is a failure, which is normal. At this time, the test case cannot be considered to be an unstable test case. Therefore, the related technologies are not accurate in detecting unstable test cases based on failures.

[0050] ​In the embodiment, the unstable probability value is determined based on the number of sets of test run data, the test result and retry information, and when the unstable probability value is greater than a preset threshold, the test case is determined as an unstable test case. Therefore, the unstable probability value is not only related to the test result, but also related to other information, and the unstable test case is determined when the unstable probability value is greater than the preset threshold. Compared with the related art in which the test case is considered as an unstable test case when a failure occurs, the detection accuracy of the unstable test case can be improved, and the processing effect can be improved.

[0051] In some embodiments, the at least one set of test run data of the test case comprises:

[0052] The at least one set of historical test run data is obtained from the historical run records of the test case as the at least one set of test run data.

[0053] That is, the at least one set of test run data can be specifically historical test run data.

[0054] The process of determining whether the test case is an unstable test case is referred to as a detection process of the unstable test case, and the historical test run data refers to test run data that has existed before the detection process of the unstable test case.

[0055] Generally, a lot of historical information has been accumulated in an automated test platform, and the historical information includes historical test run data of test cases. Therefore, the historical test run data can be obtained from the historical run records, and subsequent processing can be performed based on the historical test run data.

[0056] Since the historical test run data is existing data, it is not necessary to perform an online test process to obtain real-time test run data. Therefore, existing resources can be fully utilized, and time can be saved, and the processing efficiency can be improved.

[0057] In some embodiments, the unstable probability value is inversely proportional to a probability value of a successful test result, and is proportional to a probability value of a successful test result with retry.

[0058] In the formula, the probability value of a successful test result is referred to as a first probability value, and the probability value of a successful test result with retry is referred to as a second probability value. Therefore, the unstable probability value is inversely proportional to the first probability value, and is proportional to the second probability value.

[0059] In one implementation, the unstable probability value is a ratio of the second probability value to the first probability value. The formula is as follows:

[0060] P=P2 / P1;

[0061] P is an unstable probability value, P1 is a first probability value, and P2 is a second probability value.

[0062] The unstable probability value is used to reflect the instability of the test case.

[0063] When the test result is success, if the test case is stable, it should be success at the beginning, that is, the less the retries, the more stable the test case, and the more retries, the more unstable the test case. Therefore, based on the above direct and inverse relationship, an unstable probability value that can measure the instability of the test case can be obtained.

[0064] Further, by taking the ratio of the second probability value to the first probability value as the unstable probability value, the unstable probability value can be obtained simply and quickly.

[0065] In some embodiments, the first probability value is determined based on the number of groups and the number of times of success of the test result, including:

[0066] The ratio of the number of times of success of the test result to the number of groups is determined as the first probability value.

[0067] In some embodiments, the second probability value is determined based on the number of groups, the number of times of success of the test result, and the number of times of retries indicated by the retry information, including:

[0068] The ratio of the number of times of success of the test result and the number of times of retries indicated by the retry information to the number of groups is determined as the second probability value.

[0069] That is, the calculation formula of P1 and P2 can be as follows:

[0070] P1=N1 / N, and P2=N2 / N;

[0071] Wherein, N is the number of groups of all test running data; N1 is the number of groups of test running data with success test result in N groups of test running data, which can be referred to as the number of times of success test result; N2 is the number of groups of test running data with success test result and retries in N groups of test running data, which can be referred to as the number of times of success test result and retries indicated by the retry information.

[0072] The above calculation process conforms to the determination process of the probability value, and is more reasonable.

[0073] In some embodiments, the method further includes:

[0074] If the test case is an unstable test case, the unstable test case is isolated.

[0075] For example, if test case A is detected as a flaky test case, test case A can be isolated. Specifically, the state of test case A can be marked as a non-execution state.

[0076] By isolating flaky test cases, flaky test cases can be excluded from the automated testing process, avoiding the impact of flaky test cases on the normal testing process and ensuring the smooth execution of the testing process.

[0077] The above respectively illustrates embodiments of different aspects, and embodiments combining multiple aspects can also be provided, such as Figures 4-6 as shown.

[0078] Among them, Figure 4 a system architecture diagram is shown. As Figure 4 shown, the system architecture can include pre-preparation, automated testing service, flaky test case and isolation. These steps (or services) can be implemented through a continuous integration pipeline.

[0079] Pre-preparation is used to generate the object under test, such as the source code under test. Specifically, it can include code compilation and program packaging and deployment, so that the source code under test can be obtained.

[0080] The automated testing service performs automated testing on the source code under test. Specifically, it can include an automated testing distribution framework, an automated test case suite, and a database. During automated testing, a distributed approach can be used, and the automated testing distribution framework can distribute the source code under test to the target container for execution. The automated test case suite is a collection of test cases (which can be referred to simply as cases). During automated testing, the test cases are used to test the source code under test.

[0081] The database can store test cases and their reports, which can include test run data for the test cases.

[0082] The flaky test case detection and isolation service is used to detect flaky test cases and isolate flaky test cases. It can include flaky test case calculation and flaky test case labeling. Flaky test case calculation can calculate flaky probability values based on test run data and detect flaky test cases based on the flaky probability values. Flaky test case labeling can label test cases detected as flaky test cases as flaky, i.e., as flaky.

[0083] Figure 5 A hardware environment diagram is shown. As Figure 5As shown, the hardware environment can include a code management server, a test server, a test auxiliary computing server, a test database and a container. The container can include a processor, memory, disk, etc.

[0084] The code management server can correspond to the pre-preparation in Figure 4 , i.e., obtaining the source code to be tested.

[0085] The test server and the test database can correspond to the automated test service in Figure 4 , i.e., using test cases to automatically test the source code to be tested.

[0086] The test auxiliary computing server can correspond to the unstable test case detection and isolation in Figure 4 , which can detect and isolate unstable test cases.

[0087] The automated test can use a distributed processing mode, i.e., the source code to be tested and the test cases can be distributed in multiple containers.

[0088] Figure 6 A method flow diagram is shown.

[0089] As shown in Figure 6 , the processing method of unstable test cases provided by the embodiment can include:

[0090] 601. Obtain at least one set of historical test running data of the test cases.

[0091] The historical test running data can be as shown in Figure 7 .

[0092] As shown in Figure 7 , it is assumed that the objects to be tested include: software of test version V1.0.1, software of test version V1.0.2, software of test version V1.0.3, and test cases include: use case_1, use case_2 and use case_3.

[0093] The historical test running data is 9 sets, and each set of historical test running data can include information of the object to be tested (such as the test software version), information of the test case (such as the test case number), test results (success or failure) and retry information (such as the number of retries).

[0094] Further, the historical test running data can be obtained in a historical test process, and the historical test process can execute the flow as shown in Figure 8 to obtain the historical test running data.

[0095] As shown in Figure 8 , the historical test process can include: initialization, running the automated use case multiple times, and storing the test running data of the test cases.

[0096] wherein the test cases are represented by automated cases in Figure 8 .

[0097] At initialization, a software deployment environment can be prepared, and a set of automated cases can be prepared, the set of automated cases including at least one automated case.

[0098] For a certain automated case, there can be multiple runs of the automated case, i.e., the automated case is run multiple times. For example, Figure 7 , for each test case, there are three runs, corresponding to three versions respectively.

[0099] For each run, a retry mechanism can be used, i.e., the automated case is retried if it fails. For each run, it can be divided into multiple single execution processes. For example, if the result of the first single execution process is success, the test result of the corresponding run is success. If the result of the first single execution process is failure, a retry is automatically initiated, i.e., the second single execution process is performed, until the result of the single execution process is success or the maximum number of retries is reached, which is generally 3. If the result of the single execution process after the maximum number of retries is still failure, the test result of the corresponding run is failure.

[0100] For each run, the test result of the run and the number of retries can be recorded, i.e., the test run data of the automated case is stored.

[0101] 602、based on the number of groups and the number of times of success, a first probability value is determined, the first probability value being a probability value of success.

[0102] For each test case, its corresponding first probability value can be obtained, the first probability value being represented by P(success) in Figure 7 .

[0103] The calculation formula can be:

[0104] P1=N1 / N;

[0105] wherein P1 is the first probability value, N1 is the number of times of success, and N is the number of groups of test run data.

[0106] Taking case_1 as an example, N=3, N1=3 (three groups of test results are success), then P1=3 / 3=1.

[0107] Similarly, for case_2, N=3, N1=1, then P1=1 / 3.

[0108] Similarly, for use case_3, N=3, N1=2, then P1=2 / 3.

[0109] Figure 7 For example, each probability value retains 4 decimal places.

[0110] 603、based on the number of groups, the test result is successful and the retry information indicates the number of times of retry, determine a second probability value, the second probability value is the probability value of the test result being successful and retrying.

[0111] Among them, for each test case, its corresponding second probability value can be obtained, and the second probability value is in Figure 7 In P(retry∩success) is used.

[0112] The calculation formula can be:

[0113] P2=N2 / N;

[0114] Among them, P2 is the second probability value, N2 is the number of times of retrying when the test result is successful, and N is the number of groups of test running data.

[0115] Among them, Figure 7 The retry information in the above formula is the number of retries, and the number of retries of 0 indicates that no retry occurs, and a number other than 0 indicates that retry occurs.

[0116] For example, use case_1, N=3, N2=2 (three groups of test results are successful, of which 2 groups occur retry), then P2=2 / 3=1.

[0117] Similarly, for use case_2, N=3, N2=0, then P2=0.

[0118] Similarly, for use case_3, N=3, N2=1, then P2=1 / 3.

[0119] It can be understood that 602 and 603 have no time sequence limitation relationship, 602 can be executed first and then 603 is executed, or 603 is executed first and then 602 is executed, or 602 and 603 are executed in parallel.

[0120] 604、determine the ratio of the second probability value to the first probability value, and take the ratio as the unstable probability value.

[0121] Among them, for each test case, its corresponding unstable probability value can be obtained, and the unstable probability value is in Figure 7 In P(retry|success) is used.

[0122] The calculation formula can be:

[0123] P=P2 / P1;

[0124] wherein P is the unstable probability value, P1 is the first probability value, and P2 is the second probability value.

[0125] Taking use case_1 as an example, P1=1, P2=0.6667, thus, P=P2 / P1=0.6667.

[0126] Similarly, for use case_2, P1=0.3333, P2=0.0000, thus, P=P2 / P1=0.0000.

[0127] Similarly, for use case_3, P1=0.6667, P2=0.3333, thus, P=P2 / P1=0.4999.

[0128] It can be understood that other probability values can also be calculated for other services, for example, a retry probability P(retry) can also be calculated, P(retry)=number of times of retries / number of groups.

[0129] 605, determining whether the unstable probability value is greater than a preset threshold, if yes, executing 606, otherwise executing 607.

[0130] 606, determining that the test case corresponding to the unstable probability value is an unstable test case, and performing isolation processing on the unstable test case.

[0131] For example, assuming that the preset threshold is 0.25, referring to Figure 9 Since the unstable probability values corresponding to use case_1 and use case_3 are greater than the preset threshold, use case_1 and use case_3 are unstable test cases, and the unstable state of use case_1 can be marked as a flaky state, and the unstable state of use case_3 can be marked as a flaky state.

[0132] In addition, isolation processing can also be performed on the unstable test case, that is, the execution state of use case_1 can be marked as not executed, and the execution state of use case_3 can be marked as not executed.

[0133] 607, normally processing the test case.

[0134] That is, the test case can be used to automatically test the measured object.

[0135] In this embodiment, by detecting unstable test cases by using historical test running data, the processing efficiency can be improved without additional time investment; by performing isolation processing on unstable test cases, the influence of unstable test cases on the normal test process can be avoided, and the proportion of stable test cases can be improved; by detecting unstable test cases based on unstable probability values, the detection accuracy can be improved.

[0136] Figure 9is a schematic diagram according to a ninth embodiment of the present disclosure, and the embodiment provides a processing apparatus of unstable test cases. As shown in Figure 2 The processing apparatus 900 of unstable test cases includes an acquisition module 901, a first determination module 902, and a second determination module 903.

[0137] The acquisition module 901 is configured to acquire at least one set of test running data of a test case, each set of test running data in the at least one set of test running data including a test result and retry information; the first determination module 902 is configured to determine an unstable probability value based on a number of sets of the at least one set of test running data, the test result, and the retry information, the unstable probability value being used to determine whether the test case is an unstable test case; and the second determination module 903 is configured to determine that the test case is an unstable test case if the unstable probability value is greater than a preset threshold.

[0138] The execution subject of the embodiment can be a processing apparatus of unstable test cases, and the specific form of the apparatus is not limited, which can be hardware, software, or a combination of software and hardware. The apparatus can be located in an electronic device, which can be a user terminal or a server, the server can be a local server or a cloud server, and the user terminal can include a mobile device (such as a mobile phone or a tablet computer), a wearable device (such as a smart watch or a smart bracelet), a vehicle-mounted device (such as a car machine), and the like.

[0139] Generally, the method of the embodiment can be applied to a server.

[0140] The application scenario of the embodiment can be an automatic testing platform or framework of software.

[0141] As shown in Figure 3 A tester or a developer can transmit a software to be tested to a server 202 through a user terminal 201, the server including an automatic testing platform, the software to be tested (which can be specifically a source code of software) being input to the automatic testing platform, the automatic testing platform storing test cases or obtaining the test cases from other components, the automatic testing platform using the test cases to perform automatic testing on the software to be tested to obtain a test result.

[0142] The test result can be success (pass) or failure (fail).

[0143] It can be understood that the meaning of failure (fail) is broad, and a non-successful state can be classified as failure, that is, failure can also include a case where the test result is error.

[0144] One test case can be used to test different versions of the tested software, for example, the tested software includes a first version, a second version, a third version, etc., for each version, a set of test run data can be obtained, for example, for a certain test case A, it is used to test the above three versions of the tested software, then for the test case A, three sets of test run data can be obtained.

[0145] For a certain test case, assuming that the test case includes N (N is a positive integer) sets of test run data, at least one set of test run data of the test case can be as shown in Figure 3

[0146] Among them, for each set of test run data, it can include test results (success or failure) and retry information.

[0147] Among them, the test result refers to the final test result. In the automatic test platform, if a failure occurs, the default is to automatically perform a retry, for example, if the first test result is a failure, a new test process can be automatically initiated until the maximum number of retries is reached, which is generally 3 times, if the maximum number of retries is reached, the test result is still a failure, and the final test result is a failure. Conversely, if the first test result is a success, no retry process is needed, and the final test result is a success.

[0148] The retry information can be: the number of retries, or the identification information of whether a retry occurs. For example, if the maximum number of retries is 3, then for a certain test case, the number of retries can be any one of 0, 1, 2, and 3. The identification information of whether a retry occurs can be represented by 0 or 1, where 0 indicates that no retry has occurred, and 1 indicates that a retry has occurred.

[0149] Therefore, based on the retry information, it can be determined whether a retry has occurred. For example, if the retry information is the number of retries, if the number of retries is a non-zero value, it indicates that a retry has occurred; or if the retry information is the identification information of whether a retry occurs, it can be directly determined whether a retry has occurred based on the identification information.

[0150] Taking the identification information of whether a retry occurs as an example of the retry information, for test case A, N (N is a positive integer) sets of test run data as shown in Figure 10 can be obtained, the test results (success or failure) and the identification information of whether a retry occurs (0 or 1) in each set of test run data.

[0151] Then, based on the number of test run data sets, test results, and retry information, the instability probability value can be determined.

[0152] If the instability probability value is greater than a preset threshold, the test case can be determined to be an unstable test case. ​

[0153] In the related art, a test case with a test result of failure is considered as an unstable case. However, the test result should be failure due to a problem of the tested software itself. In this case, the test result of failure is normal after a test case tests a tested software with a problem. Therefore, the unstable test case is not detected based on the failure in the related art, which is inaccurate.

[0154] In this embodiment, the unstable probability value is determined based on the number of groups of test running data, the test result, and the retry information. When the unstable probability value is greater than a preset threshold, the test case is determined as an unstable test case. Therefore, the unstable probability value is related to not only the test result but also other information. The unstable test case is determined when the unstable probability value is greater than the preset threshold, which can improve the detection accuracy of the unstable test case and improve the processing effect.

[0155] In some embodiments, the obtaining module 901 is further configured to: obtain at least one group of historical test running data from a historical running record of the test case as the at least one kind of test running data.

[0156] That is, the at least one group of test running data can be specifically historical test running data.

[0157] The process of determining whether the test case is an unstable test case is referred to as a detection process of the unstable test case. The historical test running data refers to test running data that has existed before the detection process of the unstable test case.

[0158] Generally, a lot of historical information has been accumulated in an automated test platform. The historical information includes historical test running data of test cases. Therefore, the historical test running data can be obtained from the historical running record, and subsequent processing can be performed based on the historical test running data.

[0159] Since the historical test running data is existing data, a test process does not need to be performed online to obtain real-time test running data. Therefore, existing resources can be fully utilized, and time can be saved, and the processing efficiency can be improved.

[0160] In some embodiments, the unstable probability value is inversely proportional to a probability value of the test result being success and is proportional to a probability value of the test result being success and retrying.

[0161] In this embodiment, the probability value of the test result being success is referred to as a first probability value, and the probability value of the test result being success and retrying is referred to as a second probability value. The unstable probability value is inversely proportional to the first probability value and is proportional to the second probability value.

[0162] In an implementation, the instability probability value is a ratio of the second probability value and the first probability value. The formula is as follows:

[0163] P = P2 / P1;

[0164] wherein P is the instability probability value, P1 is the first probability value, and P2 is the second probability value.

[0165] The instability probability value is used to reflect the instability of the test case.

[0166] When the test result is success, if the test case is stable, it should be success at the beginning, that is, the less the retries, the more stable the test case, and the more retries, the more unstable the test case. Therefore, based on the above direct and inverse relationship, the instability probability value that can measure the instability of the test case can be obtained.

[0167] Further, by taking the ratio of the second probability value and the first probability value as the instability probability value, the instability probability value can be obtained conveniently and quickly.

[0168] In some embodiments, the first determining module 902 is further configured to determine, as the first probability value, a ratio of the number of times of success of the test result and the number of groups.

[0169] In some embodiments, the first determining module 902 is further configured to determine, as the second probability value, a ratio of the number of times of success of the test result and the number of groups.

[0170] That is, the calculation formula of P1 and P2 can be as follows:

[0171] P1 = N1 / N, and P2 = N2 / N;

[0172] wherein N is the number of groups of all test running data; N1 is the number of groups of test running data with success of the test result in the N groups of test running data, which can be referred to as the number of times of success of the test result; and N2 is the number of groups of test running data with success of the test result and retries in the N groups of test running data, which can be referred to as the number of times of success of the test result and retries indicated by the retry information.

[0173] The above calculation process conforms to the determination process of the probability value, and is more reasonable.

[0174] In some embodiments, the apparatus 900 further includes an isolation module configured to perform isolation processing on the unstable test case if the test case is an unstable test case.

[0175] For example, if test case A is detected as an unstable test case, test case A can be isolated. Specifically, the state of test case A can be marked as a non-execution state.

[0176] By isolating unstable test cases, unstable test cases can be prevented from being executed in the automated testing process, thereby avoiding the impact of unstable test cases on the normal testing process and ensuring smooth execution of the testing process.

[0177] It can be understood that the same or similar content in different embodiments in the embodiments of the present disclosure can be mutually referred to.

[0178] It can be understood that "first", "second", and the like in the embodiments of the present disclosure are only used for distinction, and do not represent importance, time sequence, and the like.

[0179] In the technical solutions of the present disclosure, the collection, storage, use, processing, transmission, provision, and disclosure of user personal information involved in the technical solutions comply with relevant laws and regulations and do not violate public order and good customs.

[0180] According to the embodiments of the present disclosure, the present disclosure further provides an electronic device, a readable storage medium, and a computer program product.

[0181] As shown in Figure 11 , the present disclosure further provides an automated testing platform 1000, which includes an electronic device 1001.

[0182] The electronic device 1001 can be as shown in Figure 11 .

[0183] Figure 11 A schematic block diagram of an example electronic device 1100 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptops, desktops, tablets, servers, blades, mainframes, and other appropriate computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular telephones, smart phones, wearable devices, and other similar computing devices. The components shown here, their connections and relationships, and their functions, are meant to be examples only, and are not intended to limit implementations of the present disclosure described and / or claimed in this document.

[0184] As shown in ​As shown, the electronic device 1100 includes a computing unit 1101 that can perform various appropriate actions and processes in accordance with a computer program stored in a read-only memory (ROM) 1102 or a computer program loaded into a random access memory (RAM) 1103 from a storage unit 1108. Various programs and data required for the operation of the electronic device 1100 can also be stored in the RAM 1103. The computing unit 1101, the ROM 1102, and the RAM 1103 are connected to each other through a bus 1104. An input / output (I / O) interface 1105 is also connected to the bus 1104.

[0185] Various components in the electronic device 1100 are connected to the I / O interface 1105, including an input unit 1106 such as a keyboard, a mouse, and the like, an output unit 1107 such as various types of displays, a speaker, and the like, a storage unit 1108 such as a magnetic disk, an optical disk, and the like, and a communication unit 1109 such as a network card, a modem, a wireless communication transceiver, and the like. The communication unit 1109 allows the electronic device 1100 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0186] The computing unit 1101 can be various general and / or special purpose processing components having processing and computing capabilities. Some examples of the computing unit 1101 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, and the like. The computing unit 1101 performs various methods and processes described above, such as the processing method for unstable test cases. For example, in some embodiments, the processing method for unstable test cases can be implemented as a computer software program that is tangibly embodied in a machine-readable medium, such as the storage unit 1108. In some embodiments, part or all of the computer program can be loaded and / or installed on the electronic device 1100 via the ROM 1102 and / or the communication unit 1109. When the computer program is loaded into the RAM 1103 and executed by the computing unit 1101, one or more steps of the processing method for unstable test cases described above can be performed. Alternatively, in other embodiments, the computing unit 1101 can be configured to perform the processing method for unstable test cases by any other appropriate means, such as by means of firmware.

[0187] The various embodiments of the systems and techniques described above can be implemented in digital electronic circuitry, integrated circuitry, a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a system on a chip (SOC), a complex programmable logic device (CPLD), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include implementation in one or more computer programs that are executable and / or interpretable on a programmable system including at least one programmable processor, which can be special or general purpose, coupled to receive data and instructions from, and to transmit data and instructions to, a storage system, at least one input device, and at least one output device.

[0188] Program code for carrying out methods of the present disclosure can be written in any combination of one or more programming languages. The program code can be provided to a processor or controller of a general purpose computer, special purpose computer, or other programmable processing apparatus to produce a machine, such that the program code, when executed by the processor or controller, implements the functions / acts specified in the flowcharts and / or block diagrams. The program code can be executed entirely on a machine, partially on a machine, partially on a machine as a standalone software package, or entirely on a remote machine or server.

[0189] In the context of the present disclosure, a machine-readable medium can be a tangible medium that contains or stores a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include but is not limited to an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of the machine-readable storage medium will include one or more lines of electrical connections, portable computer disks, hard disk drives, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or Flash memory), optical fibers, portable compact disc read-only memories (CD-ROMs), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0190] To provide for interaction with a user, the systems and techniques described here can be implemented on a computer having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form, including acoustic, speech, or tactile input.

[0191] The systems and techniques described here can be implemented in a computing system that includes a back end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front end component (e.g., a user computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such back end, middleware, or front end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet.

[0192] The computer system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. A server can be a cloud server, also known as cloud computing server or cloud host, which is a host product in the cloud computing service system, to solve the defects of large management difficulty and weak business scalability in traditional physical host and VPS (Virtual Private Server, or VPS for short) services. The server can also be a server of a distributed system, or a server combined with a blockchain.

[0193] It should be understood that various forms of flow shown above can be used with orders of steps reordered, added to, or removed. For example, steps recited in the present disclosure can be performed in parallel, in series, or in a different order, without limitation herein, as long as the desired results of the technical solutions of the present disclosure can be achieved.

[0194] The above detailed description does not limit the scope of the disclosure. Various modifications, combinations, sub-combinations and alternatives can be made to the detailed description. Any modification, equivalent replacement and improvement etc. made within the spirit and principle of the disclosure shall be included in the scope of the disclosure.

Claims

1. A method for processing unstable test cases, comprising: obtaining at least one set of test run data of a test case, each set of test run data in the at least one set of test run data comprising: test results and retry information; a same test case is used to test different versions of a software under test, and a set of test run data is obtained for each version; determining an unstable probability value based on a number of sets of the at least one set of test run data, the test results and the retry information, the unstable probability value being used to determine whether the test case is an unstable test case; if the unstable probability value is greater than a preset threshold, determining that the test case is an unstable test case; the determining of the unstable probability value based on the number of sets of the at least one set of test run data, the test results and the retry information comprises: determining a first probability value based on the number of sets and a number of times of success of the test results, the first probability value being a probability value of success of the test results; determining a second probability value based on the number of sets, a number of times of success of the test results and a number of times of retry indicated by the retry information, the second probability value being a probability value of success of the test results and retry; determining a ratio of the second probability value to the first probability value, and taking the ratio as the unstable probability value.

2. The method of claim 1, wherein, the obtaining of the at least one set of test run data of the test case comprises: obtaining at least one set of historical test run data from historical run records of the test case as the at least one set of test run data.

3. The method of claim 1, wherein, the determining of the first probability value based on the number of sets and the number of times of success of the test results comprises: determining a ratio of the number of times of success of the test results to the number of sets as the first probability value.

4. The method of claim 1, wherein, the determining of the second probability value based on the number of sets, the number of times of success of the test results and the number of times of retry indicated by the retry information comprises: determining a ratio of the number of times of success of the test results and the number of times of retry indicated by the retry information to the number of sets as the second probability value. 5.The method according to any one of claims 1-4, further comprising: if the test case is an unstable test case, performing isolation processing on the unstable test case. 6.An apparatus for processing unstable test cases, comprising: an obtaining module, configured to obtain at least one set of test run data of a test case, each set of test run data in the at least one set of test run data comprising: test results and retry information; a first determining module, configured to determine an unstable probability value based on a number of sets of the at least one set of test run data, the test results and the retry information, the unstable probability value being used to determine whether the test case is an unstable test case; a second determining module, configured to determine that the test case is an unstable test case if the unstable probability value is greater than a preset threshold; the first determining module is further configured to: determine a first probability value based on the number of sets and a number of times of success of the test results, the first probability value being a probability value of success of the test results; determining a second probability value based on the number of groups, the test result being successful and the retry information indicating the number of times of retries, the second probability value being a probability value of the test result being successful and retries occurring; determining a ratio of the second probability value and the first probability value, and taking the ratio as the instability probability value.

7. The apparatus of claim 6, wherein, The acquisition module is further configured to: acquire at least one set of historical test running data from historical running records of the test case as the at least one test running data.

8. The apparatus of claim 6, wherein, The first determination module is further configured to: determine the ratio of the number of times of the test result being successful and the number of groups as the first probability value.

9. The apparatus of claim 6, wherein, The first determination module is further configured to: determine the ratio of the number of times of the test result being successful and the number of groups as the first probability value.

10. The apparatus according to any one of claims 6-9, further comprising: an isolation module configured to, if the test case is an unstable test case, perform isolation processing on the unstable test case.

11. An electronic device, comprising: at least one processor; and a memory connected to the at least one processor in communication; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-5.

12. A non-transitory computer readable storage medium having stored thereon computer instructions, wherein, The computer instructions are used to enable the computer to perform the method of any one of claims 1-5.

13. A computer program product comprising a computer program which, when executed by a processor, implements the method of any one of claims 1-5.

14. An automated test platform comprising: The electronic device of claim 11.

Images