Method, device, medium and system for determining execution result of high-availability test case

Abstract

The invention provides a method, a device, a medium and a system for determining execution results of high-availability test cases. A new data sequence is generated on the basis of a data sequence, a preset TPS minimum value and a preset TPS maximum value, a first interpolation sequence is obtained, and under the condition that the current sequence similarity is larger than a similarity threshold value and / or the current TPS failure frequency is larger than a failure frequency threshold value, it is determined that the execution result of the high-availability test case is the scene starting abnormity. Therefore, the sequence similarity and the TPS failure times are taken into consideration of a high-availability test result, the accuracy of a test conclusion is improved, and the problem that the accuracy of the test conclusion is difficult to guarantee in a traditional high-availability test is solved.

Description

Technical Field

[0001] The present application relates to the technical field of IT systems, and in particular to a method, device, medium and system for determining execution results of high-availability test cases. Background Art

[0002] High availability is one of the factors that must be considered in the design of IT system architecture today. It is crucial for many important systems, so it is of great significance to conduct high availability testing on the system. However, there are two problems in traditional high availability testing. First, the testing capabilities provided by the existing high availability testing tools in the industry are mainly based on virtual machine abnormality simulation, and lack relevant functions for determining the correctness of transaction results; second, the current analysis and judgment of high availability test results mainly rely on testers, and the workload of result analysis is large, making it difficult to carry out continuous testing. To a certain extent, the judgment of the correctness of the results depends on the experience and skill level of the testers, making it difficult to ensure the uniformity and accuracy of the test conclusions. Summary of the invention

[0003] The main purpose of the present application is to provide a method, device, medium and system for determining the execution results of a high-availability test case, so as to at least solve the problem that it is difficult to ensure the accuracy of the test conclusion in traditional high-availability testing.

[0004] In order to achieve the above object, according to one aspect of the present application, a method for determining a high-availability test case execution result is provided, the method comprising:

[0005] Acquire execution process information of a high-availability test case, wherein the execution process information includes at least one data sequence;

[0006] Based on the data sequence, the preset TPS minimum value and the preset TPS maximum value, a new data sequence is generated to obtain a first interpolation sequence;

[0007] The current sequence similarity and the current TPS failure number are obtained, and when the current sequence similarity is greater than a similarity threshold and / or the current TPS failure number is greater than a failure number threshold, it is determined that the execution result of the high-availability test case is a scenario startup exception, wherein the current sequence similarity represents the Euclidean distance between the data sequence and the first interpolation sequence, and the current TPS failure number is the number of TPS failures in the data sequence.

[0008] Optionally, in the process of generating a new data sequence based on the data sequence, the preset TPS minimum value and the preset TPS maximum value to obtain the first interpolation sequence, the method further includes:

[0009] When the target parameter in the data sequence is within a preset range, the target parameter is inserted into the first interpolation sequence, and the preset range is [preset TPS minimum value, preset TPS maximum value]; when the target parameter in the data sequence is less than the preset TPS minimum value, the preset TPS minimum value is inserted into the first interpolation sequence; when the target parameter in the data sequence is greater than the preset TPS maximum value, the preset TPS maximum value is inserted into the first interpolation sequence.

[0010] Optionally, when the current sequence similarity is less than or equal to the similarity threshold, and the current TPS failure number is less than or equal to the failure number threshold, the method further includes: determining that the current window sequence is a window sequence starting with the abnormal operation point and ending with a preset duration in the execution process information; generating a new data sequence based on the current window sequence, the preset TPS minimum value and the preset TPS maximum value to obtain a second interpolation sequence; and shifting the current window sequence right by the preset duration to obtain a first corrected window sequence.

[0011] Optionally, after obtaining the first corrected window sequence, the method further includes: when the similarity between the current window sequence and the second interpolation sequence is less than or equal to the similarity threshold, based on the first corrected window sequence, the preset TPS minimum value and the preset TPS maximum value, generating a new data sequence to obtain a third interpolation sequence; when the similarity between the current window sequence and the second interpolation sequence is less than or equal to the similarity threshold, and the similarity between the first parameter in the current window sequence and the first parameter in the second interpolation sequence is less than or equal to the similarity threshold, determining that the execution result of the high-availability test case is a scenario failure; when the similarity between the current window sequence and the second interpolation sequence is less than or equal to the similarity threshold, determining that the execution result of the high-availability test case is a scenario failure; When the similarity is less than or equal to the similarity threshold, and the similarity between the first parameter in the current window sequence and the first parameter in the second interpolation sequence is greater than the similarity threshold, the actual impact degree of TPS is obtained, and the actual impact degree of TPS includes the number of TPS failures, the TPS impact time and the TPS decline amplitude, the TPS impact time is the difference between the TPS fluctuation end point and the TPS fluctuation start point, and the TPS decline amplitude is the ratio of the TPS when it returns to a normal level after the TPS fluctuation time to the TPS when it returns to a stable level but has not returned to a normal level; when the similarity between the current window sequence and the second interpolation sequence is greater than the similarity threshold, the actual impact degree of TPS is obtained.

[0012] Optionally, after obtaining the actual impact degree of TPS, the method further includes: when the actual impact degree of TPS is different from the expected result, determining that the execution result of the high-availability test case is a scenario failure; when the actual impact degree of TPS is the same as the expected result, executing a use case exception recovery operation to restore the execution process of the high-availability test case.

[0013] Optionally, after obtaining the first corrected window sequence, the method further includes: when the similarity between the current window sequence and the second interpolation sequence is greater than the similarity threshold, generating a new data sequence based on the first corrected window sequence, the preset TPS minimum value and the preset TPS maximum value to obtain a third interpolation sequence; when the similarity between the first corrected window sequence and the third interpolation sequence is greater than the similarity threshold, performing a use case exception recovery operation to restore the execution process of the high-availability test case; when the similarity between the first corrected window sequence and the third interpolation sequence is less than or equal to the similarity threshold, determining that the execution result of the high-availability test case is a scenario failure.

[0014] Optionally, after executing the use case exception recovery operation, the method further includes: obtaining the use case exception recovery result, and when the use case exception recovery result indicates that the use case exception recovery failed, determining that the execution result of the high-availability test case is a scenario failure; when the use case exception recovery result indicates that the use case exception recovery was successful, determining that the target window sequence is a window sequence that starts after the recovery operation point and ends with a preset duration in the execution process information; when the number of TPS failures of the target window sequence is greater than the failure number threshold, determining that the execution result of the high-availability test case is a scenario failure; when the number of TPS failures of the target window sequence is less than or equal to the failure number threshold, generating a new data sequence based on the target window sequence, the preset TPS minimum value and the preset TPS maximum value to obtain a fourth interpolation sequence; when the similarity between the target window sequence and the fourth interpolation sequence is less than or equal to the similarity threshold, determining the The execution result of the high-availability test case is a scenario failure; when the similarity between the target window sequence and the fourth interpolation sequence is greater than the similarity threshold, the target window sequence is right-shifted by a preset time length to obtain a second corrected window sequence; based on the second corrected window sequence, the preset TPS minimum value and the preset TPS maximum value, a new data sequence is generated to obtain a fifth interpolation sequence; when the number of failures of the second corrected window sequence is greater than the failure number threshold, and / or the similarity between the second corrected window sequence and the fifth interpolation sequence is greater than the similarity threshold, it is determined that the execution result of the high-availability test case is that the transaction is first not affected after the recovery operation, and then the transaction fails; when the number of failures of the second corrected window sequence is less than or equal to the failure number threshold, and the similarity between the second corrected window sequence and the fifth interpolation sequence is greater than the similarity threshold, it is determined that the execution result of the high-availability test case is that the transaction is not affected after the recovery operation.

[0015] According to another aspect of the present application, a device for determining a high-availability test case execution result is provided, the device comprising:

[0016] An acquisition unit, used to acquire execution process information of a high-availability test case, wherein the execution process information includes at least one data sequence;

[0017] A first processing unit is used to generate a new data sequence based on the data sequence, a preset TPS minimum value and a preset TPS maximum value to obtain a first interpolation sequence;

[0018] The second processing unit is used to obtain the current sequence similarity and the current TPS failure number, and determine that the execution result of the high-availability test case is a scenario startup exception when the current sequence similarity is greater than a similarity threshold and / or the current TPS failure number is greater than a failure number threshold, the current sequence similarity represents the Euclidean distance between the data sequence and the first interpolation sequence, and the current TPS failure number is the number of TPS failures in the data sequence.

[0019] According to another aspect of the present application, a computer-readable storage medium is provided, wherein the computer-readable storage medium includes a stored program, wherein when the program is executed, the device where the computer-readable storage medium is located is controlled to execute any one of the methods described.

[0020] According to another aspect of the present application, a system for determining the execution results of a high-availability test case is provided, the system comprising: one or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by the one or more processors, and the one or more programs include methods for executing any one of the described methods.

[0021] By applying the technical solution of the present application, a new data sequence is generated based on the data sequence, the preset TPS minimum value and the preset TPS maximum value to obtain a first interpolation sequence. When the current sequence similarity is greater than a similarity threshold, and / or the current TPS failure number is greater than a failure number threshold, it is determined that the execution result of the high-availability test case is a scenario startup exception, thereby incorporating sequence similarity and TPS failure number into consideration of the high-availability test result, thereby improving the accuracy of the test conclusion, and further solving the problem that it is difficult to ensure the accuracy of the test conclusion in traditional high-availability testing. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] The drawings constituting part of the present application are used to provide a further understanding of the present application. The exemplary embodiments and descriptions of the present application are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0023] Figure 1 A flow chart of a method for determining a high-availability test case execution result provided in accordance with an embodiment of the present application is shown;

[0024] Figure 2 A schematic diagram of the process of stage 2 provided according to an embodiment of the present application is shown;

[0025] Figure 3 A schematic diagram of a process for calculating the TPS impact degree provided according to an embodiment of the present application is shown;

[0026] Figure 4 A structural block diagram of a device for determining execution results of a high-availability test case provided according to an embodiment of the present application is shown. DETAILED DESCRIPTION

[0027] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0028] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0029] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present application described here. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0030] For the convenience of description, some nouns or terms involved in the embodiments of the present application are explained below:

[0031] High availability: High availability refers to the ability of a system to perform its functions without interruption, indicating the degree of availability of the system. Through design, the time that the system cannot provide services normally can be reduced. When a part of the system fails for various reasons, the system can be ensured to be continuously available and the system service failure time can be minimized.

[0032] High availability testing: High availability testing is a non-functional test. It verifies whether the system can provide services normally (or degrade services) by simulating situations when the system has an abnormality, and verifies whether the system can recover to the operating state before the abnormality occurs after the abnormality is recovered.

[0033] Boxplot: Also known as box-and-whisker plot or box plot, it is a statistical chart used to show the dispersion of a set of data and can be used to detect data outliers.

[0034] Sliding window: It is a commonly used algorithm idea. The basic principle is to maintain a window that slides in the string. Each slide can maintain some specific status or information and determine whether the current window meets the requirements of the problem.

[0035] Time series data: refers to time series data, which is a data column recorded in chronological order. Each data in the same data column must be of the same caliber and must be comparable.

[0036] Similarity: Similarity refers to the degree of similarity between two things and is usually used to compare the similarity between two data sets.

[0037] Euclidean distance: also known as Euclidean distance, is the most common distance metric. It measures the absolute (i.e. straight-line) distance between two points in multidimensional space. The smaller the Euclidean distance value, the more similar the two data sets are.

[0038] Similarity threshold: The similarity threshold refers to the maximum value set when performing similarity comparison. The similarity value used in this article is the Euclidean distance value. When the similarity calculation result is not greater than the similarity threshold, the two are considered similar, otherwise they are not similar.

[0039] TPS failure threshold: When the high-availability scenario is running stably, there may be a small number of occasional errors in transactions (the success rate of each transaction is not necessarily 100% when it is running normally), that is, there may be a certain number of TPS failures in each window. In this patent, the TPS failure threshold means that if the number of TPS failures in a window is not greater than the TPS failure threshold, then it is considered that the number of TPS failures in the window is in line with expectations.

[0040] As introduced in the background technology, high availability is one of the factors that must be considered in the design of IT system architecture today. It is crucial for many important systems, so it is of great significance to conduct high availability testing on the system. However, in traditional high availability testing, there are the following two problems. First, the testing capabilities provided by the existing high availability testing tools in the industry are mainly based on virtual machine abnormality simulation, and lack relevant functions for determining the correctness of transaction results; second, the current analysis and judgment of high availability test results mainly rely on testers, and the workload of result analysis is large, making it difficult to carry out continuous testing, and to a certain extent, the judgment of the correctness of the results depends on the experience and skill level of the testers, making it difficult to ensure the uniformity and accuracy of the test conclusions. In order to solve the problem that it is difficult to ensure the accuracy of test conclusions in traditional high availability testing, the embodiments of the present application provide a method, device, medium and system for determining the execution results of a high availability test case.

[0041] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the accompanying drawings in the embodiments of the present invention.

[0042] In this embodiment, a method for determining the execution results of a high-availability test case is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0043] Figure 1 1 is a flow chart of a method for determining the execution result of a high-availability test case provided in accordance with an embodiment of the present application. Figure 1 As shown, the method comprises the following steps:

[0044] Step S101, obtaining execution process information of a high-availability test case, wherein the execution process information includes at least one data sequence;

[0045] Specifically, preprocessing:

[0046] TPS target value range preprocessing:

[0047] Take a TPS data sequence C for a period of time (such as 30 minutes) when the high availability scenario is running stably.

[0048] Since there may be a small number of abnormal points in the TPS data sequence C, in order to improve the accuracy of subsequent calculations, according to the time series data characteristics and curve characteristics of TPS during stable operation in high-availability scenarios, C is denoised based on the box plot method to obtain the data sequence Cq.

[0049] Calculate the minimum value min(Cq) and maximum value max(Cq) of Cq, that is, the preset TPS minimum value and the preset TPS maximum value, and determine the TPS target value range R as [min(Cq), max(Cq)], which will be used for subsequent outlier correction.

[0050] Similarity threshold:

[0051] Take a TPS data sequence T1 of a period (such as 8 hours) during stable operation of the high-availability scenario as the training set, and a TPS data sequence T2 of a period (such as 4 hours) as the validation set.

[0052] The data in the training set T1 is preprocessed point by point through the outlier correction algorithm, that is, the values ​​that are not in the target value range R are replaced with min(Cq) or max(Cq) to generate the corrected training set TR1.

[0053] Calculate the similarity between the right window T1 and TR1 point by point, and obtain the three-stage large window similarity sequence D1 and the influence degree small window similarity sequence D2;

[0054] First, sort the D1 and D2 sequences in ascending order, and then take out the numbers in the D1 and D2 sequences according to the preset accuracy h (if the accuracy h is 95%, then take out the maximum value of the first 95% numbers in the D1 and D2 sequences respectively), and finally assign values ​​to δ1 and δ2. The similarity threshold is related to the window size;

[0055] Use the validation set T2 to verify and adjust the parameters of δ1 and δ2. If δ1 and δ2 cannot detect the number of similarity windows with an accuracy of h or above, increase the values ​​of δ1 and δ2 so that the adjusted values ​​can detect the number of similarity windows with an accuracy of h or above, and finally obtain the similarity thresholds δ1 and δ2.

[0056] Failure count threshold:

[0057] Take a TPS data sequence T1 of a period (e.g., 8 hours) during stable operation of the high-availability scenario as the training set, and a TPS data sequence T2 of a period (e.g., 4 hours) as the validation set;

[0058] Calculate the number of TPS failures of the right window point by point to obtain the window failure number Fail sequence. The window size v is the three-stage window size.

[0059] First, sort the Fail sequence in ascending order, then take out the numbers in the Fail sequence according to the preset accuracy h (if the accuracy h is 95%, then take out the maximum value of the first 95% of the numbers in the Fail sequence), and finally assign it to f;

[0060] Use the validation set T2 to verify and adjust the parameters of f. If f cannot detect the expected number of windows with an accuracy of h or above, increase the value of f so that the adjusted value can detect the expected number of windows with an accuracy of h or above, and finally obtain the failure threshold f.

[0061] Test Case Management:

[0062] Test case import: Import high-availability test cases into the device. The test cases include test descriptions, test steps, expected test results, etc.

[0063] Execution command parsing: According to the use case test steps, parse out the exception simulation commands and exception recovery commands of each use case.

[0064] Expected result analysis: Based on the expected test results of the use case, the expected affected transaction list, impact time, TPS reduction, etc. of each use case are analyzed.

[0065] Test case execution:

[0066] The high availability scenario is automatically triggered and started according to preset conditions (such as time, scheduling order, etc.); after the result of stage one is passed, the use case exception simulation command is automatically triggered to execute; after the result of stage two is passed, the use case exception recovery command is automatically triggered to execute; after stage three is completed, the high availability scenario is automatically ended, and the execution results of commands in each stage are detected.

[0067] Process information collection:

[0068] After the high availability scenario is started, process information such as successful TPS and failed TPS of each transaction is collected in real time and written into the database.

[0069] Step S102, based on the above data sequence, the preset TPS minimum value and the preset TPS maximum value, a new data sequence is generated to obtain a first interpolation sequence;

[0070] In one embodiment of the present application, in step S102, i.e., in the process of generating a new data sequence based on the above data sequence, the preset TPS minimum value and the preset TPS maximum value to obtain the first interpolation sequence, the above method further includes:

[0071] When the target parameter in the data sequence is within a preset range, the target parameter is inserted into the first interpolation sequence, and the preset range is [preset TPS minimum value, preset TPS maximum value];

[0072] In the case where the target parameter in the data sequence is less than the preset TPS minimum value, inserting the preset TPS minimum value into the first interpolation sequence;

[0073] When the target parameter in the data sequence is greater than the preset TPS maximum value, the preset TPS maximum value is inserted into the first interpolation sequence.

[0074] Outlier correction algorithm:

[0075] The specific steps are as follows:

[0076] Step 1) Based on the box plot method, denoising is performed to obtain the TPS target value reference range R sequence, where min(R) is the lower limit of the TPS target value, and max(R) is the upper limit of the TPS target value. Values ​​outside the upper and lower limits of the TPS target value are considered as abnormal values;

[0077] Step 2) Process the data sequence Cn point by point: If the TPS value Cn[i] is within [min(R), max(R)], then Cn[i] is directly inserted into the CRn sequence; if the TPS value Cn[i] < min(R), then min(R) is used to replace Cn[i] and inserted into the CRn sequence; if the TPS value Cn[i] > max(R), then max(R) is used to replace Cn[i] and inserted into the CRn sequence;

[0078] Step 3) After processing the data sequence Cn point by point according to Step 2), a new data sequence CRn is generated, corresponding to the above first interpolation sequence.

[0079] Step S103, obtain the current sequence similarity and the current number of TPS failures, and determine that the execution result of the above high-availability test case is an abnormal scenario startup when the above current sequence similarity is greater than the similarity threshold, and / or the above current number of TPS failures is greater than the failure times threshold. The above current sequence similarity represents the Euclidean distance between the above data sequence and the above first interpolation sequence, and the above current number of TPS failures is the number of TPS failures in the above data sequence.

[0080] In the above steps, by generating a new data sequence based on the above data sequence, the preset minimum TPS value, and the preset maximum TPS value, a first interpolation sequence is obtained. When the above current sequence similarity is greater than the similarity threshold, and / or the above current number of TPS failures is greater than the failure times threshold, it is determined that the execution result of the above high-availability test case is an abnormal scenario startup, thereby incorporating the sequence similarity and the number of TPS failures into the consideration of the high-availability test result, thus improving the accuracy of the test conclusion, and further solving the problem that it is difficult to ensure the accuracy of the test conclusion in traditional high-availability tests.

[0081] Similarity algorithm:

[0082] Take a TPS window data sequence Cn during the operation of the high-availability scenario, generate a data sequence CRn through an outlier correction algorithm, calculate the similarity between the data sequences Cn and CRn, and determine whether the TPS of the window Cn is in a steady state (i.e., the stable state during normal operation of the scenario) through the similarity. The specific steps are as follows:

[0083] Traverse the data points of the window Cn and generate a CRn sequence of the same length using the outlier correction algorithm;

[0084] Calculate the Euclidean distance between the TPS data sequences Cn and CRn, that is, measure the similarity between the TPS data sequences before and after outlier correction of the window based on the Euclidean distance, and avoid the influence of time-axis migration and stretching on the similarity;

[0085] Judge whether the similarity between Cn and CRn is not greater than the similarity threshold obtained by preprocessing;

[0086] If it is not greater than, it means that the TPS of window Cn is in a steady state; if it is greater than, it means that the TPS of window Cn is not in a steady state.

[0087] Phase 1: Transaction pull-up, before abnormal operation. Automatically trigger the start of the high-availability scenario. After the transaction scenario is pulled up for a period of time (such as 1min to 5min), take the TPS data sequence window C0 (the window size is determined according to the situation of each project, such as a window size of 2 minutes). First, compare the number of TPS failures in window C0 with the failure number threshold f, and then use the similarity algorithm (see the similarity algorithm above for details) to determine whether the TPS window C0 is similar to CR0. If the number of failures in C0 is greater than the failure number threshold f, or C0 is not similar to CR0, then "End: Scenario startup abnormality"; if the number of failures in C0 is less than or equal to the failure number threshold f, and C0 is similar to CR0, then enter Phase 2.

[0088] In one embodiment of the present application, when the current sequence similarity is less than or equal to the similarity threshold, and the current TPS failure number is less than or equal to the failure number threshold, the method further includes:

[0089] Determine that the current window sequence is a window sequence starting with the abnormal operation point and ending with the preset duration in the above execution process information (ie, the window C1 below);

[0090] Based on the current window sequence, the preset minimum TPS value and the preset maximum TPS value, a new data sequence is generated to obtain a second interpolation sequence (ie, CR1 below);

[0091] The current window sequence is shifted right by the preset time length to obtain a first modified window sequence (ie, the TPS sequence window Cn after the window is shifted right).

[0092] In one embodiment of the present application, after obtaining the first modified window sequence, the method further includes:

[0093] When the similarity between the current window sequence and the second interpolation sequence is less than or equal to the similarity threshold, a new data sequence is generated based on the first corrected window sequence, the preset TPS minimum value and the preset TPS maximum value to obtain a third interpolation sequence (i.e., CRn);

[0094] When the similarity between the current window sequence and the second interpolation sequence is less than or equal to the similarity threshold, and the similarity between the first parameter in the current window sequence and the first parameter in the second interpolation sequence is less than or equal to the similarity threshold, determining that the execution result of the high availability test case is a scenario failure;

[0095] In the case where the similarity between the current window sequence and the second interpolation sequence is less than or equal to the similarity threshold, and the similarity between the first parameter in the current window sequence and the first parameter in the second interpolation sequence is greater than the similarity threshold, the actual impact degree of TPS is obtained, where the actual impact degree of TPS includes the number of TPS failures, the TPS impact time, and the TPS decrease amplitude, where the TPS impact time is the difference between the TPS fluctuation end point and the TPS fluctuation start point, and the TPS decrease amplitude is the ratio of the TPS when the TPS returns to a normal level after the TPS fluctuation time to the TPS when the TPS returns to a stable level but has not returned to a normal level;

[0096] In the case where the similarity between the current window sequence and the second interpolation sequence is greater than the similarity threshold, the actual impact degree of the TPS is obtained.

[0097] In one embodiment of the present application, after obtaining the actual impact degree of TPS, the method further includes:

[0098] In the case where the actual impact degree of the above TPS is different from the expected result, the execution result of the above high availability test case is determined to be a scenario failure;

[0099] When the actual impact of the above TPS is the same as the above expected result, the use case abnormal recovery operation is performed to restore the execution process of the above high-availability test case.

[0100] like Figure 2 As shown, Phase II includes the following steps:

[0101] Step S201, automatically trigger the execution of the use case exception simulation command, and determine whether the abnormal operation is successful. If not, "End: abnormal operation execution is abnormal"; if so, enter step S202;

[0102] Step S202, take the TPS sequence window C1 after the abnormal operation point (the window size is the same as the window size in stage one), and determine whether the number of failures of C1 is less than or equal to the failure number threshold f. If not, go to step S203; if so, go to step S207.

[0103] Step S203, take the TPS sequence window Cn after the window is moved right (the window size is the same as the window size in stage 1, and the sliding step size is the same as the window size), and determine whether the number of failures of Cn is less than or equal to the number of failures threshold f. If not, "End: Transaction continues to fail after abnormal operation, scenario fails"; if so, proceed to step S204;

[0104] Step S204, determine whether Cn is similar to CRn, if not, proceed to step S205; if yes, proceed to step S206.

[0105] Step S205, determine whether Cn1 is similar to CRn1 (Cn1 and CRn1 are new sequences generated by adding the TPS mean difference to all data points in windows Cn and CRn). If not, then "End: TPS is not stable after abnormal operation, has not returned to normal level, and the scenario fails"; if so, proceed to step S206;

[0106] Step S206, calculate the actual impact of the abnormal operation on TPS (see TPS impact calculation below for details), including the affected transaction list, number of failures, impact time, and TPS drop. Compare the results with the expected results in the high-availability test case to determine whether the actual test results meet expectations. If not, "End: The actual results after the abnormal operation do not meet expectations, and the scenario fails"; if so, enter stage three;

[0107] Step S207, determine whether C1 is similar to CR1, if not, go to step S203; if yes, go to step S208;

[0108] Step S208, take the TPS sequence window Cp after the window is shifted to the right (the window size is the same as the window size in stage one, and the sliding step size is the same as the window size), and determine whether the number of failures of Cp is less than or equal to the number of failures threshold f, and whether Cp is similar to CRp. If the number of failures of Cp is greater than the number of failures threshold f, or Cp is not similar to CRp, then "End: After the abnormal operation, the transaction was not affected at first, and then the transaction failed / TPS jitter occurred, and the scenario failed"; if the number of failures of Cp is less than or equal to the number of failures threshold f and Cp is similar to CRp, then enter stage three.

[0109] TPS impact calculation, such as Figure 3 As shown, including:

[0110] 1) Determine whether the three-stage large window data output from stage 2 has a TPS failure count exceeding the failure count threshold, record the name of the affected transaction, and if so, go to 2), otherwise go to 11).

[0111] 2) Based on the three-stage large window C1 data output from stage 2, confirm the starting time point and the ending time point of TPS failure, record the total number of TPS failures for the transaction, and turn to 3).

[0112] 3) Based on the time series data of the test execution process, determine whether the TPS similarity between the left window F0 and the FR0 sequence at the TPS failure starting time point is not greater than the similarity threshold (i.e., similar). If so, record the TPS failure starting time point as the TPS fluctuation starting point and turn to 5); if not, turn to 4).

[0113] 4) Slide the window to the left (use a small window and a small sliding step, the window size is configurable, the default is 1 minute, and the sliding step is 1 second), determine whether the TPS similarity between the window Fn and the FRn sequence is not greater than the similarity threshold, if not, go back to 4), if yes, record the maximum point of the window time that is not greater than the similarity threshold as the starting point of TPS fluctuation and go to 5).

[0114] 5) Based on the time series data of the test execution process, determine whether the TPS similarity between the right window E0 and the ER0 sequence at the TPS failure termination time point is not greater than the similarity threshold. If so, go to 6), if not, go to 7).

[0115] 6) Record the minimum time point of the window En that is not greater than the similarity threshold as the end point of the TPS fluctuation, record the conclusion that "TPS returns to normal level", and turn to 10).

[0116] 7) Calculate and record the difference in TPS mean between the window En and the ERn sequence, add the difference in TPS mean to all data points in the window En to generate the En1 sequence, and determine whether the TPS similarity between the En1 sequence and the ERn1 sequence is not greater than the similarity threshold. If not, go to 8), if so, go to 9).

[0117] 8) Slide the window right (use a small window and a small sliding step, the window size is configurable, the default is 1 minute, and the sliding step is 1 second) to determine whether the TPS similarity between the window En and the Rn sequence is not greater than the similarity threshold. If so, go to 6), otherwise go to 7).

[0118] 9) Record the minimum window time that is not greater than the similarity threshold in 7) as the TPS fluctuation termination point, record the mean difference of the window as the decline amplitude, record "TPS returns to stability but not to normal level", and turn to 10).

[0119] 10) Calculate the TPS fluctuation time (the time difference between the fluctuation end point and the fluctuation start point), and output the actual situation of the transaction affected, including transaction name, total number of TPS failures, TPS fluctuation time, TPS recovery to normal level / TPS recovery but not to normal level (decline).

[0120] 11) After merging the three-stage large window data output from stage 2, delete the data points whose TPS is not less than min(TPS) in the S1 preprocessing TPS steady-state R sequence, generate the Q sequence, and turn to 12).

[0121] 12) Traverse the data points of the Q sequence from front to back in time, obtain the left window Fn of the data points, and turn to 13).

[0122] 13) Determine whether the TPS similarity between the window Fn and the FRn sequence is greater than the similarity threshold. If so, record the data point time as the TPS fluctuation starting point and turn to 14), otherwise turn to 12).

[0123] 14) Determine whether the TPS similarity between the right window E0 and the ER0 sequence of the TPS fluctuation starting point is not greater than the similarity threshold. If so, go to 6), otherwise go to 7).

[0124] The final three situations: record the end point of TPS fluctuation, and TPS returns to normal level; record the end point of TPS fluctuation, record the mean difference as the decline, and record that TPS returns to stability but does not return to normal level; record the end point of TPS fluctuation, and TPS returns to normal level.

[0125] Record TPS fluctuation time: the difference between the TPS fluctuation end point and the TPS fluctuation start point;

[0126] Output the actual impact of the transaction (transaction name, total number of TPS failures, TPS fluctuations for Y time before returning to normal levels / TPS returns to stability but has not returned to normal levels, and the extent of the decline).

[0127] In one embodiment of the present application, after obtaining the first modified window sequence, the method further includes:

[0128] When the similarity between the current window sequence and the second interpolation sequence is greater than the similarity threshold, a new data sequence is generated based on the first corrected window sequence, the preset TPS minimum value and the preset TPS maximum value to obtain a third interpolation sequence;

[0129] When the similarity between the first correction window sequence and the third interpolation sequence is greater than the similarity threshold, performing a use case abnormality recovery operation to recover the execution process of the high availability test case;

[0130] When the similarity between the first corrected window sequence and the third interpolation sequence is less than or equal to the similarity threshold, it is determined that the execution result of the high availability test case is a scenario failure.

[0131] In one embodiment of the present application, after performing the use case abnormality recovery operation, the method further includes:

[0132] Obtaining a use case abnormal recovery result, and determining that the execution result of the high availability test case is a scenario failure when the use case abnormal recovery result indicates that the use case abnormal recovery has failed;

[0133] In the case where the above-mentioned use case abnormal recovery result indicates that the use case abnormal recovery is successful, determining the target window sequence as a window sequence starting after the recovery operation point in the above-mentioned execution process information and ending with a preset duration;

[0134] When the number of TPS failures of the target window sequence is greater than the failure number threshold, determining that the execution result of the high availability test case is a scenario failure;

[0135] When the number of TPS failures of the target window sequence is less than or equal to the number of failures threshold, a new data sequence is generated based on the target window sequence, the preset minimum TPS value, and the preset maximum TPS value to obtain a fourth interpolation sequence;

[0136] In a case where the similarity between the target window sequence and the fourth interpolation sequence is less than or equal to the similarity threshold, determining that the execution result of the high availability test case is a scenario failure;

[0137] When the similarity between the target window sequence and the fourth interpolation sequence is greater than the similarity threshold, the target window sequence is right-shifted by a preset time length to obtain a second corrected window sequence;

[0138] Based on the second correction window sequence, the preset TPS minimum value and the preset TPS maximum value, a new data sequence is generated to obtain a fifth interpolation sequence;

[0139] When the number of failures of the second correction window sequence is greater than the failure number threshold, and / or the similarity between the second correction window sequence and the fifth interpolation sequence is greater than the similarity threshold, it is determined that the execution result of the high availability test case is that the transaction is not affected after the recovery operation, and then the transaction fails;

[0140] When the number of failures of the second correction window sequence is less than or equal to the failure number threshold, and the similarity between the second correction window sequence and the fifth interpolation sequence is greater than the similarity threshold, it is determined that the execution result of the high availability test case is that there is no impact on transactions after the recovery operation.

[0141] Phase 3 includes:

[0142] Step S301: After the second stage is over, the abnormal recovery command of the use case is automatically triggered to determine whether the recovery operation is successful. If it fails, "End: Recovery operation execution is abnormal"; if it succeeds, go to step S302;

[0143] Step S302, take the TPS data sequence window C1 after the recovery operation point (the window size is the same as the window size of the stage), and determine whether the number of failures of C1 is less than or equal to the failure number threshold f. If not, "End: X transactions failed after the recovery operation, the scenario failed"; if so, go to step S303;

[0144] Step S303, using a similarity algorithm to determine whether C1 is similar to CR1, if not, then "end: TPS jitter after recovery operation, scenario failed"; if yes, proceed to step S304;

[0145] Step S304, take the TPS sequence window Cp after the window is shifted to the right (the window size is the same as the window size of the same stage, and the sliding step size is the same as the window size), and determine whether the number of failures of Cp is less than or equal to the number of failures threshold f, and whether Cp is similar to CRp. If the number of failures of Cp is greater than the number of failures threshold f or Cp is not similar to CRp, then "End: After the recovery operation, the transaction is not affected at first, and then the transaction fails / TPS jitters, and the scenario fails"; if the number of failures of Cp is less than or equal to the number of failures threshold f and Cp is similar to CRp, then "End: After the recovery operation, the transaction is not affected".

[0146] Result processing: Summarize the calculation results of the three stages (including the affected transaction list / number of failures / impact time / TPS decline, etc.) and update them to the test report, output the abnormal information of the test process, so that testers can locate the cause of the failure of the use case execution.

[0147] It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and that, although a logical order is shown in the flowcharts, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0148] The embodiment of the present application also provides a device for determining the execution result of a high-availability test case. It should be noted that the device for determining the execution result of a high-availability test case in the embodiment of the present application can be used to execute the method for determining the execution result of a high-availability test case provided in the embodiment of the present application. The device is used to implement the above-mentioned embodiments and preferred implementation modes, and the descriptions that have been made will not be repeated. As used below, the term "module" can implement a combination of software and / or hardware for a predetermined function. Although the device described in the following embodiments is preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceived.

[0149] The following introduces the device for determining the execution results of a high-availability test case provided in an embodiment of the present application.

[0150] Figure 41 is a structural block diagram of a device for determining a high-availability test case execution result according to an embodiment of the present application. Figure 4 As shown, the device comprises:

[0151] An acquiring unit 41 is used to acquire execution process information of a high-availability test case, wherein the execution process information includes at least one data sequence;

[0152] A first processing unit 42 is used to generate a new data sequence based on the data sequence, a preset TPS minimum value and a preset TPS maximum value to obtain a first interpolation sequence;

[0153] The second processing unit 43 is used to obtain the current sequence similarity and the current TPS failure number, and determine that the execution result of the above-mentioned high-availability test case is a scenario startup exception when the above-mentioned current sequence similarity is greater than the similarity threshold and / or the above-mentioned current TPS failure number is greater than the failure number threshold, the above-mentioned current sequence similarity represents the Euclidean distance between the above-mentioned data sequence and the above-mentioned first interpolation sequence, and the above-mentioned current TPS failure number is the number of TPS failures in the above-mentioned data sequence.

[0154] In the above-mentioned device, a new data sequence is generated based on the above-mentioned data sequence, the preset TPS minimum value and the preset TPS maximum value to obtain a first interpolation sequence. When the above-mentioned current sequence similarity is greater than the similarity threshold, and / or the above-mentioned current TPS failure number is greater than the failure number threshold, the execution result of the above-mentioned high-availability test case is determined to be a scenario startup exception, thereby incorporating sequence similarity and TPS failure number into consideration of the high-availability test result, thereby improving the accuracy of the test conclusion, and further solving the problem that it is difficult to ensure the accuracy of the test conclusion in traditional high-availability testing.

[0155] In one embodiment of the present application, the first processing unit includes a first processing module, a second processing module and a third processing module.

[0156] The first processing module is used for generating a new data sequence based on the data sequence, the preset TPS minimum value and the preset TPS maximum value, and in the process of obtaining the first interpolation sequence, if the target parameter in the data sequence is within a preset range, inserting the target parameter into the first interpolation sequence, and the preset range is [preset TPS minimum value, preset TPS maximum value];

[0157] The second processing module is used for inserting the preset TPS minimum value into the first interpolation sequence when the target parameter in the data sequence is less than the preset TPS minimum value;

[0158] The third processing module is used for inserting the preset TPS maximum value into the first interpolation sequence when the target parameter in the data sequence is greater than the preset TPS maximum value.

[0159] In one embodiment of the present application, the above-mentioned device further includes a third processing unit, a fourth processing unit and a fifth processing unit.

[0160] The third processing unit is used to determine that the current window sequence is a window sequence starting with the abnormal operation point and ending with a preset duration in the execution process information when the current sequence similarity is less than or equal to the similarity threshold and the current TPS failure number is less than or equal to the failure number threshold;

[0161] The fourth processing unit is used to generate a new data sequence based on the current window sequence, the preset TPS minimum value and the preset TPS maximum value to obtain a second interpolation sequence;

[0162] The fifth processing unit is used to right-shift the current window sequence by the preset time length to obtain a first modified window sequence.

[0163] In one embodiment of the present application, the above-mentioned device further includes a sixth processing unit, a seventh processing unit, an eighth processing unit and a ninth processing unit.

[0164] The sixth processing unit is used for, after obtaining the first corrected window sequence, generating a new data sequence based on the first corrected window sequence, the preset TPS minimum value and the preset TPS maximum value to obtain a third interpolation sequence when the similarity between the current window sequence and the second interpolation sequence is less than or equal to the similarity threshold value;

[0165] The seventh processing unit is used to determine that the execution result of the high availability test case is a scenario failure when the similarity between the current window sequence and the second interpolation sequence is less than or equal to the similarity threshold, and the similarity between the first parameter in the current window sequence and the first parameter in the second interpolation sequence is less than or equal to the similarity threshold;

[0166] The eighth processing unit is used to obtain the actual impact degree of TPS when the similarity between the current window sequence and the second interpolation sequence is less than or equal to the similarity threshold, and the similarity between the first parameter in the current window sequence and the first parameter in the second interpolation sequence is greater than the similarity threshold, wherein the actual impact degree of TPS includes the number of TPS failures, the TPS impact time and the TPS decline range, wherein the TPS impact time is the difference between the TPS fluctuation end point and the TPS fluctuation start point, and the TPS decline range is the ratio of the TPS when the TPS returns to a normal level after the TPS fluctuation time to the TPS when the TPS returns to a stable level but has not returned to a normal level;

[0167] The ninth processing unit is used for obtaining the actual influence degree of the TPS when the similarity between the current window sequence and the second interpolation sequence is greater than the similarity threshold.

[0168] In one embodiment of the present application, the above-mentioned device further includes a tenth processing unit and an eleventh processing unit.

[0169] The tenth processing unit is used to determine that the execution result of the high availability test case is a scenario failure after obtaining the actual impact degree of the TPS, if the actual impact degree of the TPS is different from the expected result;

[0170] The eleventh processing unit is used to perform a use case exception recovery operation to restore the execution process of the high-availability test case when the actual impact degree of the above-mentioned TPS is the same as the above-mentioned expected result.

[0171] In one embodiment of the present application, the apparatus further includes a twelfth processing unit, a thirteenth processing unit and a fourteenth processing unit, after obtaining the first modified window sequence,

[0172] The twelfth processing unit is used for generating a new data sequence based on the first corrected window sequence, the preset TPS minimum value and the preset TPS maximum value to obtain a third interpolation sequence when the similarity between the current window sequence and the second interpolation sequence is greater than the similarity threshold;

[0173] The thirteenth processing unit is used for executing a case abnormality recovery operation to recover the execution process of the high-availability test case when the similarity between the first correction window sequence and the third interpolation sequence is greater than the similarity threshold;

[0174] The fourteenth processing unit is used to determine that the execution result of the high availability test case is a scenario failure when the similarity between the first corrected window sequence and the third interpolation sequence is less than or equal to the similarity threshold.

[0175] In one embodiment of the present application, the apparatus further includes a fifteenth processing unit, a sixteenth processing unit, a seventeenth processing unit, an eighteenth processing unit, a nineteenth processing unit, a twentieth processing unit, a twenty-first processing unit, a twenty-second processing unit, and a twenty-third processing unit.

[0176] The fifteenth processing unit is used to obtain a use case abnormal recovery result after executing the use case abnormal recovery operation, and determine that the execution result of the high availability test case is a scenario failure when the use case abnormal recovery result indicates that the use case abnormal recovery fails;

[0177] The sixteenth processing unit is used for determining the target window sequence as a window sequence starting after the recovery operation point and ending with a preset duration in the above-mentioned execution process information when the above-mentioned use case abnormal recovery result indicates that the use case abnormal recovery is successful;

[0178] The seventeenth processing unit is used to determine that the execution result of the high availability test case is a scenario failure when the number of TPS failures of the target window sequence is greater than the failure number threshold;

[0179] The eighteenth processing unit is used for generating a new data sequence based on the target window sequence, the preset TPS minimum value and the preset TPS maximum value to obtain a fourth interpolation sequence when the TPS failure number of the target window sequence is less than or equal to the failure number threshold;

[0180] The nineteenth processing unit is used to determine that the execution result of the high availability test case is a scenario failure when the similarity between the target window sequence and the fourth interpolation sequence is less than or equal to the similarity threshold;

[0181] The twentieth processing unit is used for shifting the target window sequence rightward by a preset time length to obtain a second corrected window sequence when the similarity between the target window sequence and the fourth interpolation sequence is greater than the similarity threshold;

[0182] The twenty-first processing unit is used to generate a new data sequence based on the second correction window sequence, the preset TPS minimum value and the preset TPS maximum value to obtain a fifth interpolation sequence;

[0183] The twenty-second processing unit is used to determine that the execution result of the high availability test case is that the transaction is not affected after the recovery operation and then fails when the number of failures of the second correction window sequence is greater than the failure number threshold and / or the similarity between the second correction window sequence and the fifth interpolation sequence is greater than the similarity threshold;

[0184] The twenty-third processing unit is used to determine that the execution result of the above-mentioned high availability test case is that there is no impact on transactions after the recovery operation when the number of failures of the above-mentioned second correction window sequence is less than or equal to the above-mentioned failure number threshold and the similarity between the above-mentioned second correction window sequence and the above-mentioned fifth interpolation sequence is greater than the above-mentioned similarity threshold.

[0185] The above-mentioned apparatus for determining the execution result of the high-availability test case includes a processor and a memory, wherein the above-mentioned acquisition unit, the first processing unit, the second processing unit, etc. are all stored in the memory as program units, and the processor executes the above-mentioned program units stored in the memory to realize the corresponding functions. The above-mentioned modules are all located in the same processor; or, the above-mentioned modules are located in different processors in any combination.

[0186] The processor contains a kernel, which calls the corresponding program unit from the memory. One or more kernels can be set, and the problem of difficulty in ensuring the accuracy of test conclusions in traditional high-availability testing can be solved by adjusting kernel parameters.

[0187] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM, and the memory includes at least one memory chip.

[0188] An embodiment of the present invention provides a computer-readable storage medium, which includes a stored program, wherein when the program is running, the device where the computer-readable storage medium is located is controlled to execute the method for determining the execution result of the high-availability test case.

[0189] An embodiment of the present invention provides a processor, which is used to run a program, wherein the method for determining the execution result of the high-availability test case is executed when the program is running.

[0190] An embodiment of the present invention provides a device, the device includes a processor, a memory, and a program stored in the memory and executable on the processor, and when the processor executes the program, at least the following steps are implemented: obtaining execution process information of a high-availability test case, the execution process information includes at least one data sequence; generating a new data sequence based on the data sequence, a preset TPS minimum value, and a preset TPS maximum value to obtain a first interpolation sequence; obtaining the current sequence similarity and the current TPS failure count, and determining that the execution result of the high-availability test case is a scenario startup exception when the current sequence similarity is greater than a similarity threshold, and / or the current TPS failure count is greater than a failure count threshold, the current sequence similarity represents the Euclidean distance between the data sequence and the first interpolation sequence, and the current TPS failure count is the TPS failure count in the data sequence. The device in this article can be a server, a PC, a PAD, a mobile phone, etc.

[0191] The present application also provides a computer program product, which, when executed on a data processing device, is suitable for executing a program that is initialized with at least the following method steps: obtaining execution process information of a high-availability test case, the execution process information including at least one data sequence; generating a new data sequence based on the data sequence, a preset TPS minimum value, and a preset TPS maximum value to obtain a first interpolation sequence; obtaining a current sequence similarity and a current TPS failure number, and determining that the execution result of the high-availability test case is a scenario startup exception when the current sequence similarity is greater than a similarity threshold, and / or the current TPS failure number is greater than a failure number threshold, the current sequence similarity represents the Euclidean distance between the data sequence and the first interpolation sequence, and the current TPS failure number is the TPS failure number in the data sequence.

[0192] The present application also provides a system for determining the execution result of a high-availability test case, the system comprising: one or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by the one or more processors, and the one or more programs include methods for executing any of the above methods. A new data sequence is generated based on the data sequence, a preset TPS minimum value, and a preset TPS maximum value to obtain a first interpolation sequence. When the current sequence similarity is greater than a similarity threshold, and / or the current TPS failure count is greater than a failure count threshold, the execution result of the high-availability test case is determined to be a scenario startup exception, thereby incorporating sequence similarity and TPS failure count into the consideration of the high-availability test result, thereby improving the accuracy of the test conclusion, and further solving the problem that it is difficult to ensure the accuracy of the test conclusion in traditional high-availability testing.

[0193] Obviously, those skilled in the art should understand that the above modules or steps of the present invention can be implemented by a general computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, and in some cases, the steps shown or described can be executed in a different order than here, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. Thus, the present invention is not limited to any specific combination of hardware and software.

[0194] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0195] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0196] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0197] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0198] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0199] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0200] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0201] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0202] From the above description, it can be seen that the above embodiments of the present application achieve the following technical effects:

[0203] 1) The method for determining the execution result of the high-availability test case of the present application generates a new data sequence based on the above-mentioned data sequence, the preset TPS minimum value and the preset TPS maximum value to obtain a first interpolation sequence. When the above-mentioned current sequence similarity is greater than the similarity threshold, and / or the above-mentioned current TPS failure number is greater than the failure number threshold, the execution result of the above-mentioned high-availability test case is determined to be a scenario startup exception, thereby incorporating sequence similarity and TPS failure number into consideration of the high-availability test result, thereby improving the accuracy of the test conclusion, and further solving the problem that it is difficult to ensure the accuracy of the test conclusion in traditional high-availability testing.

[0204] 2) The device for determining the execution result of the high-availability test case of the present application generates a new data sequence based on the above-mentioned data sequence, the preset TPS minimum value and the preset TPS maximum value to obtain a first interpolation sequence. When the above-mentioned current sequence similarity is greater than the similarity threshold, and / or the above-mentioned current TPS failure number is greater than the failure number threshold, the execution result of the above-mentioned high-availability test case is determined to be a scenario startup exception, thereby incorporating sequence similarity and TPS failure number into consideration of the high-availability test result, thereby improving the accuracy of the test conclusion, and further solving the problem that it is difficult to ensure the accuracy of the test conclusion in traditional high-availability testing.

[0205] The above description is only the preferred embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A method for determining the execution result of a high-availability test case, characterized in that: include: Acquire execution process information of a high-availability test case, wherein the execution process information includes at least one data sequence; Based on the data sequence, the preset TPS minimum value and the preset TPS maximum value, a new data sequence is generated to obtain a first interpolation sequence; The current sequence similarity and the current TPS failure number are obtained, and when the current sequence similarity is greater than a similarity threshold and / or the current TPS failure number is greater than a failure number threshold, it is determined that the execution result of the high-availability test case is a scenario startup exception, wherein the current sequence similarity represents the Euclidean distance between the data sequence and the first interpolation sequence, and the current TPS failure number is the number of TPS failures in the data sequence.

2. The method according to claim 1, characterized in that In the process of generating a new data sequence based on the data sequence, the preset TPS minimum value and the preset TPS maximum value to obtain the first interpolation sequence, the method further includes: When the target parameter in the data sequence is within a preset range, inserting the target parameter into the first interpolation sequence, the preset range being [preset TPS minimum value, preset TPS maximum value]; In the case where the target parameter in the data sequence is less than the preset TPS minimum value, inserting the preset TPS minimum value into the first interpolation sequence; In the case where the target parameter in the data sequence is greater than the preset TPS maximum value, the preset TPS maximum value is inserted into the first interpolation sequence.

3. The method according to claim 1, characterized in that In the case where the current sequence similarity is less than or equal to the similarity threshold, and the current TPS failure number is less than or equal to the failure number threshold, the method further includes: Determine that the current window sequence is a window sequence starting with the abnormal operation point and ending with a preset duration in the execution process information; Based on the current window sequence, the preset TPS minimum value and the preset TPS maximum value, a new data sequence is generated to obtain a second interpolation sequence; The current window sequence is shifted right by the preset time length to obtain a first modified window sequence.

4. The method according to claim 3, characterized in that After obtaining the first modified window sequence, the method further includes: When the similarity between the current window sequence and the second interpolation sequence is less than or equal to the similarity threshold, a new data sequence is generated based on the first modified window sequence, the preset TPS minimum value, and the preset TPS maximum value to obtain a third interpolation sequence; In a case where the similarity between the current window sequence and the second interpolation sequence is less than or equal to the similarity threshold, and the similarity between the first parameter in the current window sequence and the first parameter in the second interpolation sequence is less than or equal to the similarity threshold, determining that the execution result of the high availability test case is a scenario failure; In the case that the similarity between the current window sequence and the second interpolation sequence is less than or equal to the similarity threshold, and the similarity between the first parameter in the current window sequence and the first parameter in the second interpolation sequence is greater than the similarity threshold, the actual impact degree of TPS is obtained, and the actual impact degree of TPS includes the number of TPS failures, the TPS impact time, and the TPS decline amplitude, the TPS impact time is the difference between the TPS fluctuation end point and the TPS fluctuation start point, and the TPS decline amplitude is the ratio of the TPS when the TPS returns to a normal level after the TPS fluctuation time to the TPS when the TPS returns to a stable level but has not returned to a normal level; When the similarity between the current window sequence and the second interpolation sequence is greater than the similarity threshold, the actual impact degree of the TPS is obtained.

5. The method according to claim 4, characterized in that After obtaining the actual impact degree of TPS, the method further includes: In the case where the actual impact degree of the TPS is different from the expected result, determining that the execution result of the high availability test case is a scenario failure; When the actual impact degree of the TPS is the same as the expected result, a use case exception recovery operation is performed to restore the execution process of the high-availability test case.

6. The method according to claim 3, characterized in that After obtaining the first modified window sequence, the method further includes: When the similarity between the current window sequence and the second interpolation sequence is greater than the similarity threshold, a new data sequence is generated based on the first modified window sequence, the preset TPS minimum value, and the preset TPS maximum value to obtain a third interpolation sequence; When the similarity between the first correction window sequence and the third interpolation sequence is greater than the similarity threshold, performing a use case abnormality recovery operation to recover the execution process of the high-availability test case; When the similarity between the first correction window sequence and the third interpolation sequence is less than or equal to the similarity threshold, it is determined that the execution result of the high availability test case is a scenario failure.

7. The method according to claim 5 or 6, characterized in that: After executing the use case exception recovery operation, the method further includes: Obtaining a use case abnormal recovery result, and determining that an execution result of the high availability test case is a scenario failure when the use case abnormal recovery result indicates that the use case abnormal recovery has failed; In the case where the use case abnormal recovery result indicates that the use case abnormal recovery is successful, determining the target window sequence as a window sequence starting from after the recovery operation point and ending with a preset duration in the execution process information; When the number of TPS failures of the target window sequence is greater than the failure number threshold, determining that the execution result of the high availability test case is a scenario failure; When the number of TPS failures of the target window sequence is less than or equal to the failure number threshold, a new data sequence is generated based on the target window sequence, the preset TPS minimum value, and the preset TPS maximum value to obtain a fourth interpolation sequence; In a case where the similarity between the target window sequence and the fourth interpolation sequence is less than or equal to the similarity threshold, determining that the execution result of the high availability test case is a scenario failure; When the similarity between the target window sequence and the fourth interpolation sequence is greater than the similarity threshold, the target window sequence is right-shifted by a preset time length to obtain a second corrected window sequence; Based on the second correction window sequence, the preset TPS minimum value and the preset TPS maximum value, a new data sequence is generated to obtain a fifth interpolation sequence; When the number of failures of the second correction window sequence is greater than the failure number threshold, and / or the similarity between the second correction window sequence and the fifth interpolation sequence is greater than the similarity threshold, determining that the execution result of the high availability test case is that the transaction is first not affected after the recovery operation, and then the transaction fails; When the number of failures of the second correction window sequence is less than or equal to the failure number threshold, and the similarity between the second correction window sequence and the fifth interpolation sequence is greater than the similarity threshold, it is determined that the execution result of the high availability test case is that there is no impact on transactions after the recovery operation.

8. A device for determining execution results of a high-availability test case, characterized in that: include: An acquisition unit, used to acquire execution process information of a high-availability test case, wherein the execution process information includes at least one data sequence; A first processing unit is used to generate a new data sequence based on the data sequence, a preset TPS minimum value and a preset TPS maximum value to obtain a first interpolation sequence; The second processing unit is used to obtain the current sequence similarity and the current TPS failure number, and determine that the execution result of the high-availability test case is a scenario startup exception when the current sequence similarity is greater than a similarity threshold and / or the current TPS failure number is greater than a failure number threshold, the current sequence similarity represents the Euclidean distance between the data sequence and the first interpolation sequence, and the current TPS failure number is the number of TPS failures in the data sequence.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored program, wherein when the program is executed, the device where the computer-readable storage medium is located is controlled to execute the method according to any one of claims 1 to 7.

10. A system for determining execution results of high-availability test cases, characterized in that: include: One or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by the one or more processors, and the one or more programs include methods for executing any one of claims 1 to 7.

Images