A pressure testing system

Abstract

The present invention discloses a stress test system, including: a data acquisition system for collecting performance indicator data of a business system under different flow states; a training prediction system for obtaining performance indicator data collected from the business system, calibrating the performance indicator data into positive sample data and negative sample data, and then inputting the data into a performance test model for training and learning, so that the performance test model can output a prediction result of the business system operation state; a pressure flow determination system for adjusting the stress test flow value based on the prediction result of the business system operation state output by the performance test model until the performance limit state of the business system is determined; a pressure flow construction system for constructing a corresponding stress test flow based on the stress test flow value, and applying the stress test flow to the business system. It can realize automated testing of system performance in an unattended situation, with high test accuracy and precise feedback.

Description

Technical Field

[0001] The present invention relates to the field of testing technology, and in particular to a pressure testing system. Background Art

[0002] With the development of Internet technology, online shopping has become more and more popular. When users use apps or browsers to visit shopping websites for online shopping, their user behaviors such as searching, viewing products, viewing shopping carts, browsing products, clicking on advertisements, as well as media advertising placement, product display and other behaviors all require the use of website system resources, adding a certain load to the website's core service engine. Especially when there are large-scale promotional activities, the number of user behaviors increases significantly, which may cause the website system to be overloaded and unable to operate normally.

[0003] In order to ensure the normal operation of the shopping website system when the user behavior traffic increases, it is necessary to understand the operation of the website under high load conditions, so that the website will not have service anomalies, downtime and other abnormal situations when the user behavior traffic increases. Therefore, it is necessary to test the performance of the website system under high load scenarios and the traffic pressure it can withstand. Stress testing can also be used to respond promptly to problems that may occur under high-traffic user behavior conditions.

[0004] Traditional stress testing requires manual operation, such as manually triggering system flow pressure to simulate high-flow user behavior, manually monitoring online data such as system performance indicators after the pressure is triggered, and manually comparing the normal operation data of the system with the current online data. When the system is overloaded or abnormal, such as the feedback time is too high, the timeout rate increases, etc., it is necessary to manually confirm and adjust the pressure, and finally manually adjust the pressure flow to get the optimal solution. It can be seen that the existing system performance testing method requires a lot of manual operation, with relatively high human resource costs, cumbersome operation, and low automation; and it is impossible to detect subtle changes in system performance in a timely manner, and the accuracy of system performance testing and feedback precision are relatively low. Summary of the invention

[0005] In view of the above problems, the present invention is proposed to provide a pressure testing system that overcomes the above problems or at least partially solves the above problems.

[0006] An embodiment of the present invention provides a pressure testing system, including:

[0007] Data collection system, used to collect performance indicator data of business systems under different traffic conditions;

[0008] A training prediction system is used to obtain performance indicator data collected from the business system, calibrate the performance indicator data into positive sample data and negative sample data, and then input them into the performance test model for training and learning, so that the performance test model can output the prediction result of the business system operation status;

[0009] The pressure control system is used to adjust the pressure test flow value based on the prediction results of the business system operation status output by the performance test model until the performance limit state of the business system is determined;

[0010] A flow construction system is used to construct a corresponding stress test flow based on the stress test flow value, and to apply the stress test flow to the business system.

[0011] In some optional embodiments, the traffic construction system includes: a data intelligent construction system and a simulation scheduling system;

[0012] The data intelligent construction system is used to construct a corresponding number of user behavior requests that meet the requirements of the business system based on the pressure test flow value determined by the pressure flow determination system;

[0013] The simulation scheduling system is used to simulate the corresponding user behavior acting on the business system according to the user behavior request constructed by the data intelligent construction system.

[0014] In some optional embodiments, the simulation scheduling system includes a scheduling subsystem and a stress testing server cluster;

[0015] The scheduling subsystem is used to allocate a server in the stress testing server cluster to each user behavior included in the user behavior data according to a preset user behavior simulation strategy and a stress testing server allocation strategy;

[0016] Each server included in the stress testing server cluster is used to simulate the assigned user behavior to act on the business system.

[0017] In some optional embodiments, the training prediction system is further used to:

[0018] After acquiring the performance indicator data collected from the business system, performing normalization processing and / or denoising processing on the performance indicator data;

[0019] The performance indicators included in the processed performance indicator data are recombined and vectorized to obtain performance indicator data including multi-dimensional vectorized processing.

[0020] In some optional embodiments, the training prediction system performs normalization and denoising on the performance indicator data, including:

[0021] Determine a normalized dimension of each performance indicator in the performance indicator data, and convert each performance indicator value in each performance indicator data record into a specified order of magnitude according to the normalized dimension; and / or

[0022] For each performance indicator, the mean and variance of the performance indicator in each performance indicator data record are calculated, and the performance indicator of each performance indicator data record is filtered according to the obtained mean and variance to remove performance indicator data records whose deviation is greater than the specified condition.

[0023] In some optional embodiments, the training prediction system recombines and vectorizes the performance indicators included in the processed performance indicator data to obtain multi-dimensional vectorized performance indicator data, including:

[0024] Selecting performance indicators from the performance indicator data and recombining them to obtain at least one combined performance indicator;

[0025] For each performance indicator data record, each item of the performance indicator included therein and the obtained combined performance indicator is used as a dimension of vectorization processing to obtain the performance indicator data after multi-dimensional vectorization processing.

[0026] In some optional embodiments, the training prediction system is used to calibrate the performance indicator data into positive sample data and negative sample data, including:

[0027] The training prediction system marks the performance indicator data records in which all performance indicators in the performance indicator data are within the normal range as positive sample data, and marks the performance indicator data records in which any performance indicator in the performance indicator data is not within the normal range as negative sample data, based on the normal range of each performance indicator when the business system is operating normally.

[0028] In some optional embodiments, the training prediction system calibrates the performance indicator data into positive sample data and negative sample data and inputs the data into the performance test model for training and learning, so that the performance test model can output the prediction result of the business system operation status, including:

[0029] The training prediction system predicts the business system operating status corresponding to each performance indicator data record based on the performance indicator data records included in the input positive sample data and negative sample data, and counts whether the accuracy of the business system operating status prediction results corresponding to the positive sample data and negative sample data meets the requirements. If it does not meet the requirements, the next batch of performance indicator data is obtained to continue training and learning until a performance test model with a prediction accuracy that meets the requirements is obtained.

[0030] In some optional embodiments, the pressure control system is used to adjust the pressure test flow value based on the business system operation state prediction result output by the performance test model until the performance limit state of the business system is determined, including:

[0031] When the business system operation status prediction result output by the performance test model indicates that the business system operation status is normal, the pressure test flow value is increased according to the set pressure increase rules until the business system operation status becomes abnormal, and the performance limit state of the business system is determined; or

[0032] When the business system operation status prediction result output by the performance test model shows that the business system operation status is normal, the pressure test flow value is increased according to the set pressure increase rules until the business system operation status becomes abnormal, and the pressure test flow value is reduced according to the set pressure reduction rules until the business system operation status returns to normal, and the process returns to the step of increasing the pressure test flow value according to the set pressure increase rules. After repeating the above process for a set number of times, the performance limit state of the business system is determined.

[0033] In some optional embodiments, the pressure control system is further used for:

[0034] After the first abnormality occurs in the operation status of the business system, in the process of reducing the pressure test flow value according to the set decompression rules, if the performance indicator fluctuation of the business system is greater than the set threshold, the mean shift algorithm is used to determine the distance offset between the performance indicator value corresponding to the abnormal point where the performance indicator fluctuation is greater than the set threshold and the performance indicator mean. If the distance offset increases, it is considered that the system has reached its performance limit.

[0035] The beneficial effects of the above technical solution provided by the embodiment of the present invention include at least:

[0036] The stress testing system provided by the embodiment of the present invention trains and learns the performance indicator data of the business system collected by the data collection system under different flow states through a training prediction system, obtains a performance testing model with a prediction accuracy that meets the requirements, predicts whether the operating status of the business system is normal through the performance testing model, and adjusts the stress test flow value according to the prediction result of the operating status of the business system by the pressure control system, and constructs the stress test flow to act on the business system through the flow construction system until the performance limit state of the business system is determined, thereby realizing that each link of the system performance test, such as business system pressure flow adjustment, business system performance indicator data collection under different flow pressures, performance prediction model training and learning, and estimation of the operating status of the business system, can be automatically executed without human participation, thereby realizing unattended automated business system performance testing, reducing human resource costs, and being able to monitor subtle changes in system performance in a timely manner, and the accuracy of system performance testing and feedback accuracy are greatly improved.

[0037] Other features and advantages of the present invention will be described in the following description, and partly become apparent from the description, or understood by practicing the present invention. The purpose and other advantages of the present invention can be realized and obtained by the structures particularly pointed out in the written description, claims, and drawings.

[0038] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings:

[0040] Figure 1 This is a schematic diagram of the structure of the pressure testing system in the first embodiment of the present invention;

[0041] Figure 2 A specific architecture diagram of the stress testing system in the first embodiment of the present invention;

[0042] Figure 3 This is a flow chart of the performance test model training method in Embodiment 2 of the present invention;

[0043] Figure 4 This is a specific implementation flow chart of the performance test model training method in Embodiment 2 of the present invention;

[0044] Figure 5 This is a principle block diagram of the performance test model training in the second embodiment of the present invention;

[0045] Figure 6 This is a flow chart of the pressurization test method in Embodiment 3 of the present invention;

[0046] Figure 7 This is a specific implementation flow chart of the pressure test method in Embodiment 3 of the present invention;

[0047] Figure 8 This is a use case diagram of an unattended performance test solution in Embodiment 3 of the present invention;

[0048] Fig. 9 This is a logic diagram of unattended performance testing in Embodiment 3 of the present invention;

[0049] Fig.10 Schematic diagram of the principle of the hill climbing algorithm in the third embodiment of the present invention. DETAILED DESCRIPTION

[0050] The exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided to enable a more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.

[0051] In order to solve the problems in the prior art that system stress testing (also known as stress testing) requires manual intervention, is cumbersome to operate, has a low degree of automation and low test accuracy, the embodiment of the present invention provides a stress testing system, which trains a performance testing model based on a supervised training algorithm, determines the operating status of the business system based on the prediction results of the performance testing model, adjusts the stress testing flow value based on the operating status of the business system, constructs a corresponding stress testing flow to act on the business system, obtains the system performance limit state by continuously increasing pressure, realizes unattended performance testing, reduces human resource costs, and can timely monitor subtle changes in system performance, greatly improving the accuracy of system performance testing and feedback precision.

[0052] Embodiment 1

[0053] The pressure testing system provided by the first embodiment of the present invention has a structure as follows: Figure 1 As shown, including:

[0054] The data collection system 1 is used to collect performance indicator data of the business system 5 under different traffic conditions;

[0055] The training prediction system 2 is used to obtain the performance indicator data collected from the business system 5, calibrate the performance indicator data into positive sample data and negative sample data, and then input them into the performance test model for training and learning, so that the performance test model can output the prediction result of the business system operation status;

[0056] The pressure control system 3 is used to adjust the pressure test flow value based on the prediction result of the operation state of the business system 5 output by the performance test model until the performance limit state of the business system 5 is determined;

[0057] The flow construction system 4 is used to construct a corresponding pressure test flow based on the pressure test flow value provided by the pressure control system 3, and to apply the pressure test flow to the business system 5.

[0058] In the above system, during model training, it is necessary to mark positive and negative samples. Through the learning of positive and negative samples, the performance test model can output the prediction results of the business system operation status. During the pressure prediction, it is optional that there is no need to mark positive and negative samples. It is only necessary to input the performance test model according to the collected performance indicator data and output the prediction results of the business system operation status. Then, the pressure control system 3 can adjust the pressure test flow value based on the prediction results of the business system 5 operation status output by the performance test model.

[0059] A specific architecture of the stress testing system is as follows Figure 2 shown.

[0060] The data acquisition system 1 included in the above-mentioned stress testing system can collect performance indicator data generated in real time under different flow states from the business system, thereby obtaining the values ​​of various performance indicators of the business system under different flow states, and can store the obtained performance indicator data in a database (DB), and can also display the collected performance indicator data to users.

[0061] like Figure 2 As shown, the stress test executor can monitor the changes in traffic and performance indicators during the stress test through the acp stress test platform page. The data collection system can collect user behavior data and store it in a database (DB). It can also display the performance indicator data on the acp stress test platform page, for example, display the statistical results of the collected user behavior data.

[0062] The training prediction system 2 included in the above stress testing system is used for training the performance testing model.

[0063] After obtaining the performance indicator data collected from the business system 5, the performance indicator data can be processed and used for training the performance test model. Specifically, the training prediction system 2 is also used to obtain the performance indicator data collected from the business system 5, and after obtaining the performance indicator data collected from the business system, perform normalization processing and / or denoising processing on the performance indicator data; and recombine and vectorize the performance indicators included in the processed performance indicator data to obtain performance indicator data including multi-dimensional vectorized processing.

[0064] The training prediction system performs normalization and / or denoising on the performance indicator data, including:

[0065] Determine a normalized dimension of each performance indicator in the performance indicator data, and convert each performance indicator value in each performance indicator data record into a specified order of magnitude according to the normalized dimension; and / or

[0066] For each performance indicator, the mean and variance of the performance indicator in each performance indicator data record are calculated, and the performance indicator of each performance indicator data record is filtered according to the obtained mean and variance to remove the performance indicator data records whose deviation is greater than the specified condition.

[0067] Among them, the training prediction system recombines and vectorizes the performance indicators included in the processed performance indicator data to obtain multi-dimensional vectorized performance indicator data, including: selecting performance indicators from the performance indicator data for recombining to obtain at least one combined performance indicator; for each performance indicator data record, using the performance indicators included therein and each item in the obtained combined performance indicator as a dimension of vectorization processing to obtain multi-dimensional vectorized performance indicator data.

[0068] Among them, the training prediction system is used to calibrate the performance indicator data into positive sample data and negative sample data, including: the training prediction system marks the performance indicator data records in which each performance indicator in the performance data is within the normal range as positive sample data according to the normal range of each performance indicator when the business system is operating normally, and marks the performance indicator data records in which any performance indicator in the performance indicator data is not within the normal range as negative sample data.

[0069] The training prediction system calibrates the performance indicator data into positive sample data and negative sample data and inputs them into the performance test model for training and learning, so that the performance test model can output the prediction results of the business system operation status, including:

[0070] The training prediction system predicts the business system operating status corresponding to each performance indicator data record based on the performance indicator data records included in the input positive sample data and negative sample data, and counts whether the accuracy of the business system operating status prediction results corresponding to the positive sample data and negative sample data meets the requirements. If it does not meet the requirements, the next batch of performance indicator data is obtained to continue training and learning until a performance test model with a prediction accuracy that meets the requirements is obtained.

[0071] The function of training prediction system can be found in Figure 2 The model training flow shown in the figure, after data collection, goes through sample normalization, sample denoising, single-dimensional sample conversion to multi-dimensional sample, negative sample prediction after supervised classification, and bag rate (i.e. accuracy) calculation to obtain a solidifiable performance training model.

[0072] The pressure control system 3 included in the above-mentioned pressure test system is used to adjust the pressure test flow value based on the prediction result of the business system operation state output by the performance test model until the performance limit state of the business system is determined. For example, the pressure test can be performed based on the hill climbing algorithm, and one of the following methods can be selected:

[0073] Method 1: Continue to increase pressure until the system fails and determines the performance limit state of the business system.

[0074] When the business system operation status prediction result output by the performance test model shows that the business system operation status is normal, the pressure test flow value is increased according to the set pressure increase rules until the business system operation status becomes abnormal, and the performance limit state of the business system is determined.

[0075] Method 2: Determine the performance limit state of the business system through multiple pressurization and depressurization.

[0076] When the business system operation status prediction result output by the performance test model shows that the business system operation status is normal, the pressure test flow value is increased according to the set pressure increase rules until the business system operation status becomes abnormal, and the pressure test flow value is reduced according to the set pressure reduction rules until the business system operation status returns to normal, and the process returns to the step of increasing the pressure test flow value according to the set pressure increase rules. After repeating the above process for a set number of times, the performance limit state of the business system is determined.

[0077] The pressure control system 3 is also used to, after the first abnormality occurs in the operation state of the business system, in the process of reducing the pressure test flow value according to the set decompression rule, if the business system has a performance indicator fluctuation greater than the set threshold, use the mean shift algorithm to determine the distance offset between the performance indicator value corresponding to the abnormal point where the performance indicator fluctuation is greater than the set threshold and the performance indicator mean, if the distance offset increases, it is considered that the system has reached the performance limit. This realizes the abnormal handling of the situation where the system performance indicator fluctuates greatly during the pressurization and decompression process.

[0078] The function of the pressure control system 3 is shown in Figure 2 The unattended intelligent stress testing flow shown in the figure includes normalization of monitoring indicators, model prediction after vectorization of indicators, flow adjustment according to the prediction results, flow reduction processing when the mean drifts significantly, and calling the anomaly detection module to implement anomaly detection and other processes. Stress testing is completed intelligently through models and algorithms, and unattended stress testing is achieved. Compared with the previous stress testing that required multiple people to manually monitor, the efficiency is greatly improved. The calling of the anomaly detection module can perform anomaly detection based on historical mean, sample normalization to calculate Mahalanobis distance, and anomaly detection algorithm based on kmeans algorithm to implement anomaly detection.

[0079] The flow construction system 4 included in the above-mentioned pressure test system includes: a data intelligent construction system and a simulation scheduling system; wherein,

[0080] A data intelligent construction system is used to construct a corresponding number of user behavior requests that meet the requirements of the business system based on the pressure test flow value determined by the pressure flow determination system;

[0081] The simulation scheduling system is used to simulate the corresponding user behavior acting on the business system based on the user behavior requests constructed by the data intelligence construction system.

[0082] The simulation scheduling system may include a scheduling subsystem and a stress testing server cluster;

[0083] The scheduling subsystem is used to allocate a server in the stress testing server cluster to each user behavior included in the user behavior data according to a preset user behavior simulation strategy and a stress testing server allocation strategy;

[0084] Each server included in the stress testing server cluster is used to simulate the assigned user behavior acting on the business system.

[0085] The function of flow structure system 4 can be found in Figure 2 The data intelligent construction system and simulation scheduling system can construct the real data of users, such as the products displayed on the website, build user cookies and other related information, and simulate user behavior through the server cluster.

[0086] Among them, the data intelligent construction system abandons the drawbacks of manual handwriting data construction in the past. The use of big data learning can greatly improve the data coverage and cover more engine logic. The data intelligent construction system can construct user behavior requests through corpus, word segmentation, Bayesian analysis, word frequency statistics, word cloud creation, and weighted hotspot requests (queries). It can also learn and fit online requests from massive data based on semantics and data mining, and can also obtain hundreds of millions of online real user request data through map-reduce offline tasks. And perform persistent storage, and store the data in the HDFS storage system built with dozens of 10,000-gigabyte machines.

[0087] The simulation scheduling system can actively pull data from the storage system and test the server cluster ( Figure 2 A group of server charts in the diagram section) simulates user behavior to simulate the corresponding amount of stress test traffic. The simulated server cluster can simulate user behavior based on the collected user behavior statistics, such as the need to simulate the behavior of tens of millions of users. The scheduling system can simulate the corresponding number of user behaviors based on the collected user behavior data statistics, such as the current traffic of 50 million. When performing simulations, simulations are performed based on certain rules and strategies, such as: same-computer room priority strategy, greedy strategy, load balancing strategy, etc. Taking the same-computer room priority strategy as an example, for users in City A, the corresponding servers in City A are used for simulation.

[0088] The simulation scheduling system can simulate user behavior through real servers, such as the scheduling subsystem ( Figure 2 The square part with M in the middle can realize the scheduling of simulated traffic and write the scheduling information to zk. The user behavior simulation strategies used by the scheduling subsystem can include the same-computer room priority strategy, greedy strategy, load balancing strategy, etc. The stress test server allocation strategy used by the scheduling subsystem can include allocation strategies such as intelligent allocation of stress test models according to different stress test types. The scheduling system also supports automatic loading of various scheduling algorithm plug-ins (plug in). The stress test server cluster can also obtain tasks through the agent.

[0089] The stress testing system provided by the embodiment of the present invention is an unattended stress testing technology based on supervised training algorithm regulation. It implements automated system performance testing based on a pre-trained performance testing model, automatically adjusts the user flow applied to the system under test through a hill climbing algorithm, continuously increases system pressure, and obtains the system limit state. The performance testing method can be implemented through a performance testing platform (ACP). The above stress testing refers to system stress testing and performance testing, and the performance state of the test system under different pressures. The specific implementation processes of the training and stress testing of the performance testing model are described below through specific embodiments.

[0090] Embodiment 2

[0091] Embodiment 2 of the present invention provides a performance test model training method, the process of which is as follows: Figure 3 As shown, the following steps are included:

[0092] Step S101: Acquire performance indicator data of a business system as sample data.

[0093] The performance indicator data of the business system under different traffic flows are obtained as sample data. The performance indicator data includes the performance indicator values ​​of the business system corresponding to the system under different user behavior traffic flows.

[0094] Collect performance indicator data generated during system operation to obtain performance indicator data samples in the system environment, such as system feedback time (rt), number of requests per second (qps), timeout rate (timeout), request success rate (succrate) between modules, bandwidth (bytes) and other performance indicators under different user behavior traffic (i.e. user traffic).

[0095] Step S102: classify the sample data.

[0096] To classify sample data, you can label the sample data based on set rules to mark positive samples and negative samples.

[0097] For example, a performance indicator (i.e., a dimension) of a certain sample record: if the feedback time exceeds a certain threshold, it is marked as a negative sample; if the success rate drops below a certain threshold, it is marked as a negative sample; if all indicators are within the normal range, they are marked as positive samples. For example, the sample record with an rt exceeding 200ms is marked as a negative sample, that is, marked: FLAG=-1; the sample record with an rt within 100-200ms is marked as a positive sample, that is, marked FLAG=1.

[0098] Step S103: input the classified sample data into the performance test model for training and learning, and output the prediction result of the business system operation status.

[0099] After the sample data is constructed, modeling operations are performed based on the sample data. The algorithm library can be called for modeling, such as the algorithm library of tensorflow and the algorithm library of random forest implemented based on python. Then the sample data is input into the model, and the business system operation status prediction results are predicted based on the sample data, such as whether the business system operation status prediction results output by the positive sample data are normal, and whether the business system operation status prediction results output by the negative sample data are abnormal. The accuracy of the prediction results will be determined based on the algorithm library.

[0100] Step S104: Determine whether the prediction accuracy meets the requirements. If yes, execute step S106; if no, execute step S105.

[0101] The algorithm library generates a binary index / binary tree storage structure based on the input parameters and samples, and verifies the output model with various samples, calculates the model evaluation index (area under the curve, AUC), and obtains a model whose AUC meets the requirements. AUC can be the bag rate or accuracy rate, and AUC is used to determine whether the accuracy of the model prediction meets the requirements.

[0102] For example, for the classified sample data, in a sample data set with m positive samples and n negative samples, there are a total of m*n pairs of samples. The number of the probability of accurate prediction of positive samples and the probability of accurate prediction of negative samples in the m*n pairs of samples can be counted as the AUC of positive samples and negative samples respectively. Here, the algorithm library function can be called: model = train (sampleList) sampleList is the sample list, and Model is the returned binary model. The AUC is calculated by calling the algorithm library function.

[0103] Step S105: adjust the parameters of the performance test model, obtain the next batch of sample data and return to step S102 to continue training and learning.

[0104] If the AUC of the model does not meet the requirements, it is necessary to continue iterative training. Through multiple iterative training of multiple batches of samples, a model whose AUC meets the requirements is obtained.

[0105] Step S106: solidify and store the performance test model.

[0106] The performance test model is trained cyclically through multiple batches of samples until the accuracy meets the requirements, and then the performance test model is solidified and stored.

[0107] The performance testing model construction method provided in the embodiment of the present invention performs learning and training based on the historical performance indicator data of the system to obtain a solidifiable performance testing model. By learning from multiple batches of samples and adjusting the model parameters, a performance testing model with a system performance indicator prediction accuracy that meets the requirements is obtained, which is used for system performance testing to achieve unattended automated system performance testing.

[0108] One implementation of the training of the above performance test model is as follows Figure 4 As shown, its principle block diagram can be found in Figure 5 As shown, the following steps are included:

[0109] Step S201: Acquire system performance indicator data as sample data.

[0110] See also Figure 5 As shown, the data collection pipeline (batch reading hbase) collects user behavior data through data collection devices and stores it in the database. When needed, it can be read in batches from the database (DB). The data collection task can be monitored online.

[0111] Step S202: performing denoising processing on the sample data.

[0112] Optionally, after obtaining the sample data, you can filter the sample data to filter out abnormal sample data, which is also called sample denoising, such as removing performance indicator data of system jitter, removing -1 data, etc. Figure 5 The sample denoising pipeline shown in Figure 1 can eliminate abnormal and invalid samples.

[0113] Sample data denoising can be performed using certain rules, such as filtering out data with large deviations through certain algorithms. For example, for each system performance indicator, the mean and variance of the system performance indicator in each sample record are calculated, and the system performance indicator of each sample record is filtered based on the obtained mean and variance, and sample records with deviations greater than the specified conditions are removed.

[0114] Specifically, we can traverse each performance indicator of each sample data, calculate the mean and variance of each performance indicator, filter out samples with large deviations, exclude them from model training, and eliminate the influence of abnormal data. For example, if the normal feedback time rt is about 100 milliseconds, we filter sample data with feedback time less than 10 milliseconds and greater than 500 milliseconds, because these data cannot truly reflect the system status.

[0115] Step S203: normalize the sample data.

[0116] Normalization is to normalize the dimensions of the sample, determine the normalized dimensions of each systematic performance indicator in the sample data, and convert the values ​​of each systematic performance indicator in each sample record to the specified order of magnitude according to the normalized dimensions. This step achieves a unified sample data interval, see Figure 5 The sample normalization pipeline shown in the figure realizes the normalization of dimensions and unifies the data interval.

[0117] For example, the bandwidth is in the billions, while the feedback time is in milliseconds. These values ​​need to be unified to one order of magnitude to ensure that the value ranges of each dimension of each sample data are similar. For example, the absolute value of the bandwidth is 1500000B, and the absolute value of the feedback time is 0.1s. The normalized values ​​are the absolute value of the bandwidth: 1500M, and the absolute value of the feedback time is 100ms. At this time, the two values ​​are of similar order of magnitude.

[0118] The denoising process in step S102 and the normalization process in step S103 are both optional steps. Both processes may be performed, or only one of them may be performed. When both processes are performed, the execution order is not particular, and they may be performed simultaneously.

[0119] Step S204: regrouping the performance indicators included in the processed sample data.

[0120] Select system performance indicators from the sample data and recombine them to obtain at least one combined performance indicator. This process can be called sample transformation. Sample transformation can combine new sample dimensions to obtain combined sample data; see Figure 5 The single sample is converted into a multi-dimensional sample pipeline, for example, the feedback time rt and the number of requests per second qps can be combined, or the bandwidth and the success rate can be combined.

[0121] For example, the service response time and return bandwidth are combined into a new one-dimensional sample. When the service response time is large, the return bandwidth is also large, so the two are strongly correlated. The two are integrated into one sample through calculation. For example, if the response time is 100ms and the return bandwidth is 100M, a new one-dimensional sample sample1=100ms / 100M=1ms / M is constructed. If the sample deviates significantly from this estimated value, it means that the system is abnormal.

[0122] This step combines the data of existing dimensions in the existing sample data to generate new combined dimension data.

[0123] Step S205: performing vectorization processing on the reassembled sample data to obtain multi-dimensional vectorized sample data.

[0124] For each sample record, each item in the system performance index and the obtained combined performance index is used as a dimension for vectorization processing to obtain the sample data after multi-bit quantization processing. This step realizes the vectorization conversion of each dimension data in the sample data to obtain the serialized vector expression of the sample data. For example, in the serialized vector expression, each performance index, including the combined performance index and the single performance index, corresponds to a dimension data in the vector. Figure 5 The sample vector serialization pipeline shown in , for example, serializes more than a hundred dimensional samples into vectors.

[0125] Step S206: classify the sample data.

[0126] See also Figure 5 The supervised classification pipeline shown in , classifies samples into positive samples and negative samples through rules.

[0127] Step S207: input the classified sample data into the performance test model for training and learning, and output the prediction result of the business system operation status.

[0128] See also Figure 5 As shown in the figure, the core model training pipeline and model training flow. Billions of sample data are fed into the model, and a performance prediction model based on dnn / rrnn / random forest is generated. At the same time, an entity index file is generated, the model is used for testing, and the negative sample prediction pipeline is used for prediction.

[0129] Step S208: Determine whether the prediction accuracy meets the requirements. If yes, execute step S210; if no, execute step S209.

[0130] See also Figure 5 The calculation auc / bag rate calculation pipeline shown in Calculate the prediction accuracy of the model.

[0131] Step S209: adjust the parameters of the performance test model, obtain the next batch of sample data and return to step S202 to continue training and learning.

[0132] See also Figure 5 As shown, if auc fails, the parameters are adjusted and the next batch of sample training is continued.

[0133] Step S210: solidify and store the performance test model.

[0134] This step implements model persistence processing. For the corresponding model that meets the expectations of AUC, it will be persistently stored, and the binary model will be stored in the distributed system HDFS, and real-time estimation will be performed in the unattended stress testing flow.

[0135] See also Figure 5 As shown, if auc is qualified, it will enter the model dump stage to solidify the model.

[0136] Embodiment 3

[0137] Embodiment 3 of the present invention provides a pressure test method, the process of which is as follows: Figure 6 As shown, the following steps are included:

[0138] Step S301: adjusting the stress test flow value applied to the business system according to the set rules.

[0139] The pressure test flow value applied to the business system is adjusted through the pressure control system, and the user behavior is simulated through the flow construction system to achieve flow pressurization of the tested business system. The degree of pressurization can be controlled according to predetermined rules.

[0140] Step S302: construct corresponding stress test traffic based on the stress test traffic value, and apply the stress test traffic to the business system.

[0141] Step S303: Obtain performance indicator data of the business system under different traffic conditions.

[0142] Step S304: The performance test model predicts whether the business system operation status is normal based on the acquired performance indicator data.

[0143] If yes, go to step S305; if no, go to step S306.

[0144] Step S305: Increase the user behavior traffic according to the set rules, and return to continue executing step S302.

[0145] Step S306: Determine the performance limit state of the system.

[0146] This embodiment is implemented based on a hill climbing algorithm to find the system performance limit state, that is, when the system performance parameters are normal, the system flow pressure is continuously increased until the system performance becomes abnormal, thereby determining the system performance limit state.

[0147] In this embodiment, the automatic limit detection technology of the hill climbing algorithm is used to implement system performance testing. The hill climbing algorithm is applied and the model is called in real time to estimate the current system state. When the model feedback is normal, the hill climbing and pressurization operation continues. When the model feedback is abnormal, the peak is reached.

[0148] When performance indicators show abnormal drift, the anomaly detection technology of the drift algorithm is used to handle it. That is, when the system performance indicators fluctuate greatly at the peak value, the mean drift algorithm is used to filter out the anomalies to ensure the normal stability of the unattended stress testing flow.

[0149] The performance testing method provided in the embodiment of the present invention simulates the system performance through a pre-trained performance testing model, and can control the user behavior flow applied to the business system, change the pressure on the system, test whether the system performance indicators of the business system under different pressures can remain stable and normal, and obtain the extreme state of abnormal performance indicators of the system, thereby realizing automated testing of system performance without human intervention, and can realize automated system stress and performance testing without manual supervision, thus reducing human resource costs, and can timely monitor subtle changes in system performance, and the accuracy of system performance testing and feedback precision are greatly improved.

[0150] A specific implementation process of the above pressure test method is as follows: Figure 7 As shown, the following steps are included:

[0151] Step S401: Adjust the stress test flow value applied to the business system according to the set rules.

[0152] Step S302: construct corresponding stress test traffic based on the stress test traffic value, and apply the stress test traffic to the business system.

[0153] Step S403: Obtain performance indicator data of the business system under different traffic conditions.

[0154] Step S404: performing normalization processing and / or denoising processing on the system performance prediction data.

[0155] Step S405: recombining and vectorizing the performance indicators included in the processed performance indicator data to obtain multi-dimensional vectorized performance indicator data.

[0156] Step S406: The performance test model predicts whether the business system operation status is normal based on the acquired performance indicator data.

[0157] If yes, go to step S407; if no, go to step S408.

[0158] Step S407: Increase the user behavior traffic according to the set rules, and return to continue executing step S402.

[0159] Step S408: Check whether the number of repetitions reaches the set threshold; if not, execute step S409; if yes, execute step S410.

[0160] Step S409: Reduce user behavior traffic according to the set rules, and return to continue executing step S402.

[0161] Step S410: Determine the performance limit state of the system.

[0162] The loop process executed by the above steps S408 and 409 realizes that after any system performance indicator is abnormal, the user behavior traffic applied to the business system is reduced according to the set rules until the system performance indicator returns to the normal range, and the user behavior traffic applied to the business system continues to be increased according to the set rules until any system performance indicator is abnormal. After repeating the above process for a set number of times, the performance limit state of the business system is determined.

[0163] like Figure 8 The figure shows a scenario use case of unattended performance testing, which detects the system limit state based on the automatic limit exploration solution. Users (such as users) can trigger test tasks on the ACP platform - unattended stress testing services based on supervised training algorithms. The service will call the hill climbing algorithm and performance testing model to implement stress testing. The overall stress testing process uses the hill climbing algorithm, as shown in Figure 1.1, and calls the climbing algorithm, as shown in Figure 1.2, calling the performance testing model (estimation model). When there is an anomaly, the mean shift algorithm is used to detect the anomaly point, as shown in Figure 1.3, calling the mean shift algorithm. The test process finally obtains the system limit, as shown in Figure 1.4, obtaining the system limit of the stress tested service.

[0164] In this embodiment, the unattended performance test logic is as follows: Fig. 9As shown in the figure, the overall stress test process is scheduled using the hill climbing algorithm. The performance test model is used to estimate whether the current system performance indicators are normal. If the system performance indicators are normal, the hill climbing algorithm is used to execute the hill climbing process. If the system performance indicators are abnormal, the kmeans algorithm and mean shift algorithm are called to calculate the spatial distance of the abnormal points. The hill climbing algorithm executes the downhill process and calls the performance test model to estimate whether the current system performance indicators are normal. When the system returns to normal, the hill climbing process continues. If the system is still abnormal, it is considered that the system limit has been reached and the task is completed. In the entire test logic, the hill climbing algorithm is called and the current system state is estimated through the model. After climbing to the top of the first mountain, the pressure is reduced and then the hill climbing operation is performed to increase the pressure. When the mountain top is reached again, the model prediction reaches the limit, which is considered to be the system limit, and the task is completed at this time.

[0165] Use the automatic limit detection technology based on the hill climbing algorithm to control the user traffic pressure applied in the test system. Apply the hill climbing algorithm and call the python algorithm library in real time to load the above trained binary model. Convert the real-time monitoring data of the current system into a sample format through the above conversion. Call the model prediction algorithm of the algorithm library and use the sample format data as input parameters to get the model's estimated prediction results. When the model feedback is normal, continue the hill climbing and pressurization operation. When the model feedback is abnormal, it means that some system performance indicators are abnormal and the hill climbing and pressurization operation can no longer be continued. This point is the vertex. The implementation process of the hill climbing algorithm is as follows: Fig.10 As shown in the figure, the horizontal axis is time and the vertical axis is pressure (qps). The system limit is measured by controlling the pressure (qps), which is divided into a warm-up phase, a climbing phase, and a small adjustment phase after reaching the model prediction limit (i.e., a small descent after the local optimal solution, a small climbing phase). The kmeans algorithm is used to ensure that there is no large average deviation until the optimal solution, i.e., the system limit, is reached and the stress test stops. For example, when stress testing a system, the pressure starts at 10,000qps, and the pressure step is 5,000qps every five minutes. Before each pressure increase, the model is used to estimate the current state of the system. When the model feedback system is normal, the pressure is increased by 5,000qps and the climbing mode is turned on. When the model feedback system is abnormal, the downhill mode is entered, i.e., 5,000qps is reduced. That is, the local optimal solution is obtained. After the system returns to normal, the climbing mode is continued. When the model feedback system is abnormal again, the point is determined to be the top of the mountain, i.e., the system limit, and the pressure is stopped, and the stress test ends.

[0166] Optionally, anomaly detection technology of drift algorithm can be used to detect anomalies. After any system performance indicator data prediction value shows an anomaly for the first time, in the process of reducing or increasing user behavior traffic included in the system performance prediction data according to set rules, if the system performance indicator fluctuates greatly, the mean shift algorithm is used to determine the distance offset between the system performance indicator value corresponding to the anomaly point where the performance indicator fluctuates greatly and the mean of the system performance indicator. If the distance offset increases, it is considered that the system has reached its performance limit.

[0167] Specifically, when the system is at its peak, the system indicators fluctuate greatly, and the model-estimated AUC decreases, indicating that the estimation accuracy decreases. The mean shift algorithm is used to check whether the current system is normal and to ensure that the unattended stress test flow is normal and stable. For example, if the current system response time, bandwidth, and timeout rate all fluctuate greatly, the model estimation may be inaccurate at this time. Therefore, the mean shift algorithm is used at the top of the system to calculate the outliers, that is, the spatial distance between the abnormal points and the mean points. If the distance increases, that is, the indicator deviates greatly, it means that the current system indicators have fluctuated greatly compared with the historical trend and have reached the limit.

[0168] The above-mentioned system and method provided by the embodiment of the present invention provide a modeling technology based on supervised learning and a stress testing process based on algorithm regulation, which samples and characterizes the performance indicator data and then conducts a supervised training modeling process and an algorithm-regulated performance stress testing process, achieving single-link stress testing and unattended full-link stress testing, and as the number of samples increases, the estimation accuracy will be greatly improved. In the modeling process, services such as tensorflow can also be used to build large-scale dnn and rnn models, so that the model estimation is more accurate and the feedback is more precise, and at the same time, minor system anomalies can be discovered in a timely manner. More outlier detection algorithms and flow control algorithms can also be used.

[0169] Unless otherwise specifically stated, terms such as processing, computing, calculating, determining, displaying, etc. may refer to the actions and / or processes of one or more processing or computing systems, or similar devices, which operate and convert data represented as physical (e.g., electronic) quantities within registers or memories of a processing system into other data similarly represented as physical quantities within memories, registers, or other such information storage, transmission, or display devices of the processing system. Information and signals may be represented using any of a variety of different techniques and methods. For example, data, instructions, commands, information, signals, bits, symbols, and chips mentioned throughout the above description may be represented by voltages, currents, electromagnetic waves, magnetic fields or particles, light fields or particles, or any combination thereof.

[0170] It should be understood that the specific order or hierarchy of steps in the disclosed process is an example of an exemplary method. Based on design preferences, it should be understood that the specific order or hierarchy of steps in the process can be rearranged without departing from the scope of protection of the present disclosure. The attached method claims present the elements of the various steps in an exemplary order and are not intended to be limited to the specific order or hierarchy described.

[0171] In the above detailed description, various features are grouped together in a single embodiment to simplify the disclosure. This method of disclosure should not be interpreted as reflecting an intention that the embodiments of the claimed subject matter require more features than are clearly stated in each claim. On the contrary, as reflected in the appended claims, the invention is in a state of having less than all the features of the disclosed individual embodiments. Therefore, the appended claims are hereby expressly incorporated into the detailed description, with each claim standing on its own as a separate preferred embodiment of the invention.

[0172] Those skilled in the art will also appreciate that the various illustrative logic blocks, modules, circuits, and algorithmic steps described in conjunction with the embodiments herein can all be implemented as electronic hardware, computer software, or a combination thereof. In order to clearly illustrate the interchangeability between hardware and software, various illustrative components, blocks, modules, circuits, and steps are generally described above around their functions. Whether such functions are implemented as hardware or software depends on specific applications and the design constraints imposed on the entire system. A skilled person can implement the described functions in an alternative manner for each specific application, but such implementation decisions should not be interpreted as departing from the scope of protection of the present disclosure.

[0173] The steps of the method or algorithm described in conjunction with the embodiments herein may be directly embodied as hardware, a software module executed by a processor, or a combination thereof. The software module may be located in a RAM memory, a flash memory, a ROM memory, an EPROM memory, an EEPROM memory, a register, a hard disk, a mobile disk, a CD-ROM, or any other form of storage medium well known in the art. An exemplary storage medium is connected to the processor so that the processor can read information from the storage medium and can write information to the storage medium. Of course, the storage medium may also be an integral part of the processor. The processor and the storage medium may be located in an ASIC. The ASIC may be located in a user terminal. Of course, the processor and the storage medium may also be present in a user terminal as discrete components.

[0174] For software implementation, the techniques described in this application can be implemented with modules (e.g., procedures, functions, etc.) that perform the functions described in this application. These software codes can be stored in a memory unit and executed by a processor. The memory unit can be implemented within the processor or outside the processor. In the latter case, it is coupled to the processor in a communication manner via various means, which are well known in the art.

[0175] The above description includes examples of one or more embodiments. Of course, it is impossible to describe all possible combinations of components or methods for the purpose of describing the above embodiments, but it should be recognized by those skilled in the art that the various embodiments may be further combined and arranged. Therefore, the embodiments described herein are intended to cover all such changes, modifications and variations that fall within the scope of protection of the appended claims. In addition, with respect to the term "comprising" used in the specification or claims, the word is covered in a manner similar to the term "including", just as "including," is explained as a transitional word in the claims. In addition, any term "or" used in the specification of the claims is intended to mean "non-exclusive or".

Claims

1. A pressure testing system, characterized in that: include: Data collection system, used to collect performance indicator data of business systems under different traffic conditions; A training prediction system is used to obtain performance indicator data collected from the business system, calibrate the performance indicator data into positive sample data and negative sample data, and then input them into the performance test model for training and learning, so that the performance test model can output the prediction result of the business system operation status; A pressure control system is used to increase the pressure test flow value according to the set pressure increase rules if the business system operation status prediction result output by the performance test model is that the business system operation status is normal, until the business system operation status becomes abnormal, so as to determine the performance limit state of the business system; or If the business system operation status prediction result output by the performance test model is that the business system operation status is normal, the pressure test flow value is increased according to the set pressure increase rule until the business system operation status becomes abnormal, and the pressure test flow value is reduced according to the set pressure reduction rule until the business system operation status returns to normal, and the step of increasing the pressure test flow value according to the set pressure increase rule is returned. After repeating the above process for a set number of times, the performance limit state of the business system is determined; A flow construction system is used to construct a corresponding stress test flow based on the stress test flow value, and to apply the stress test flow to the business system.

2. The system according to claim 1, characterized in that The traffic construction system includes: a data intelligent construction system and a simulation scheduling system; The data intelligent construction system is used to construct a corresponding number of user behavior requests that meet the requirements of the business system based on the pressure test flow value determined by the pressure flow determination system; The simulation scheduling system is used to simulate the corresponding user behavior acting on the business system according to the user behavior request constructed by the data intelligent construction system.

3. The system according to claim 2, characterized in that The simulation scheduling system includes a scheduling subsystem and a stress testing server cluster; The scheduling subsystem is used to allocate a server in the stress testing server cluster to each user behavior included in the user behavior data according to a preset user behavior simulation strategy and a stress testing server allocation strategy; Each server included in the stress testing server cluster is used to simulate the assigned user behavior to act on the business system.

4. The system according to claim 1, characterized in that The training prediction system is also used for: After acquiring the performance indicator data collected from the business system, performing normalization processing and / or denoising processing on the performance indicator data; The performance indicators included in the processed performance indicator data are recombined and vectorized to obtain performance indicator data including multi-dimensional vectorized processing.

5. The system according to claim 4, characterized in that The training prediction system performs normalization and denoising processing on the performance indicator data, including: Determine a normalized dimension of each performance indicator in the performance indicator data, and convert each performance indicator value in each performance indicator data record into a specified order of magnitude according to the normalized dimension; and / or For each performance indicator, the mean and variance of the performance indicator in each performance indicator data record are calculated, and the performance indicator of each performance indicator data record is filtered according to the obtained mean and variance to remove performance indicator data records whose deviation is greater than the specified condition.

6. The system according to claim 4, characterized in that The training prediction system recombines and vectorizes the performance indicators included in the processed performance indicator data to obtain multi-dimensional vectorized performance indicator data, including: Selecting performance indicators from the performance indicator data and recombining them to obtain at least one combined performance indicator; For each performance indicator data record, each item of the performance indicator included therein and the obtained combined performance indicator is used as a dimension of vectorization processing to obtain the performance indicator data after multi-dimensional vectorization processing.

7. The system according to claim 1, characterized in that The training prediction system is used to calibrate the performance indicator data into positive sample data and negative sample data, including: The training prediction system marks the performance indicator data records in which all performance indicators in the performance indicator data are within the normal range as positive sample data, and marks the performance indicator data records in which any performance indicator in the performance indicator data is not within the normal range as negative sample data, based on the normal range of each performance indicator when the business system is operating normally.

8. The system of claim 1, wherein: The training prediction system calibrates the performance indicator data into positive sample data and negative sample data and inputs them into the performance test model for training and learning, so that the performance test model can output the prediction result of the business system operation status, including: The training prediction system predicts the business system operating status corresponding to each performance indicator data record based on the performance indicator data records included in the input positive sample data and negative sample data, and counts whether the accuracy of the business system operating status prediction results corresponding to the positive sample data and negative sample data meets the requirements. If it does not meet the requirements, the next batch of performance indicator data is obtained to continue training and learning until a performance test model with a prediction accuracy that meets the requirements is obtained.

9. The system according to claim 1, characterized in that The pressure control system is also used for: After the first abnormality occurs in the operation status of the business system, in the process of reducing the pressure test flow value according to the set decompression rules, if the performance indicator fluctuation of the business system is greater than the set threshold, the mean shift algorithm is used to determine the distance offset between the performance indicator value corresponding to the abnormal point where the performance indicator fluctuation is greater than the set threshold and the performance indicator mean. If the distance offset increases, it is considered that the system has reached its performance limit.

Images