A system stress testing method based on machine deep learning self-optimization model

Abstract

The present invention relates to a system stress testing method based on a machine deep learning self-optimization model, which specifically includes the following steps: S1, obtaining historical production log information, as model training log information, and inputting it into a stress testing model based on a bidirectional long short-term memory recurrent neural network; S2, comparing the prediction results of the stress testing model with the production log information of the real system, and calculating the accuracy of the stress testing model; S3, judging whether the accuracy is greater than a preset threshold, if so, go to step S5, otherwise go to step S4; S4, adjust the model parameters of the stress testing model, and go to step S1; S5, stress test the system according to the stress testing model corresponding to the current prediction result, and make corresponding program modifications based on the obtained error information and finally go online. Compared with the prior art, the present invention has the advantages of realizing the prediction of uncertain future trends based on the evolution of basic data, improving the accuracy of stress testing and testing efficiency.

Description

Technical Field

[0001] The present invention relates to the technical field of system stress testing, and in particular to a system stress testing method based on a machine deep learning self-optimization model. Background Art

[0002] With the rapid development of the securities industry in recent years, the stable operation of trading systems, the precise location and automatic isolation of faults, and the rapid self-recovery capabilities have become increasingly important. Log analysis is a fundamental method for achieving these goals. Traditional rule-based log analysis methods suffer from high rates of false positives and false negatives, and are also inefficient when analyzing massive log volumes. This becomes particularly complex in large-scale systems with numerous independent processes running concurrently, and when log messages from different processes are intertwined.

[0003] For this reason, new systems need to be stress-tested to ensure they don't crash or crash under the heavy client traffic and operations. Stress testing involves setting up a test environment similar to the actual production environment and using a test program to send a specified number of requests to the system simultaneously or within a very short period of time. This is done to test the system's performance under varying stress conditions and the maximum stress it can withstand. The goal is to continuously stress the system and observe its performance, thereby understanding its performance limits and flaws. Currently, the simplest method is the traditional randomized algorithm, which extracts the frequency of occurrence of different instructions, but its accuracy is only around 10%.

[0004] The solution to the above-mentioned phenomenon is to introduce a log analysis model based on machine deep learning to simulate the actual instruction sending situation in real production as realistically and accurately as possible, so as to more realistically reflect the requirements of the actual scenario in the stress testing phase, provide data model support for the subsequent stress test case arrangement, and achieve the goal of improving system security, stability, and agility.

[0005] Deep learning is a type of machine learning, originating from artificial neural networks. It uses neural networks modeled after the human brain to analyze, learn, and interpret data. By attempting to mimic the way neurons in the brain transmit and process information, deep learning can combine multiple low-level features to form more abstract high-level features, enabling it to better learn the inherent patterns of sample data and represent hierarchical structures. The ARIMA autoregressive moving average algorithm was one of the most advanced algorithms before the advent of machine learning. Many business models and Excel forecasts utilize this model, but its accuracy is only around 46%. Summary of the Invention

[0006] The purpose of the present invention is to overcome the defects of the above-mentioned prior art and provide a system stress testing method based on a machine deep learning self-optimization model, which is suitable for learning tasks with large data volumes and complex but deterministic patterns. By utilizing a bidirectional long short-term memory recurrent neural network (Bi-directional LSTM RNN) model, log data within a given period of time is trained, and prediction data is automatically generated through an algorithm model. The cycle of calculation-generation of predicted new data-recalculation of new data is then repeated to improve the accuracy of model prediction.

[0007] The purpose of the present invention can be achieved by the following technical solutions:

[0008] A system stress testing method based on a machine deep learning self-optimization model specifically includes the following steps:

[0009] S1. Obtain historical production log information as model training log information and input it into a stress testing model based on a bidirectional long short-term memory recurrent neural network;

[0010] S2. Compare the prediction results of the stress test model with the production log information of the real system to calculate the accuracy of the stress test model;

[0011] S3, determine whether the accuracy is greater than a preset threshold, if so, go to step S5, otherwise go to step S4;

[0012] S4, adjust the model parameters of the stress test model and go to step S1;

[0013] S5. Perform stress testing on the system based on the stress testing model corresponding to the current prediction result, and make corresponding program modifications based on the error information obtained before finally launching the system.

[0014] The basic idea of ​​a bidirectional recurrent neural network is to construct two recurrent neural networks (RNNs) for each training sequence, one for the forward and one for the backward, connected to an output layer. This architecture provides the output layer with complete past and future contextual information for every point in the input sequence. Six unique weights are reused at each time step, corresponding to the input to the forward and backward hidden layers, the hidden layers to themselves, and the forward and backward hidden layers to the output layer. It is important to note that there is no information flow between the forward and backward hidden layers.

[0015] A key advantage of recurrent neural networks (RNNs) is their ability to leverage contextual information when mapping input and output sequences. However, standard RNNs are limited in the scope of contextual information they can access. This problem causes the influence of hidden layer inputs on network outputs to diminish as the network loops recur. Therefore, to address this issue, the long short-term memory (LSTM) architecture was developed. Rather than being a type of RNN, LSTM is more like an enhanced component incorporated into an RNN. Specifically, the small circles in the hidden layer of an RNN are replaced with LSTM modules.

[0016] The stress testing model is provided with a module using a bidirectional long short-term memory recurrent neural network method.

[0017] Furthermore, the module using the bidirectional long short-term memory recurrent neural network method includes an input end, an output end, a forgetting end and a relay end.

[0018] Furthermore, the workflow of the module using the bidirectional long short-term memory recurrent neural network method includes a forward calculation stage and a backward calculation stage.

[0019] The formula for the input end of the module using the bidirectional long short-term memory recurrent neural network method in the forward calculation stage is as follows:

[0020]

[0021] in, is the i-th input value of the external input at the input terminal at time t, w iτ is the weight of the i-th input value, is the hth output value of the hidden unit at the input end at time t-1, w hτ is the weight of the h-th output value, is the cth output value of the input terminal at the relay terminal at time t-1, w cτ is the weight of the c-th output value, f() is the activation function, is the output value of the input terminal at time t, I is the total amount of external input data, H is the total amount of data of the hidden unit, and C is the total amount of data at the relay terminal.

[0022] The formula for the forgetting end of the module using the bidirectional long short-term memory recurrent neural network method in the forward calculation phase is as follows:

[0023]

[0024] Among them, w iφ is the weight of the i-th input value of the external input of the forgetting end, w hφis the weight of the h-th output value of the hidden unit at the forgetting end, w cφ is the weight of the c-th output value of the forgetting end at the relay end, is the output value of the forget terminal at time t.

[0025] The formula of the relay end of the module using the bidirectional long short-term memory recurrent neural network method in the forward calculation stage is as follows:

[0026]

[0027] in, is the output value of the forget terminal at time t, is the cth output value of the relay terminal at time t-1, w ic is the weight of the i-th input value of the external input of the relay end, w hc is the weight of the h-th output value of the hidden unit at the relay end, is the output value of the input at time t, and g() is the activation function.

[0028] The formula for the output of the module using the bidirectional long short-term memory recurrent neural network method in the forward calculation phase is as follows:

[0029]

[0030] Among them, w iw is the weight of the i-th input value of the external input at the output end, w hw is the weight of the h-th output value of the output hidden unit, is the cth output value of the output terminal at the relay end at time t, w cw is the weight of the c-th output value of the output terminal at the relay end, is the output value of the output terminal at time t;

[0031] The formula for the output value of the relay end is as follows:

[0032]

[0033] in, is the output value of the relay terminal at time t, and h() is the activation function.

[0034] The formula for the input end of the module using the bidirectional long short-term memory recurrent neural network method in the backward calculation stage is as follows:

[0035]

[0036] in, is the output value of the input terminal at time t, f′() is the activation function;

[0037] The formula for the forget side is as follows:

[0038]

[0039] in, is the output value of the forget terminal at time t;

[0040] The formula on the relay side is as follows:

[0041]

[0042] in, is the output value of the relay terminal at time t, is the input value of the relay end at time t, is the input value of the relay end at time t+1, is the kth output value of the non-hidden unit at the relay end at time t, w ck is the weight of the k-th output value of the relay terminal, is the hth output value of the hidden unit at the relay end at time t+1, w ch is the weight of the h-th output value of the relay end.

[0043] The formula for the output of the module using the bidirectional long short-term memory recurrent neural network method in the backward calculation stage is as follows:

[0044]

[0045] in, is the output value of the output terminal.

[0046] Compared with the prior art, the present invention has the following beneficial effects:

[0047] 1. This invention is based on an upgrade of the existing machine learning algorithm, uses historical production logs as training data, and utilizes a bidirectional long-short-term memory recurrent neural network model to construct a stress testing model. It can achieve closed-loop stress testing simulation with a small amount of operation logs. By converting production logs into model results and generating new test cases based on the results, it can realize self-deduction detection of various potential stress testing scenarios, effectively improving the accuracy of stress testing. The model accuracy can reach over 85%, and it can realize the prediction of uncertain future trends based on the evolution of basic data.

[0048] 2. The present invention does not require preliminary testing and can be directly put into operation through the algorithm, breaking through the constraints of manual parameter detection. It can effectively improve the test efficiency of the stress test through a self-closed-loop stress test cycle. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 It is a schematic diagram of the process of the present invention;

[0050] Figure 2 Schematic diagram of the bidirectional recurrent neural network of the present invention unfolding in time;

[0051] Figure 3 A schematic diagram of a module using a bidirectional long short-term memory recurrent neural network method according to the present invention;

[0052] Figure 4 This is a schematic diagram of a test log in an embodiment of the present invention;

[0053] Figure 5 A schematic diagram of a test log prediction result according to an embodiment of the present invention;

[0054] Figure 6 This is a schematic diagram of a production log in an embodiment of the present invention;

[0055] Figure 7 Schematic diagram of production log prediction results in an embodiment of the present invention. DETAILED DESCRIPTION

[0056] The present invention is described in detail below with reference to the accompanying drawings and specific embodiments. This embodiment is implemented based on the technical solution of the present invention, and provides a detailed implementation method and specific operation process, but the protection scope of the present invention is not limited to the following embodiments.

[0057] Example

[0058] like Figure 1 As shown, a system stress testing method based on a machine deep learning self-optimization model specifically includes the following steps:

[0059] S1. Obtain historical production log information as model training log information and input it into a stress testing model based on a bidirectional long short-term memory recurrent neural network;

[0060] S2. Compare the prediction results of the stress test model with the production log information of the real system to calculate the accuracy of the stress test model;

[0061] S3, determine whether the accuracy is greater than a preset threshold, if so, go to step S5, otherwise go to step S4;

[0062] S4, adjust the model parameters of the stress test model and go to step S1;

[0063] S5. Perform stress testing on the system based on the stress testing model corresponding to the current prediction result, and make corresponding program modifications based on the error information obtained before finally launching the system.

[0064] like Figure 2As shown in the figure, the basic idea of ​​a bidirectional recurrent neural network is to propose that each training sequence consists of two recurrent neural networks, one for the forward and one for the backward, connected to an output layer. This structure provides the output layer with complete past and future contextual information for every point in the input sequence. Six unique weights are reused at each time step, corresponding to the input to the forward and backward hidden layers (w1, w3), the hidden layer to the hidden layer itself (w2, w5), and the forward and backward hidden layer to the output layer (w4, w6). It is worth noting that there is no information flow between the forward and backward hidden layers.

[0065] A key advantage of recurrent neural networks (RNNs) is their ability to leverage contextual information when mapping input and output sequences. However, standard RNNs are limited in the scope of contextual information they can access. This problem causes the influence of hidden layer inputs on network outputs to diminish as the network loops recur. Therefore, to address this issue, the long short-term memory (LSTM) architecture was developed. Rather than being a type of RNN, LSTM is more like an enhanced component incorporated into an RNN. Specifically, the small circles in the hidden layer of an RNN are replaced with LSTM modules.

[0066] The stress testing model has a module that uses a bidirectional long short-term memory recurrent neural network method.

[0067] like Figure 3 As shown, the module using the bidirectional long short-term memory recurrent neural network method includes an input end, an output end, a forgetting end, and a relay end.

[0068] The workflow of the module using the bidirectional long short-term memory recurrent neural network method includes a forward calculation phase and a backward calculation phase.

[0069] The formula for the input of the module using the bidirectional long short-term memory recurrent neural network method during the forward calculation phase is as follows:

[0070]

[0071] in, is the i-th input value of the external input at the input terminal at time t, w iτ is the weight of the i-th input value, is the hth output value of the hidden unit at the input end at time t-1, w hτ is the weight of the h-th output value, is the cth output value of the input terminal at the relay terminal at time t-1, w cτ is the weight of the c-th output value, f() is the activation function, is the output value of the input terminal at time t, I is the total amount of external input data, H is the total amount of data of the hidden unit, and C is the total amount of data at the relay terminal.

[0072] The formula for the forgetting end of the module using the bidirectional long short-term memory recurrent neural network method during the forward calculation phase is as follows:

[0073]

[0074] Among them, w iφ is the weight of the i-th input value of the external input of the forgetting end, w hφ is the weight of the h-th output value of the hidden unit at the forgetting end, w cφ is the weight of the c-th output value of the forgetting end at the relay end, is the output value of the forget terminal at time t.

[0075] The formula for the relay end of the module using the bidirectional long short-term memory recurrent neural network method in the forward calculation phase is as follows:

[0076]

[0077] in, is the output value of the forget terminal at time t, is the cth output value of the relay terminal at time t-1, w ic is the weight of the i-th input value of the external input of the relay end, w hc is the weight of the h-th output value of the hidden unit at the relay end, is the output value of the input at time t, and g() is the activation function.

[0078] The formula for the output of the module using the bidirectional long short-term memory recurrent neural network method during the forward calculation phase is as follows:

[0079]

[0080] Among them, w iw is the weight of the i-th input value of the external input at the output end, w hw is the weight of the h-th output value of the output hidden unit, is the cth output value of the output terminal at the relay end at time t, w cw is the weight of the c-th output value of the output terminal at the relay end, is the output value of the output terminal at time t;

[0081] The formula for the output value of the relay end is as follows:

[0082]

[0083] in, is the output value of the relay terminal at time t, and h() is the activation function.

[0084] The formula for the input of the module using the bidirectional long short-term memory recurrent neural network method during the backward calculation phase is as follows:

[0085]

[0086] in, is the output value of the input terminal at time t, f′() is the activation function;

[0087] The formula for the forget side is as follows:

[0088]

[0089] in, is the output value of the forget terminal at time t;

[0090] The formula on the relay side is as follows:

[0091]

[0092] in, is the output value of the relay terminal at time t, is the input value of the relay end at time t, is the input value of the relay end at time t+1, is the kth output value of the non-hidden unit at the relay end at time t, w ck is the weight of the k-th output value of the relay terminal, is the hth output value of the hidden unit at the relay end at time t+1, w ch is the weight of the h-th output value of the relay end.

[0093] The output formula of the module using the bidirectional long short-term memory recurrent neural network method in the backward calculation phase is as follows:

[0094]

[0095] in, is the output value of the output terminal.

[0096] In existing technology, the test data used during development and testing is hypothesized during the development process. Only after all tests meet the required standards will the data be gradually introduced into production and put into production, ultimately generating production logs. In other words, the test data is unrelated to actual production logs. The purpose of building a production log analysis model is to apply the information in production logs to testing, making the test data as close to reality as possible. Therefore, production log modeling summarizes production logs and simulates the sequence of operations, facilitating better coverage of these operating environments and enabling better testing. This transforms passive testing into active testing. As shown in the flowchart, the production log module only needs to obtain production log data from online maintenance for a period of time. It can then provide the constructed production log model to the test unit for comprehensive, multi-dimensional testing, thus avoiding the occurrence of single-issue scenarios and achieving active testing. Using a bidirectional long-short-term memory recurrent neural network model algorithm, even with only optimization of the order and frequency of interface calls and no pre-processing of the logs, such as missing value filling or data denoising, the model still achieves a prediction accuracy of approximately 85%. In addition, this accuracy will also be affected by the amount of data and slight fluctuations caused by the number of different instructions.

[0097] In specific implementation, the function number calling situation in the login scenario is shown in Table 1. Table 1 is as follows:

[0098] Table 1 Log table

[0099]

[0100] The test log column in Table 1 is the function number sequence sent by a complete test case for the test login scenario. After repeatedly sending the test case instructions, the following is generated: Figure 4 The curve shown. Figure 5 As shown in the figure, the curve in the test log prediction result represents the predicted data curve of the test log by the constructed model. Although there are some differences between the two, it can be seen that the predicted result is not much different from the actual test case, indicating that the model can reflect the log situation relatively realistically.

[0101] The production log column in Table 1 is the function number sequence of some production logs, as shown in the following example: Figure 6 As shown. Figure 7 As shown in the figure, the red curve in the production log prediction result represents the prediction curve of the production log by the established model. Due to the irregularity and uncertainty of the production log, the predicted data will be significantly different from the production log, which is exactly the prediction result of the future regularity of the production log.

[0102] Comparing test logs with production logs shows that using only a single, repeated test case cannot accurately interpret the true results of stress testing. Deep learning algorithms can effectively address this problem.

[0103] In addition, it should be noted that the specific embodiments described in this specification may be named differently, and the above content described in this specification is merely an example of the structure of the present invention. Any equivalent changes or simple changes made based on the structure, features and principles of the present invention are included in the protection scope of the present invention. Those skilled in the art of the present invention may make various modifications or supplements to the specific examples described or adopt similar methods, as long as they do not deviate from the structure of the present invention or exceed the scope defined by the claims, they should all fall within the protection scope of the present invention.

Claims

1. A system stress testing method based on a machine deep learning self-optimization model, characterized in that: The specific steps include: S1. Obtain historical production log information as model training log information, and input it into a stress testing model based on a bidirectional long short-term memory recurrent neural network. The stress testing model includes a module using a bidirectional long short-term memory recurrent neural network method, including an input end, an output end, a forgetting end, and a relay end. The workflow of the module using the bidirectional long short-term memory recurrent neural network method includes a forward calculation phase and a backward calculation phase. The formula of the relay end of the module using the bidirectional long short-term memory recurrent neural network method in the forward calculation stage is as follows: in, is the output value of the forget terminal at time t, is the cth output value of the relay terminal at time t-1, w ic is the weight of the i-th input value of the external input of the relay end, is the i-th input value of the external input at the input terminal at time t, w hc is the weight of the h-th output value of the hidden unit at the relay end, is the h-th output value of the hidden unit at the input end at time t-1, is the output value of the input terminal at time t, g() is the activation function, I is the total amount of external input data, and H is the total amount of hidden unit data; S2. Compare the prediction results of the stress test model with the production log information of the real system to calculate the accuracy of the stress test model; S3, determine whether the accuracy is greater than a preset threshold, if so, go to step S5, otherwise go to step S4; S4. Adjust the model parameters of the stress test model and go to step S1; S5. Perform stress testing on the system based on the stress testing model corresponding to the current prediction result, and make corresponding program modifications based on the error information obtained before finally launching the system.

2. A system stress testing method based on a machine deep learning self-optimization model according to claim 1, characterized in that: The formula for the input end of the module using the bidirectional long short-term memory recurrent neural network method in the forward calculation stage is as follows: in, is the i-th input value of the external input at the input terminal at time t, w iτ is the weight of the i-th input value, is the hth output value of the hidden unit at the input end at time t-1, w hτ is the weight of the h-th output value, is the cth output value of the input terminal at the relay terminal at time t-1, w cτ is the weight of the c-th output value, f() is the activation function, is the output value of the input terminal at time t, I is the total amount of external input data, H is the total amount of data of the hidden unit, and C is the total amount of data at the relay terminal.

3. A system stress testing method based on a machine deep learning self-optimization model according to claim 2, characterized in that: The formula for the forgetting end of the module using the bidirectional long short-term memory recurrent neural network method in the forward calculation phase is as follows: Among them, w iφ is the weight of the i-th input value of the external input of the forgetting end, w hφ is the weight of the h-th output value of the hidden unit at the forgetting end, w cφ is the weight of the c-th output value of the forgetting end at the relay end, is the output value of the forget terminal at time t.

4. A system stress testing method based on a machine deep learning self-optimization model according to claim 3, characterized in that: The formula for the output of the module using the bidirectional long short-term memory recurrent neural network method in the forward calculation phase is as follows: Among them, w iw is the weight of the i-th input value of the external input at the output end, w hw is the weight of the h-th output value of the output hidden unit, is the cth output value of the output terminal at the relay end at time t, w cw is the weight of the c-th output value of the output terminal at the relay end, is the output value of the output terminal at time t; The formula for the output value of the relay end is as follows: in, is the output value of the relay terminal at time t, and h() is the activation function.

5. A system stress testing method based on a machine deep learning self-optimization model according to claim 4, characterized in that: The formula for the input end of the module using the bidirectional long short-term memory recurrent neural network method in the backward calculation stage is as follows: in, is the output value of the input terminal at time t, f′() is the activation function; The formula for the forget side is as follows: in, is the output value of the forget terminal at time t; The formula on the relay side is as follows: in, is the output value of the relay terminal at time t, is the input value of the relay end at time t, is the input value of the relay end at time t+1, is the kth output value of the non-hidden unit at the relay end at time t, w ck is the weight of the k-th output value of the relay end, is the hth output value of the hidden unit at the relay end at time t+1, w ch is the weight of the h-th output value of the relay end.

6. A system stress testing method based on a machine deep learning self-optimization model according to claim 5, characterized in that: The formula for the output of the module using the bidirectional long short-term memory recurrent neural network method in the backward calculation stage is as follows: in, is the output value of the output terminal.

Images