A benchmark test method for e-commerce killing application

Abstract

The application discloses a benchmark test method for e-commerce application, characterized in that a benchmark test system is deployed in a computer system running an e-commerce application to be evaluated, wherein the benchmark test system comprises a data acquisition and processing system, a Transformer performance evaluation model and a report generation subsystem; the benchmark test system is started when the e-commerce application to be evaluated has no interference flow, a stress test is initiated, performance data of the computer system are obtained through the data acquisition and processing system, and the stress test is ended; the obtained performance data are input into the Transformer performance evaluation model to obtain a performance score; the performance data and the performance score are input into the report generation subsystem to generate and output a user report; and the method has the advantages that the evaluation result is more accurate and objective, the special requirements of the e-commerce application are fully considered, and the method has stronger representativeness and applicability in the field of e-commerce applications.

Description

TECHNICAL FIELD

[0001] The application relates to a benchmark testing method, in particular to a benchmark testing method for e-commerce killing application. BACKGROUND

[0002] The killing activity is an important function for e-commerce marketing, and the activity has the characteristics of short time and large concurrent access volume. If the activity is deployed together with the original application of the website, it will inevitably impact the existing business, and may even cause the entire website to crash. If the e-commerce application lacks the load capacity of the killing scenario, it is easy to cause system lag, crash and other adverse conditions, causing serious losses. Therefore, the benchmark testing can be used to quantitatively and comparably test the performance indicators such as high concurrency of the killing function in the e-commerce application.

[0003] The existing benchmark testing method tends to be general evaluation, and fails to fully consider the special needs of e-commerce killing application. Although this general evaluation is conducive to the wide application of technology, it ignores the performance optimization and stability requirements in the e-commerce killing scenario.

[0004] In addition, the current benchmark testing method focuses more on client performance evaluation, while ignoring the performance impact of the complex micro-service environment on the server. With the popularity of cloud computing and micro-service architecture, the performance of the server has an increasingly significant impact on user experience. Therefore, performance evaluation in the complex micro-service environment of the server becomes particularly important.

[0005] Furthermore, the existing benchmark testing method usually relies on a simple mathematical model to construct the evaluation standard. Although this method is intuitive, it lacks a deep understanding of the behavior of complex systems, making the evaluation results less accurate. SUMMARY

[0006] The technical problem to be solved by the application is to provide a benchmark testing method for e-commerce killing application, which can provide a more scientific and effective solution for performance evaluation of e-commerce killing application, and make the evaluation results more accurate.

[0007] The technical solution adopted by the application to solve the above technical problems is: a benchmark testing method for e-commerce killing application, comprising the following steps:

[0008] Step 1: deploying a benchmark testing system in a computer system that has run an e-commerce killing application to be evaluated, wherein the benchmark testing system comprises a data acquisition and processing system, a Transformer performance evaluation model and a report generation subsystem;

[0009] The Transformer performance evaluation model is obtained by the following method:

[0010] Step 1, based on the Spring Boot and Spring Cloud development framework, a simulation application based on the e-commerce application is developed;

[0011] Step 2, based on the simulation application, a large number of simulation is used to build an e-commerce application performance test data set;

[0012] Step 3, the e-commerce application performance test data set is input into the Transformer model to be trained to obtain the Transformer performance evaluation model;

[0013] Step 2, start the benchmark test system when the to-be-evaluated e-commerce application has no interference flow, initiate stress testing, obtain the performance data of the computer system through the data acquisition and processing system, and end the stress testing;

[0014] Step 3, input the obtained performance data into the Transformer performance evaluation model to obtain the performance score;

[0015] Step 4, input the performance data and the performance score into the report generation subsystem to generate and output the user report.

[0016] Compared with the prior art, the advantages of the present application are that the simulation application based on the e-commerce application is developed based on the Spring Boot and Spring Cloud development framework, and the performance evaluation of the e-commerce application in the complex micro-service environment of the server is fully considered. Then, based on the simulation application, a large number of simulation is used to build an e-commerce application performance test data set, and then the e-commerce application performance test data set is input into the Transformer model to be trained to obtain the Transformer performance evaluation model. The Transformer performance evaluation model is introduced into the benchmark test system, so that the performance evaluation result of the to-be-evaluated e-commerce application is more accurate and objective. Through the benchmark test system deployed in the computer system running the to-be-evaluated e-commerce application, the performance of the e-commerce application under high concurrency conditions can be accurately simulated and evaluated. At the same time, according to the performance data and the performance score, an easy-to-understand user report is generated to help decision-makers quickly grasp the performance status and make data-based decisions. In addition, the present application fully considers the special needs of the e-commerce application, and has stronger representativeness and applicability in the e-commerce application field.

[0017] Further, the data acquisition and processing system and the report generation subsystem are constructed based on the Java programming language.

[0018] Further, the specific operation process of step 1 is as follows:

[0019] Step ①-1, abstract modeling of e-commerce flash sale application, abstracting the most essential features, and designing a lightweight simulation application architecture based on the most essential features;

[0020] Step ①-2, based on Maven tool to build the development framework based on Spring Boot and Spring Cloud;

[0021] Step ①-3, based on the development framework of Spring Boot and Spring Cloud, develop the lightweight simulation application architecture from bottom to top, get the simulation application based on e-commerce flash sale application:

[0022] Step ①-3-1, develop data access layer code, used to access MySQL database and Redis cache;

[0023] Step ①-3-2, develop business logic layer code, used to process data returned from control layer and data access layer data interaction;

[0024] Step ①-3-3, develop controller layer code, used to handle HTTP network access request, pass network data to business logic layer to complete simulation e-commerce flash sale business logic processing. A representative e-commerce flash sale system, namely lightweight simulation application architecture, is designed and developed, which is used for the training of Transformer model in the following. After abstraction, the bloated function is abandoned, and the performance impact of complex micro-service environment on the server side is focused on. The focus is on the flash sale function of e-commerce system, which represents e-commerce system and is more suitable for benchmark testing.

[0025] Further, the most essential features include: the server can handle large concurrent traffic and the data is stored in a relational database and accessed through cache acceleration;

[0026] The lightweight simulation application architecture includes controller layer, business logic layer and data access layer connected in turn, the controller layer includes e-commerce commodity server and e-commerce user server, the business logic layer includes e-commerce Redis server and e-commerce MySQL server, and the data access layer includes Redis cache and MySQL database. The response of Redis cache and MySQL database to high concurrent request is simulated, which reflects the more essential bottom features of e-commerce system.

[0027] Further, the specific operation process of step ② is as follows:

[0028] Step ②-1, through JMeter tool, intensive HTTP access request is initiated to the target interface of the simulation application based on e-commerce flash sale application, and stress test is carried out;

[0029] Step 2-2, during the pressure test, the number of response threads in the pressure test is adjusted constantly and recorded until the start of the request failure, the maximum number of response threads without request failure is obtained, and is taken as the maximum load capacity of the computer system to the e-commerce killing application under the current environment;

[0030] Step 2-3, during the pressure test, the performance data of the computer system at the maximum number of response threads is collected by sar and perf tools, and the performance data is combined with the corresponding maximum number of response threads to form an optimal performance data;

[0031] Step 2-4, the resource configuration of the computer system where the e-commerce killing simulation application is located is adjusted constantly by the control variable method, and steps 2-1 to 2-3 are repeated until 10,000 optimal performance data are obtained;

[0032] Step 2-5, the optimal performance data obtained in step 2-4 is de-duplicated and de-emptied to obtain processed optimal performance data, each optimal performance data is evaluated by an expert to obtain a corresponding final score and is taken as a data label, each optimal performance data and the corresponding data label are combined to form a data sample, and all data samples are combined to form an e-commerce killing application performance test dataset. The pressure test and performance data collection during the pressure test are realized by combining JMeter, sar and perf tools, and these tools are open source and free, and have excellent effect; the resource configuration of the computer system where the e-commerce killing simulation application is located is adjusted constantly by the control variable method, which innovates the method of obtaining a large amount of performance data to construct a dataset.

[0033] Further, the specific operation process of step 3 is as follows:

[0034] Step 3-1, randomly select 70% of the data samples in the e-commerce killing application performance test dataset as a training set;

[0035] Step 3-2, pre-process the training set, specifically: first convert each data sample in the training set into corresponding text data, then convert each text data into a token sequence, and assign a unique ID to each token sequence to obtain a pre-processed training set;

[0036] Step 3-3, set the hyperparameters of model training, including learning rate 0.001, regularization parameter 0.000001, and encoder layer number 2;

[0037] Step 3-4, input the pre-processed training set into the Transformer model to be trained for training;

[0038] Step ③-5, the trained Transformer model is iteratively optimized using the Adam optimizer according to the Huber Loss loss function, and the iteration process is stopped when the conditions for stopping iteration are met, and the Transformer performance evaluation model is obtained. The Huber Loss loss function compares the two traditional loss functions of mean square error and mean absolute error, which can maintain the efficiency of mean square error in most cases, and maintain the robustness of mean absolute error when abnormal values occur; The condition for early stopping can make the Transformer performance evaluation model avoid overfitting.

[0039] Further, the condition for stopping iteration is one of that the iteration round reaches 100, or the loss value has not improved for 5 consecutive iterations, and the loss value is obtained by comparing the evaluation results output by the Transformer model with the data labels.

[0040] Further, the performance data includes: CPU running time, CPU utilization, disk input / output read / write speed, network input / output transmission rate, network error rate, memory occupancy, memory page frequency and the number of processes created per second. BRIEF DESCRIPTION OF DRAWINGS

[0041] Figure 1 The overall flowchart of the present application is shown in the figure;

[0042] Figure 2 The structure diagram of the Transformer model to be trained in the embodiment is shown in the figure. DETAILED DESCRIPTION

[0043] The present application will be further described in detail below in conjunction with the embodiments of the drawings.

[0044] As shown in the figure, a benchmark testing method for e-commerce killing application includes the following steps: Figure 1

[0045] Step one, deploy a benchmark testing system in a computer system that has run an e-commerce killing application to be evaluated, the benchmark testing system includes a data acquisition and processing system, a Transformer performance evaluation model and a report generation subsystem;

[0046] The Transformer performance evaluation model is obtained by the following method:

[0047] Step ①, develop a simulation application based on e-commerce killing application based on the development framework of Spring Boot and Spring Cloud, specifically:

[0048] ​Step 1-1, abstract modeling of e-commerce flash sale application, abstracting the most essential features, designing a lightweight simulation application architecture according to the most essential features; the most essential features include: the server can handle large concurrent traffic (simulate purchase and seize the interface) and the data is stored in a relational database and accessed through cache acceleration;

[0049] The lightweight simulation application architecture includes a controller layer, a business logic layer and a data access layer connected in turn, the controller layer includes an e-commerce commodity server and an e-commerce user server, the business logic layer includes an e-commerce Redis server and an e-commerce MySQL server, and the data access layer includes Redis cache and MySQL database;

[0050] Step 1-2, based on Maven tool to build development framework based on Spring Boot and Spring Cloud;

[0051] Step 1-3, develop the lightweight simulation application architecture from bottom to top based on the development framework of Spring Boot and Spring Cloud, get the simulation application based on e-commerce flash sale application:

[0052] Step 1-3-1, develop data access layer code to access MySQL database and Redis cache;

[0053] Step 1-3-2, develop business logic layer code to handle data returned from the control layer and data access layer data interaction;

[0054] Step 1-3-3, develop controller layer code to handle HTTP network access requests, pass network data to business logic layer to complete simulation e-commerce flash sale business logic processing;

[0055] Spring Boot is a special Java development framework, Spring Cloud is built on the basis of Spring Boot, which uses its features to simplify distributed system development, through Spring Cloud and Spring Boot development framework, applications can be more easily built as part of a distributed system;

[0056] The lightweight simulation application architecture includes data access layer, business logic layer and controller layer from bottom to top, the data access layer is used to interact with database data, and the cache is used to accelerate data access, so it also involves interaction with the cache; the business logic layer is used for intermediate business logic processing, which correctly processes the incoming data obtained from the upper layer and reads and writes to the database by calling the data access layer; the controller layer is used to accept HTTP requests, process request data and hand it over to the business logic layer for processing; among them, the data access layer is closest to the database, and the controller layer is closest to the network request;

[0057] Step 2, based on the simulation application, a large number of simulations are used to build an e-commerce application performance test dataset, specifically:

[0058] Step 2-1, use the JMeter tool to initiate intensive HTTP access requests to the target interface of the e-commerce application simulation application, and perform stress testing; JMeter is a tool for stress testing;

[0059] Step 2-2, during the stress test, continuously adjust the response thread number in the stress test and record it until the request failure occurs, then stop, get the maximum response thread number without request failure, and use it as the maximum load capacity of the computer system for the e-commerce application in the current environment;

[0060] Because the stress test simulates users participating in the e-commerce application, the response thread number is the number of HTTP requests sent by the JMeter tool to the e-commerce application at the same time, and the e-commerce application needs to respond to the response thread, which is one-to-one;

[0061] Step 2-3, during the stress test, use the sar and perf tools to collect performance data of the computer system under the maximum response thread number, and form an optimal performance data with the corresponding maximum response thread number;

[0062] Step 2-4, continuously adjust the resource configuration of the computer system where the e-commerce application simulation application is located by the control variable method, and repeat steps 2-1 to 2-3 until 10,000 optimal performance data are obtained;

[0063] Control variable method: keep other resource configurations unchanged each time, adjust other resource configurations, same CPU, disk and network, change memory allocation, the effect will be different, specific example:

[0064] CPU frequency is 3GHz, disk quota is 100GB, which is large enough, and memory is only 4GB, which may become a performance bottleneck, at this time, memory is added to 8GB, which can better load the application program (can bear more access requests);

[0065] Step 2-5, perform de-duplication and empty operation on the optimal performance data obtained in step 2-4 to obtain processed optimal performance data, evaluate each optimal performance data by experts, get the corresponding final score and use it as a data label, each optimal performance data and the corresponding data label form a data sample, and all data samples form an e-commerce application performance test dataset;

[0066] For example, a data sample is

CPU running time: 15ms, CPU utilization: 99%, …, 8

[0067] Step ③, input the e-commerce application performance test data set into the Transformer model to be trained for training, and obtain the Transformer performance evaluation model, specifically:

[0068] Step ③-1, randomly select 70% of the data samples in the e-commerce application performance test data set as the training set;

[0069] Step ③-2, preprocessing the training set, specifically: first, convert each data sample in the training set into corresponding text data, then convert each text data into a token sequence, and assign a unique ID to each token sequence, to obtain the preprocessed training set;

[0070] Step ③-3, set the hyperparameters for model training, including learning rate 0.001, regularization parameter 0.000001, and encoder layer number 2;

[0071] Step ③-4, input the preprocessed training set into the Transformer model to be trained; the Transformer model to be trained is constructed by the deep learning framework PyTorch, which is prior art, and the specific structure is as shown in Figure 2 The left side is the encoder block, the right side is the decoder block, the multi-head attention layer is composed of multiple self-attention mechanisms, and it can be seen that the encoder block includes a multi-head attention layer, and the decoder block includes two multi-head attention layers (one of which uses a mask), and the multi-head attention layer also includes a residual connection above for preventing network degradation, and layer normalization is used to normalize the activation value of each layer.

[0072] Step ③-5, according to the Huber Loss loss function, use the Adam optimizer to iteratively optimize the Transformer model to be trained, and stop the iteration process when the stopping iteration condition is met, and obtain the Transformer performance evaluation model; the stopping iteration condition is one of the iteration round reaching 100, or the loss value still not improving after 5 consecutive iterations; the loss value is obtained by comparing the evaluation results output by the Transformer model with the data label;

[0073] Huber Loss loss function, also known as smooth L1 loss, is a loss function used for regression tasks that introduces smoothness between predicted values and target values, making it more robust to outliers than mean squared error (MSE); the main function of the Adam optimizer is to update neural network parameters based on gradient information to minimize the loss function;

[0074] Step two, start the benchmark test system when there is no interference traffic in the e-commerce application to be evaluated, initiate stress testing, obtain computer system performance data through the data collection and processing system, and end the stress testing; performance data includes: CPU running time, CPU utilization, disk input / output read / write rate, network input / output transmission rate, network error rate, memory usage, memory page frequency and the number of processes created per second;

[0075] Step three, input the obtained performance data into the Transformer performance evaluation model to obtain the performance score;

[0076] Step four, input the performance data and performance score into the report generation subsystem to generate and output user reports; the output user reports support multiple formats, such as txt text format, HTML web page format, and csv data table format; these formats can be selected and obtained by Java language string splicing; the data source of the user report is the performance data obtained in step two and the performance score obtained in step three; the user report can give intuitive evaluation results and corresponding reference data, and is more objective.

[0077] In this embodiment, the data collection and processing system and the report generation subsystem are constructed based on the Java programming language; the Transformer performance evaluation model, the data collection and processing system, and the report generation subsystem are assembled into a system for parallel debugging; user accessible commands are configured for the system after debugging to obtain the benchmark test system.

[0078] By combining the efficient development framework of Spring Boot and Spring Cloud, the construction of large-scale simulation test data sets, the training of the Transformer performance evaluation model, the data collection and processing system of Java language, and the integrated report generation subsystem, a highly modularized, automated and user-friendly benchmark test system is realized, which can accurately evaluate the performance of e-commerce applications, while having good scalability and real-time performance, providing a high-efficiency, flexible and cost-effective test solution for users.

Claims

1. A benchmarking method for e-commerce flash sale application, characterized in that The method comprises the following steps: Step one, deploying a benchmark test system in a computer system running an e-commerce application to be evaluated, the benchmark test system comprising a data acquisition and processing system, a Transformer performance evaluation model and a report generation subsystem; The Transformer performance evaluation model is obtained by the following method: Step ①, developing a simulation application based on the e-commerce application based on the Spring Boot and Spring Cloud development framework, the specific operation process being as follows: Step ①-1, abstract modeling the e-commerce application to abstract the most essential features, and designing a lightweight simulation application architecture according to the most essential features; The most essential features include that the server can handle large concurrent traffic and the data is stored in a relational database and accessed through caching; Step ①-2, building a development framework based on Spring Boot and Spring Cloud based on the Maven tool; Step ①-3, developing the lightweight simulation application architecture from bottom to top based on the Spring Boot and Spring Cloud development framework to obtain the simulation application based on the e-commerce application: Step ①-3-1, developing data access layer code for accessing MySQL database and Redis cache; Step ①-3-2, developing business logic layer code for processing data returned from the control layer and interacting with data access layer data; Step ①-3-3, developing controller layer code for processing HTTP network access requests, passing network data to the business logic layer to complete the simulation of e-commerce application business logic processing; Step ②, based on the simulation application, a large amount of simulation is used to build an e-commerce application performance test data set, the specific operation process being as follows: Step ②-1, launching intensive HTTP access requests to the target interface of the simulation application based on the e-commerce application through the JMeter tool and performing stress testing; Step ②-2, during the stress testing, constantly adjusting the response thread number in the stress testing and recording until the request failure occurs, obtaining the maximum response thread number without request failure, and taking it as the maximum load capacity of the computer system to the e-commerce application under the current environment; Step ②-3, during the stress testing, collecting the performance data of the computer system at the maximum response thread number through the sar and perf tools, and forming an optimal performance data with the corresponding maximum response thread number; Step ②-4, adjusting the resource configuration of the computer system where the e-commerce simulation application is located by the control variable method, and repeating steps ②-1 to ②-3 until 10,000 optimal performance data are obtained; Step 2-5, the optimal performance data obtained in step 2-4 is de-duplicated and de-emptied to obtain processed optimal performance data, each piece of processed optimal performance data is evaluated by an expert to obtain a corresponding final score as a data label, each piece of optimal performance data and the corresponding data label form a data sample, and all data samples form an e-commerce flash sale application performance test data set; Step 3, input the e-commerce flash sale application performance test data set into the Transformer model to be trained to obtain a Transformer performance evaluation model; Step two, start the benchmark test system when there is no interference flow in the e-commerce flash sale application to be evaluated, initiate the stress test, obtain the performance data of the computer system through the data acquisition and processing system, and end the stress test; Step three, input the obtained performance data into the Transformer performance evaluation model to obtain a performance score; Step four, input the performance data and the performance score into the report generation subsystem to generate and output a user report.

2. The benchmarking method for e-commerce flash sale application according to claim 1, characterized in that The data acquisition and processing system and the report generation subsystem are constructed based on the Java programming language. 3.The benchmarking method for e-commerce flash sale application of claim 1, wherein The lightweight simulation application architecture includes a controller layer, a business logic layer and a data access layer connected in sequence, the controller layer includes an e-commerce commodity server and an e-commerce user server, the business logic layer includes an e-commerce Redis server and an e-commerce MySQL server, and the data access layer includes a Redis cache and a MySQL database.

4. The benchmarking method for e-commerce flash sale application according to claim 1, characterized in that The specific operation process of step 3 is as follows: Step 3-1, randomly select 70% of the data samples in the e-commerce flash sale application performance test data set as a training set; Step 3-2, preprocess the training set, specifically: first convert each data sample in the training set into corresponding text data, then convert each text data into a token sequence, and assign a unique ID to each token sequence to obtain a preprocessed training set; Step 3-3, set the hyperparameters for model training, including a learning rate of 0.001, a regularization parameter of 0.000001, and an encoder layer number of 2; Step 3-4, input the preprocessed training set into the Transformer model to be trained; Step 3-5, according to the Huber Loss loss function, use the Adam optimizer to iteratively optimize the Transformer model to be trained until the stopping iteration condition is met, stop the iteration process, and obtain the Transformer performance evaluation model.

5. The benchmarking method for e-commerce flash sale application according to claim 4, characterized in that The stopping iteration condition is one of the iteration round reaching 100 or the loss value not improving for 5 consecutive iterations, and the loss value is obtained by comparing the evaluation results output by the Transformer model with the data label.

6. The benchmarking method for e-commerce flash sale application according to claim 1, characterized in that The performance data includes CPU running time, CPU utilization, disk input / output read / write rate, network input / output transmission rate, network error rate, memory occupancy, memory page frequency and the number of processes created per second.

Images