E-commerce seckilling application-oriented benchmark test method

Abstract

The invention discloses an e-commerce seckilling application-oriented benchmark test method, which is characterized in that a benchmark test system is deployed in a computer system in which a to-be-evaluated e-commerce seckilling application is operated, and the benchmark test system comprises a data acquisition and processing system, a Transform performance evaluation model and a report generation subsystem; when the to-be-evaluated e-commerce seckilling application has no interference flow, starting the benchmark test system, initiating a pressure test, obtaining performance data of the computer system through the data acquisition and processing system, and ending the pressure test; inputting the obtained performance data into a Transform performance evaluation model to obtain a performance score; inputting the performance data and the performance score into a report generation subsystem, and generating and outputting a user report; the method has the advantages that the evaluation result is more accurate and objective, the special requirements of e-commerce seckilling application are fully considered, and the method has higher representativeness and applicability in the field of e-commerce application.

Description

Technical Field

[0001] The present invention relates to a benchmark testing method, in particular to a benchmark testing method for e-commerce flash sales applications. Background Art

[0002] Flash sales are an important function for e-commerce marketing. This activity is short-lived and has a large number of concurrent visits. If it is deployed together with the original website application, it will inevitably have an impact on the existing business, and the entire website may be paralyzed if not handled with care. If the e-commerce application lacks the load capacity for flash sales scenarios, it is easy to cause system freezes, crashes, and other adverse situations, causing serious losses. Therefore, benchmark testing can be used to quantitatively and comparatively test the performance indicators of the flash sales function in e-commerce applications, such as high concurrency.

[0003] Existing benchmarking methods tend to focus on general evaluation and fail to fully consider the special needs of e-commerce flash sales applications. Although this general evaluation is conducive to the widespread application of technology, it ignores the performance optimization and stability requirements in e-commerce flash sales scenarios.

[0004] In addition, current benchmarking methods focus on client performance evaluation, while ignoring the performance impact of complex microservice environments on the server side. With the popularization of cloud computing and microservice architecture, the impact of server performance on user experience is becoming increasingly significant. Therefore, performance evaluation in complex microservice environments on the server side has become particularly important.

[0005] Furthermore, existing benchmarking methods usually rely on simple mathematical models to construct evaluation criteria. Although this method is intuitive, it lacks a deep understanding of the behavior of complex systems, making the evaluation results inaccurate. Summary of the invention

[0006] The technical problem to be solved by the present invention is to provide a benchmark testing method for e-commerce flash sale applications, which can provide a more scientific and effective solution for the performance evaluation of e-commerce flash sale applications, so that the obtained evaluation results are more accurate.

[0007] The technical solution adopted by the present invention to solve the above technical problems is: a benchmarking method for e-commerce flash sale applications, comprising the following steps: Step 1: deploying a benchmark test system in a computer system that has been running the e-commerce flash sale application to be evaluated, wherein the benchmark test system includes a data acquisition and processing system, a Transformer performance evaluation model, and a report generation subsystem; The Transformer performance evaluation model is obtained by the following method: Step 1: Develop a simulation application based on the e-commerce flash sale application based on the Spring Boot and Spring Cloud development framework; Step 2: Based on the simulation application, a performance test data set for e-commerce flash sales applications is constructed through a large number of simulations. Step ③, input the e-commerce flash sale application performance test data set into the Transformer model to be trained to obtain the Transformer performance evaluation model; Step 2: When the e-commerce flash sale application to be evaluated has no interference traffic, the benchmark test system is started, a stress test is initiated, performance data of the computer system is obtained through the data acquisition and processing system, and the stress test is ended; Step 3: Input the obtained performance data into the Transformer performance evaluation model to obtain a performance score; Step 4: Input the performance data and performance scores into the report generation subsystem to generate and output user reports.

[0008] Compared with the prior art, the advantages of the present invention are that a simulation application based on an e-commerce flash sale application is first developed based on the development framework of Spring Boot and Spring Cloud, and the performance evaluation of the e-commerce flash sale application in a complex microservice environment on the server side is fully considered. Then, based on the simulation application, a performance test data set of the e-commerce flash sale application is constructed through a large number of simulations, and then the performance test data set of the e-commerce flash sale application is input into the Transformer model to be trained for training to obtain a Transformer performance evaluation model, and the Transformer performance evaluation model is introduced into the benchmark test system, so that the performance evaluation result of the e-commerce flash sale application to be evaluated is more accurate and objective; by deploying the benchmark test system in a computer system that has been running the e-commerce flash sale application to be evaluated, the performance of the e-commerce flash sale application under high concurrency conditions can be accurately simulated and evaluated, and easy-to-understand user reports are generated according to performance data and performance scores, so as to help decision makers quickly grasp the performance status and make data-based decisions; in addition, the present invention fully considers the special needs of e-commerce flash sale applications, and has stronger representativeness and applicability to the field of e-commerce applications.

[0009] Furthermore, the data collection and processing system and the report generation subsystem are constructed based on the Java programming language.

[0010] Furthermore, the specific operation process of step ① is as follows: Step ①-1: abstractly model the e-commerce flash sale application, extract the most essential features, and design a lightweight simulation application architecture based on the most essential features; Step ①-2, build a development framework based on Spring Boot and Spring Cloud based on Maven tools; Step ①-3, based on the development framework of Spring Boot and Spring Cloud, develop the lightweight simulation application architecture from bottom to top layer by layer to obtain a simulation application based on the e-commerce flash sale application: Step ①-3-1, develop data access layer code for accessing MySQL database and Redis cache; Step ①-3-2, develop business logic layer code to process the interaction between data returned from the control layer and data access layer data; Step ①-3-3, develop controller layer code to process HTTP network access requests and pass network data to the business logic layer to complete the simulated e-commerce flash sale business logic processing. Design and develop a representative e-commerce flash sale system, namely a lightweight simulation application architecture, for the subsequent training of the Transformer model. After abstraction, it abandons bloated functions, pays attention to the performance impact of the complex microservice environment on the server side, and focuses on the flash sale function of the e-commerce system. This represents the e-commerce system and is more suitable for benchmarking.

[0011] Furthermore, the most essential features include: the server can handle large concurrent traffic and data is stored in a relational database and access is accelerated through caching; The lightweight simulation application architecture includes a controller layer, a business logic layer, and a data access layer connected in sequence. The controller layer includes an e-commerce product service end and an e-commerce user service end, the business logic layer includes an e-commerce Redis service end and an e-commerce MySQL service end, and the data access layer includes a Redis cache and a MySQL database. The Redis cache and MySQL database responses of high concurrent requests are simulated, reflecting the more essential underlying characteristics of the e-commerce system.

[0012] Furthermore, the specific operation process of step ② is as follows: Step ②-1, use the JMeter tool to initiate intensive HTTP access requests to the target interface of the simulated application based on the e-commerce flash sale application, and perform stress testing; Step ②-2, during the stress test, the number of response threads in the stress test is continuously adjusted and recorded until request failures begin to occur, and the maximum number of response threads without request failures is obtained, and used as the maximum load capacity of the computer system for the e-commerce flash sale application in the current environment; Step ②-3, during the stress test, the performance data of the computer system at the maximum number of response threads is collected through the sar and perf tools, and the performance data and the corresponding maximum number of response threads are combined into an optimal performance data; Step ②-4, continuously adjust the resource configuration of the computer system where the e-commerce flash sale simulation application is located through the control variable method, and repeat steps ②-1 to ②-3 until a total of 10,000 optimal performance data are obtained; Step ②-5, perform deduplication and de-duplication operations on the optimal performance data obtained in step ②-4 to obtain the processed optimal performance data, and evaluate each piece of processed optimal performance data through experts to obtain the corresponding final score and use it as a data label, and form each piece of optimal performance data and the corresponding data label into a data sample, and all data samples form an e-commerce flash sale application performance test data set. The performance data collection during the stress test and stress test is realized through the combination of JMeter, sar and perf tools, and these tools are open source and free, and the effect is excellent; the resource configuration of the computer system where the e-commerce flash sale simulation application is located is continuously adjusted through the control variable method, and an innovative method of obtaining a large amount of performance data to construct a data set is created.

[0013] Furthermore, the specific operation process of step ③ is as follows: Step ③-1, randomly select 70% of the data samples in the e-commerce flash sale application performance test data set as the training set; Step ③-2, preprocessing the training set, specifically: first convert each data sample in the training set into corresponding text data, then convert each text data into a token sequence, and assign a unique ID to each token sequence to obtain the preprocessed training set; Step ③-3, set the hyperparameters for model training including: learning rate is 0.001, regularization parameter is 0.000001, and the number of encoder layers is 2; Step ③-4, input the preprocessed training set into the Transformer model to be trained for training; Step ③-5, according to the Huber Loss loss function, use the Adam optimizer to iteratively optimize the Transformer model to be trained until the conditions for stopping iteration are met, then stop the iteration process to obtain the Transformer performance evaluation model. The Huber Loss loss function compares the two traditional loss functions, mean square error and mean absolute error, and can maintain the efficiency of mean square error in most cases, while maintaining the robustness of mean absolute error when outliers appear; setting the early stopping conditions can prevent the Transformer performance evaluation model from overfitting.

[0014] Furthermore, the condition for stopping iteration is that the number of iterations reaches 100, or the loss value does not improve after 5 consecutive iterations, and the loss value is obtained by comparing the evaluation result output by the Transformer model with the data label.

[0015] Furthermore, the performance data includes: CPU running time, CPU utilization, disk input and output read and write rates, network input and output transmission rates, network error rates, memory usage, memory paging frequency and the number of processes created per second. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 It is a schematic diagram of the overall process of the present invention; Figure 2 Schematic diagram of the structure of the Transformer model to be trained in this embodiment. DETAILED DESCRIPTION

[0017] The present invention is further described in detail below with reference to the accompanying drawings.

[0018] like Figure 1 As shown, a benchmarking method for e-commerce flash sales applications includes the following steps: Step 1: deploy a benchmark test system in a computer system that has been running the e-commerce flash sale application to be evaluated. The benchmark test system includes a data acquisition and processing system, a Transformer performance evaluation model, and a report generation subsystem. The Transformer performance evaluation model is obtained by the following method: Step ①: Develop a simulation application based on the e-commerce flash sale application based on the Spring Boot and Spring Cloud development framework, specifically: Step ①-1, abstract modeling of the e-commerce flash sale application, extracting the most essential features, and designing a lightweight simulation application architecture based on the most essential features; the most essential features include: the server can handle large concurrent traffic (simulating the purchase of the flash sale interface) and the data is stored in a relational database and access is accelerated through caching; The lightweight simulation application architecture includes a controller layer, a business logic layer, and a data access layer, which are connected in sequence. The controller layer includes an e-commerce product server and an e-commerce user server. The business logic layer includes an e-commerce Redis server and an e-commerce MySQL server. The data access layer includes a Redis cache and a MySQL database. Step ①-2, build a development framework based on Spring Boot and Spring Cloud based on Maven tools; Step ①-3, based on the development framework of Spring Boot and Spring Cloud, develop the lightweight simulation application architecture from bottom to top layer by layer to obtain a simulation application based on the e-commerce flash sale application: Step ①-3-1, develop data access layer code for accessing MySQL database and Redis cache; Step ①-3-2, develop business logic layer code to process the interaction between data returned from the control layer and data access layer data; Step ①-3-3, develop the controller layer code to process HTTP network access requests and pass network data to the business logic layer to complete the business logic processing of simulated e-commerce flash sales; Spring Boot is a specialized Java development framework. Spring Cloud is built on top of Spring Boot and uses its features to simplify distributed system development. With Spring Cloud and Spring Boot development frameworks, applications can be more easily built as part of a distributed system. The lightweight simulation application architecture includes the data access layer, business logic layer and controller layer from bottom to top. The data access layer is used to interact with database data. Since the cache accelerates data access, it also involves interaction with the cache. The business logic layer is used for intermediate business logic processing, correctly processing the incoming data obtained by the upper layer and realizing the reading and writing of the database by calling the data access layer. The controller layer is used to accept HTTP requests and hand over the request data to the business logic layer for processing after processing. Among them, the data access layer is closest to the database, and the controller layer is closest to the network request. Step ②, based on the simulation application, a large number of simulations are used to build an e-commerce flash sale application performance test data set, specifically: Step ②-1, use the JMeter tool to initiate intensive HTTP access requests to the target interface of the simulated application based on the e-commerce flash sale application, and perform stress testing; JMeter is a tool for stress testing; Step ②-2, during the stress test, the number of response threads in the stress test is continuously adjusted and recorded until request failures begin to occur, and the maximum number of response threads without request failures is obtained, and used as the maximum load capacity of the computer system for the e-commerce flash sale application in the current environment; Because the stress test simulates users participating in the flash sale, the number of response threads is the number of HTTP requests that the JMeter tool sends to the e-commerce flash sale application at the same time. The e-commerce flash sale application needs to use response threads to respond, which is a one-to-one correspondence; Step ②-3, during the stress test, the performance data of the computer system at the maximum number of response threads is collected through the sar and perf tools, and the performance data and the corresponding maximum number of response threads are combined into an optimal performance data; Step ②-4, continuously adjust the resource configuration of the computer system where the e-commerce flash sale simulation application is located through the control variable method, and repeat steps ②-1 to ②-3 until a total of 10,000 optimal performance data are obtained; The control variable method is: each time control other resource configurations unchanged, adjust other resource configurations, the same CPU, disk and network, change the memory quota, the effect will be different, for example: The CPU frequency is 3GHz and the disk quota is 100GB, which are both large enough. The memory is only 4GB, and the memory may become a performance bottleneck. In this case, increasing the memory to 8GB can better load the application (can handle more access requests); Step ②-5, perform deduplication and de-duplication operations on the optimal performance data obtained in step ②-4 to obtain the processed optimal performance data, and have experts evaluate each piece of processed optimal performance data to obtain the corresponding final score and use it as a data label. Each piece of optimal performance data and the corresponding data label form a data sample, and all data samples form an e-commerce flash sale application performance test data set; For example, a data sample is [CPU running time: 15ms, CPU utilization: 99%…, 8], where 8 represents the data label; Step ③: Input the e-commerce flash sale application performance test data set into the Transformer model to be trained to obtain the Transformer performance evaluation model, which is as follows: Step ③-1, randomly select 70% of the data samples in the e-commerce flash sale application performance test data set as the training set; Step ③-2, preprocessing the training set, specifically: first convert each data sample in the training set into corresponding text data, then convert each text data into a token sequence, and assign a unique ID to each token sequence to obtain the preprocessed training set; Step ③-3, set the hyperparameters for model training including: learning rate is 0.001, regularization parameter is 0.000001, and the number of encoder layers is 2; Step ③-4, input the preprocessed training set into the Transformer model to be trained; the Transformer model to be trained is constructed by the deep learning framework PyTorch, which is an existing technology, and the specific structure is as follows Figure 2 As shown in the figure, the encoder block is on the left and the decoder block is on the right. The multi-head attention layer is composed of multiple self-attention mechanisms. It can be seen that the encoder block contains one multi-head attention layer, and the decoder block contains two multi-head attention layers (one of which uses a mask). The residual connection above the multi-head attention layer is used to prevent network degradation, and layer normalization is used to normalize the activation values ​​of each layer. Step ③-5, according to the Huber Loss loss function, use the Adam optimizer to iteratively optimize the Transformer model to be trained until the conditions for stopping iteration are met, then stop the iteration process to obtain the Transformer performance evaluation model; the conditions for stopping iteration are that the iteration rounds reach 100, or the loss value does not improve after 5 consecutive iterations; the loss value is obtained by comparing the evaluation result output by the Transformer model with the data label; The Huber Loss loss function, also known as the smoothed L1 loss, is a loss function used for regression tasks. It introduces smoothness between the predicted value and the target value and is more robust to outliers than the mean square error (MSE). The main function of the Adam optimizer is to update the neural network parameters based on the gradient information to minimize the loss function. Step 2: When the e-commerce flash sale application to be evaluated has no interference traffic, the benchmark test system is started, a stress test is initiated, and the performance data of the computer system is obtained through the data acquisition and processing system, and the stress test is ended; the performance data includes: CPU running time, CPU utilization, disk input and output read and write rate, network input and output transmission rate, network error rate, memory usage, memory paging frequency, and the number of processes created per second; Step 3: Input the obtained performance data into the Transformer performance evaluation model to obtain a performance score; Step 4: Input the performance data and performance score into the report generation subsystem to generate and output the user report. The output user report supports multiple formats, such as txt text format, HTML web page format, csv data table format, etc. These formats are selectable and are obtained by concatenating Java language strings to obtain the corresponding three formats. The data source of the user report is the performance data obtained in step 2 and the performance score obtained in step 3. The user report can give both intuitive evaluation results and corresponding reference data, which is more objective.

[0019] In this embodiment, the data collection and processing system and the report generation subsystem are constructed based on the Java programming language. The Transformer performance evaluation model, the data collection and processing system, and the report generation subsystem are assembled into a system and debugged in conjunction with each other. User-accessible commands are configured for the debugged system to obtain a benchmark test system.

[0020] By combining the efficient development framework of Spring Boot and Spring Cloud, the construction of large-scale simulation test data sets, the training of Transformer performance evaluation models, the data collection and processing system of the Java language, and the integrated report generation subsystem, a highly modular, automated and user-friendly benchmarking system is implemented. It can accurately evaluate the performance of e-commerce flash sale applications, and has good scalability and real-time performance, providing users with an efficient, flexible and cost-effective testing solution.

Claims

1. A benchmarking method for e-commerce flash sales applications, characterized in that The following steps are involved: Step 1: deploying a benchmark test system in a computer system that has been running the e-commerce flash sale application to be evaluated, wherein the benchmark test system includes a data acquisition and processing system, a Transformer performance evaluation model, and a report generation subsystem; The Transformer performance evaluation model is obtained by the following method: Step 1: Develop a simulation application based on the e-commerce flash sale application based on the Spring Boot and Spring Cloud development framework; Step 2: Based on the simulation application, a performance test data set for e-commerce flash sales applications is constructed through a large number of simulations. Step ③, input the e-commerce flash sale application performance test data set into the Transformer model to be trained to obtain the Transformer performance evaluation model; Step 2: When the e-commerce flash sale application to be evaluated has no interference traffic, the benchmark test system is started, a stress test is initiated, performance data of the computer system is obtained through the data acquisition and processing system, and the stress test is ended; Step 3: Input the obtained performance data into the Transformer performance evaluation model to obtain a performance score; Step 4: Input the performance data and performance scores into the report generation subsystem to generate and output user reports.

2. A benchmarking method for e-commerce flash sales applications according to claim 1, characterized in that The data collection and processing system and the report generation subsystem are constructed based on the Java programming language.

3. A benchmarking method for e-commerce flash sales applications according to claim 1, characterized in that The specific operation process of step ① is as follows: Step ①-1: abstractly model the e-commerce flash sale application, extract the most essential features, and design a lightweight simulation application architecture based on the most essential features; Step ①-2, build a development framework based on Spring Boot and Spring Cloud based on Maven tools; Step ①-3, based on the development framework of Spring Boot and Spring Cloud, develop the lightweight simulation application architecture from bottom to top layer by layer to obtain a simulation application based on the e-commerce flash sale application: Step ①-3-1, develop data access layer code for accessing MySQL database and Redis cache; Step ①-3-2, develop business logic layer code to process the interaction between data returned from the control layer and data access layer data; Step ①-3-3, develop the controller layer code to process HTTP network access requests and pass the network data to the business logic layer to complete the simulated e-commerce flash sale business logic processing.

4. A benchmarking method for e-commerce flash sales applications according to claim 3, characterized in that The most essential features described include: the server can handle large concurrent traffic and data is stored in a relational database and access is accelerated through caching; The lightweight simulation application architecture includes a controller layer, a business logic layer and a data access layer which are connected in sequence. The controller layer includes an e-commerce product server and an e-commerce user server. The business logic layer includes an e-commerce Redis server and an e-commerce MySQL server. The data access layer includes a Redis cache and a MySQL database.

5. A benchmarking method for e-commerce flash sales applications according to claim 1, characterized in that The specific operation process of step ② is as follows: Step ②-1, use the JMeter tool to initiate intensive HTTP access requests to the target interface of the simulated application based on the e-commerce flash sale application, and perform stress testing; Step ②-2, during the stress test, the number of response threads in the stress test is continuously adjusted and recorded until request failures begin to occur, and the maximum number of response threads without request failures is obtained, and used as the maximum load capacity of the computer system for the e-commerce flash sale application in the current environment; Step ②-3, during the stress test, the performance data of the computer system at the maximum number of response threads is collected through the sar and perf tools, and the performance data and the corresponding maximum number of response threads are combined into an optimal performance data; Step ②-4, continuously adjust the resource configuration of the computer system where the e-commerce flash sale simulation application is located through the control variable method, and repeat steps ②-1 to ②-3 until a total of 10,000 optimal performance data are obtained; Step ②-5, deduplicate and remove empty data from the optimal performance data obtained in step ②-4 to obtain processed optimal performance data. Experts evaluate each piece of processed optimal performance data to obtain the corresponding final score as a data label. Each piece of optimal performance data and the corresponding data label are combined into a data sample, and all data samples are combined into an e-commerce flash sale application performance test data set.

6. A benchmarking method for e-commerce flash sales applications according to claim 5, characterized in that The specific operation process of step ③ is as follows: Step ③-1, randomly select 70% of the data samples in the e-commerce flash sale application performance test data set as the training set; Step ③-2, preprocessing the training set, specifically: first convert each data sample in the training set into corresponding text data, then convert each text data into a token sequence, and assign a unique ID to each token sequence to obtain the preprocessed training set; Step ③-3, set the hyperparameters for model training including: learning rate is 0.001, regularization parameter is 0.000001, and the number of encoder layers is 2; Step ③-4, input the preprocessed training set into the Transformer model to be trained for training; Step ③-5: According to the Huber Loss function, use the Adam optimizer to iteratively optimize the Transformer model to be trained until the conditions for stopping iteration are met. Then stop the iteration process and obtain the Transformer performance evaluation model.

7. A benchmarking method for e-commerce flash sales applications according to claim 6, characterized in that The condition for stopping iteration is that the iteration round reaches 100, or the loss value still does not improve after 5 consecutive iteration rounds. The loss value is obtained by comparing the evaluation result output by the Transformer model with the data label.

8. A benchmarking method for e-commerce flash sales applications according to claim 1, characterized in that The performance data includes: CPU running time, CPU utilization, disk input and output read and write rates, network input and output transmission rates, network error rates, memory usage, memory paging frequency and the number of processes created per second.

Images