A performance testing method for real-time streaming computing

Abstract

The present invention discloses a performance testing method for real-time streaming computing, including a performance testing environment for real-time streaming computing, a performance testing scenario for real-time streaming computing, simulation of real-time streaming data, writing producer classes for various business scenarios, a method for preparing test data for real-time streaming computing, stress testing script preparation, implementation of the stress testing process for real-time streaming computing, and output of a test report. The method is a comprehensive and complete performance testing solution for real-time streaming computing. Compared with the prior art, the method simulates real-time streaming data of different scenarios through code, has stronger flexibility and reusability, and is combined with stress testing tools to complete the performance testing of real-time streaming computing.

Description

Technical Field

[0001] This patent is a performance testing method applied to real-time streaming computing, which involves the fields of computer technology and Internet testing. Background Art

[0002] The era of big data has brought people a rich and colorful lifestyle, allowing people to fully enjoy the value mined from big data. However, it is precisely because big data is generated too much and too fast that we currently have two main ways to process data: offline computing and real-time computing, and the technical features are divided into batch processing and streaming. The value of real-time streaming computing lies in the ability to quickly obtain business results, analyze and mine the real-time value of data after the data is generated. For real-time computing, once the data cannot be processed in real-time and the results are generated immediately, the value of the data will decrease over time.

[0003] There are many application scenarios for real-time streaming computing, such as real-time data warehouse, real-time risk control, real-time recommendation, and real-time machine learning. With the emergence of real-time computing frameworks such as Storm, Spark Streaming, and Flink, and the rise of Kafka and ES, the technology in the field of real-time computing has become more and more perfect, making real-time streaming computing play an increasingly important role in the financial industry, online education industry, e-commerce industry, advertising industry and other fields.

[0004] The main characteristics of real-time streaming data are: 1) Real-time, the data processed by real-time computing has real-time value, which requires that the computing delay must be small; 2) Randomness, the data flow is random in time and quantity, the flow may decrease in a period of time, and may increase in a period of time; 3) Disorder, streaming data is a sequence of events about time, and it is difficult to guarantee global order in a strict sense, or even almost impossible. We can make the data orderly in a local time window. In the current mainstream real-time streaming computing framework, the common practice is to distribute the received events to each time window shard according to the timestamp, and then trigger the unified processing operation of the data in the time window shard after waiting for a period of time; 4) Infinity, streaming data has no concept of "each time", it is always generated, endlessly. In terms of system architecture, the infinity of real-time streaming data requires the system to have high availability and real-time processing capabilities. When the system processing capacity cannot keep up with the speed of data flow generation, the messages to be processed will accumulate. When the backlog exceeds the threshold, the system with limited storage space will inevitably crash. In order to eliminate the existing backlog of messages, the system processing capacity must exceed the rate at which data is generated, otherwise the backlog will continue to exist.

[0005] Based on the characteristics of real-time streaming data, in addition to ensuring the accuracy of functional logic, testers must perform performance testing on real-time streaming computing programs, obtain indicators of the performance capacity of real-time computing programs, provide performance test reports, and provide developers with tuning references, ultimately providing reliable and stable services that meet online needs. Summary of the invention

[0006] The present invention provides a performance testing method applied to real-time streaming computing, which simulates data in different scenarios through code and completes performance testing with the help of a stress testing tool.

[0007] The sources of real-time streaming data vary, and the most common ones are log collection, such as collecting point data through filebeat, collecting database incremental log binlog through canal, and then outputting the collected data to the Kafka distributed messaging system. The real-time computing program then consumes and processes the data in Kafka, and stores it in databases such as redis, es, kudu, hive, and clinkhouse after processing.

[0008] The performance test method of real-time streaming computing requires simulating a large amount of data. There are two methods:

[0009] 1. Connect part or all of the online data to the test environment while communicating with the online environment. The advantages of this method are: the data is real. Disadvantages: First, data security issues; its performance results; and like buried data, this method is difficult to collect data statistics.

[0010] 2. Simulate data by writing code, such as Java, Python, etc., and send data by writing multiple threads. The advantage of this method is that it is more flexible and can simulate data for all business scenarios; the disadvantage is that it requires a high level of code for testers, cannot be visualized, and is difficult to control the pressure.

[0011] The present invention provides a performance testing method for real-time streaming computing, and the solution can solve the above problems. The data of different scenarios are simulated by code, and the performance test is completed with the help of a stress testing tool. The advantages of this method are: 1) high flexibility, familiar programming languages ​​​​such as Java and Python can be used to simulate data of different scenarios through code; 2) The stress testing tool is visualized, very friendly, and has complete functions. It can set thread groups, control the size of pressure, parameterize, and assert return results. Even multiple machines can be stressed, server resources can be monitored, test reports can be generated, etc., such as open source tools Apache JMeter (corresponding to Java language) and Locust (corresponding to Python). BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1: Stress test physical diagram

[0013] Figure 2 :Test process diagram DETAILED DESCRIPTION

[0014] 1. Prepare the test environment

[0015] Prepare a real-time streaming computing test environment, prepare servers, and deploy a performance testing environment, including stress testing tools, message middleware, real-time streaming computing programs, cache clusters, es middleware, databases, etc. A clean stress testing environment is best.

[0016] 2. Test scenario design

[0017] Design test scenarios for real-time streaming computing, i.e. test cases. Consider the following aspects: 1) Consider business scenarios, such as the deliveryman's real-time data dashboard, such as the number of completed orders, canceled orders, income, good reviews, bad reviews, etc. Then, we need to simulate the following scenarios: take one order->deliveryman grabs the order->completes the order->evaluation, take multiple orders->dispatch the order->transfer the order->completes the order, and cancel the order, etc., and design single transaction load, mixed stress test, etc.; 2) Consider the test plan, stress test, reliability test, stability test, scalability test, cluster effectiveness, etc.; 3) Consider the characteristics of streaming computing, single parallelism, multiple parallelism.

[0018] 3. Simulate real-time streaming data and write producer classes (written in Java)

[0019] 1) Introduce corresponding dependency packages, mainly the dependency packages of the message system Kafka and the stress testing tool; simulate data changes in different business scenarios and send them to the message system, and use different classes to manage different business scenarios;

[0020] 2) When writing a producer class, you must first ensure that your program has no performance issues. Use the singleton mode and do not frequently create and destroy objects when using a stress testing tool for multi-threaded stress testing, thereby reducing memory usage.

[0021] 3) Rewrite the method of the stress testing tool. The data should be as real as possible, and variables and parameterized fields should be reserved, such as Kafka's IP, topic and other variables, as well as parameterized fields in sent messages.

[0022] 4) Generate jar package

[0023] 4. Prepare test data

[0024] It includes two parts: 1) parameterized data in the stress testing script. This part is mainly based on the fields that need to be parameterized in the producer class, which makes the simulated data more realistic. 2) the data volume of the database that affects the performance. For example, if the embedded data is written into the Kudu table through real-time calculation, it is necessary to simulate Kudu tables of different data volumes, which will affect the performance of inserting and querying data.

[0025] Creating different amounts of database test data depends on the specific database. There are several ways: 1) directly write code to insert data. Although most databases provide methods for batch generating data, this method is very time-consuming to create billions of data. 2) insert by querying other tables, or insert a part of the data first. Then increase the amount of data exponentially through Cartesian product. This method is very efficient, but it is only applicable to some databases.

[0026] 5. Write stress testing scripts

[0027] In the stress testing tool, import the jar package, select related classes according to requirements in different requests, and simulate different scenarios. Add parameter files required for parameterization. Add a timer to control the size of QPS based on the test scenario in the test plan.

[0028] 6. Real-time computing stress testing process

[0029] In the test environment preparation stage, prepare the corresponding stress test server and the corresponding version of the stress test tool on the server. In view of the actual situation of real-time computing, the stress test server and the program under test should be in the same intranet environment to avoid inaccurate performance results caused by network overhead. Before stress testing, deploy the script and parameter files, including the jar package used in the script, on the stress test server.

[0030] When executing the test scenario, do the following things during stress testing: 1) Monitor the corresponding resource consumption, including all servers on the stress testing end and the end being stress tested, mainly monitoring server resources such as cpu, memory, IO, and network. Real-time computing involves servers, cache servers, and storage servers for spark and flink computing, which can be obtained through resource monitoring platforms or tools such as nmon; 2) Check whether real-time computing can process messages stably, by writing code (java, python), monitoring the lag value of kafka, obtaining it once per second, and obtaining a curve chart of program consumption messages within the time period; 3) Check the taskmanager log of the real-time computing program to see if there is an error log.

[0031] 7. Real-time calculation of stress test results

[0032] The performance results of real-time computing cannot be obtained directly through stress testing tools. The stress testing process only writes data to Kafka under a certain pressure. Real-time computing consumes data from Kafka and finally outputs the results after calculation through the program. One way is to display it in the form of indicators, such as the number of completed orders, number of canceled orders, and revenue in the real-time data dashboard of the deliveryman. When there is information such as orders and revenue, the real-time computing program completes the calculation and stores it in the database or cache, and then writes an interface call to display it to the deliveryman. The accuracy rate is to compare the results of the indicators with the simulated data to confirm the data. The performance results are counted through logs to obtain the processing time. Another way is to write directly into a large data table, such as a real-time data warehouse and a tracking system. The accuracy of the data can be checked by comparing the number of entries in the large data table and the data of some events. The processing volume per second, i.e., QPS, is obtained by dividing the total number of data entries by the time difference between the writing time of the last record and the writing time of the first record.

[0033] If the stress test results do not meet expectations, consider the following aspects: 1) First analyze the script you wrote to see if there are any problems with the script and parameters; 2) Analyze the returned results to see if there are any errors, and check the log of the real-time computing program to see if there are any errors; 3) The resource consumption of the machine that issues the stress test can be improved by adding a proxy machine; 4) Analyze the stress test phenomenon in detail, such as the gradient curve.

[0034] If the stress test results meet expectations, a performance test report for real-time computing will be prepared, which includes the following parts: 1) Description of the test results, including the stress test version, server configuration, and comparison between the test and production servers; 2) Test results, stress test results for each test scenario, and performance indicators such as total processing, number of concurrent connections, response time, QPS, and success rate; 3) Server resource usage report for each scenario; 4) Performance test conclusions, which will be given when a new service is launched.

Claims

1. A performance testing method for real-time streaming computing, comprising: Test environment for real-time streaming computing, prepare servers, and deploy performance testing environment, including stress testing tools, message middleware, real-time streaming computing programs, cache clusters, es middleware, and databases; Test scenarios for real-time streaming computing,1) Considering business scenarios; 2) Considering the test plan, stress test, reliability test, and stability test; 3) Considering the characteristics of streaming computing, single parallelism and multiple parallelism; Simulate real-time streaming data, write producer classes for various business scenarios, combine the dependency packages provided by the stress testing tool, rewrite the methods of the stress testing tool, and reserve parameterized fields; Prepare test data for real-time streaming computing: 1) parameterized data in the stress testing script; 2) The amount of database data that affects performance; Prepare stress testing scripts, select relevant classes according to requirements in different requests, simulate different scenarios, and add parameter files required for parameterization; Implement the stress testing process of real-time streaming computing, execute test scenarios, monitor and record the resources of each server, and monitor the logs of real-time computing programs; Troubleshooting and tuning: reusable stress testing scripts and data to assist developers in troubleshooting and tuning real-time program problems; Output test reports, test result descriptions, test results, resource consumption reports, and performance test conclusions to provide performance references for online real-time computing programs.

2. The performance testing method for real-time streaming computing as described in claim 1 is a comprehensive and complete performance testing solution for real-time streaming computing and a solution for obtaining the performance capacity of streaming computing programs.

3. The performance testing method of real-time streaming computing according to claim 2, characterized in that: It also includes methods for designing stress testing scenarios for real-time streaming computing, methods for simulating real-time streaming data in different scenarios through coding, methods for parameterizing data, writing performance testing scripts, the execution process, and methods for obtaining the final real-time computing performance test results.

4. The performance testing method of real-time streaming computing according to claim 3, characterized in that: It also includes writing code in any coding language to simulate real-time streaming data in different business scenarios, and combining it with stress testing tools to conveniently implement stress in various test scenarios.

Images