Data processing link stress testing method, device and equipment and storage medium

Abstract

The present disclosure provides a data processing link stress testing method and device, equipment and storage medium, which relates to the technical field of computer. The method comprises the following steps: obtaining a first real-time data stream in a data stream processing service and partition information of the first real-time data stream in the data stream processing service; storing the first real-time data stream in a database according to the partition information to obtain historical simulation data; reading the historical simulation data from the database and sending the historical simulation data to a first topic of the data stream processing service to obtain a stress testing simulation data source; connecting the stress testing simulation data source to a to-be-tested link for data processing and storage operation to obtain stress testing performance data; and analyzing the stress testing performance data. The method improves the simulation degree of the stress testing data source and improves the stress testing effect.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and more specifically, to a data processing link stress testing method, apparatus, electronic device, and readable storage medium. Background Technology

[0002] Load testing is a process of simulating data volume peaks under certain conditions by increasing data traffic, thereby stress testing the system and verifying the task processing capabilities of the links in the system.

[0003] Related technologies employ a method of manually stopping the consumption of data streams on the streaming platform without halting its upstream production. Once a sufficient amount of data stream records have accumulated on the platform, a task processing program is initiated to consume the data, simulating the effect of a surge in demand within a short period. However, the data accumulated in this way consists of daily data, which deviates significantly from the data during actual peak sales periods in terms of data distribution and simulation accuracy. Therefore, using data accumulated in this way for load testing may not necessarily expose problems in the system's interconnectivity, resulting in poor load testing performance.

[0004] As mentioned above, improving the simulation level of the load testing data source to enhance the load testing effect is an urgent problem to be solved.

[0005] The information disclosed in the background section is only intended to enhance the understanding of the background of this disclosure, and therefore may include information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0006] The purpose of this disclosure is to provide a data processing link stress testing method, apparatus, electronic device, and readable storage medium, which at least to some extent improves the simulation level of the stress testing data source to improve the stress testing effect.

[0007] Other features and advantages of this disclosure will become apparent from the following detailed description, or may be learned in part from practice of this disclosure.

[0008] According to one aspect of this disclosure, a data processing link stress testing method is provided, comprising: acquiring a first real-time data stream in a data stream processing service and its partition information in the data stream processing service; dividing and storing the first real-time data stream in a database according to the partition information to obtain historical simulation data; reading the historical simulation data from the database and sending it to a first topic of the data stream processing service to obtain a stress testing simulation data source; connecting the stress testing simulation data source to the link under test for data processing and storage operations to obtain stress testing performance data; and analyzing the stress testing performance data.

[0009] According to one embodiment of this disclosure, the partition information of the first real-time data stream in the data stream processing service includes the partition identifier of the first real-time data stream in the data stream processing service; the first real-time data stream is divided and stored in the database according to the partition information to obtain historical simulation data, including: determining whether to perform partition expansion; if partition expansion is performed, obtaining the partition expansion coefficient; reading data from each partition of the first real-time data stream and writing it into the database to the partition expansion coefficient file corresponding to the partition identifier, wherein the historical simulation data includes the data in the database to the partition expansion coefficient file corresponding to the partition identifier.

[0010] According to one embodiment of this disclosure, the partition information of the first real-time data stream in the data stream processing service includes the number of partitions of the first real-time data stream in the data stream processing service; storing the first real-time data stream in a database according to the partition information to obtain historical simulation data further includes: if partition expansion is performed, creating multiple files in the database according to the number of partitions and the partition expansion coefficient, wherein the number of files created is obtained by multiplying the number of partitions by the partition expansion coefficient; after reading data from each partition of the first real-time data stream, writing it into the database into the number of files corresponding to the partition identifier and the partition expansion coefficient, including: after reading data from each partition of the first real-time data stream, concurrently writing it into the multiple files in the database.

[0011] According to one embodiment of this disclosure, the load testing simulation data source is connected to the link under test for data processing and storage operations to obtain load testing performance data. This includes: connecting the second real-time data stream in the data stream processing service and the load testing simulation data source to the link under test for data processing and storage operations to obtain corresponding real-time data processing results and load testing simulation data processing results; and obtaining performance data of the link under test during the data processing and storage operations on the second real-time data stream and the load testing simulation data source to obtain the load testing performance data.

[0012] According to one embodiment of this disclosure, reading the historical simulation data from the database and sending it to a first topic of the data stream processing service to obtain a stress test simulation data source includes: obtaining a preset unit read volume; reading the preset unit read volume of data from the historical simulation data at a preset read frequency and sending it to a first topic of the data stream processing service to obtain the stress test simulation data source.

[0013] According to one embodiment of this disclosure, reading historical simulation data from the database and sending it to a first topic of the data stream processing service to obtain a stress test simulation data source includes: acquiring a preset starting read volume, a preset increment read volume, and a preset upper limit read volume; reading data from the historical simulation data starting from the preset starting read volume, according to the preset increment read volume and a preset read frequency, and sending it to the first topic of the data stream processing service to obtain the stress test simulation data source, wherein the amount of data read does not exceed the preset upper limit read volume.

[0014] According to one embodiment of this disclosure, the link under test includes a task processing engine and a data storage engine; connecting the stress test simulation data source to the link under test for data processing and storage operations to obtain stress test performance data includes: connecting the stress test simulation data source to the task processing engine for data processing operations to obtain stress test simulation data task processing information; sending the stress test simulation data task processing information to the data storage engine for storage operations; and requesting the interfaces of the task processing engine and the data storage engine respectively via Hypertext Transfer Protocol to obtain performance data during the data processing operations of the task processing engine on the stress test simulation data source and the storage operations of the data storage engine on the stress test simulation data task processing information according to preset configurations, thereby obtaining the stress test performance data.

[0015] According to one embodiment of this disclosure, the pressure test performance data includes pressure test performance index values ​​and their corresponding times; analyzing the pressure test performance data includes: generating a pressure test performance curve with time as the horizontal axis and the pressure test performance index values ​​corresponding to time; performing at least one of the following inspection or detection operations based on the pressure test performance data: compliance inspection, average value inspection, peak value detection, threshold inspection, and outlier detection; generating and outputting a pressure test performance report based on the pressure test performance curve and the results of at least one of the inspection or detection operations.

[0016] According to one embodiment of this disclosure, the first real-time data stream includes high-frequency processed business data collected on a preset date.

[0017] According to another aspect of this disclosure, a data processing link stress testing apparatus is provided, comprising: an acquisition module for acquiring a first real-time data stream in a data stream processing service and its partition information in the data stream processing service; a data logging module for separating and storing the first real-time data stream in a database according to the partition information to obtain historical simulation data; a stress testing module for reading the historical simulation data from the database and sending it to a first topic of the data stream processing service to obtain a stress testing simulation data source; a processing module for connecting the stress testing simulation data source to the link under test for data processing and storage operations to obtain stress testing performance data; and an analysis module for analyzing the stress testing performance data.

[0018] According to another aspect of this disclosure, an electronic device is provided, comprising: a memory, a processor, and executable instructions stored in the memory and executable in the processor, wherein the processor, when executing the executable instructions, implements any of the methods described above.

[0019] According to another aspect of this disclosure, a computer-readable storage medium is provided that stores computer-executable instructions thereon, which, when executed by a processor, implement any of the methods described above.

[0020] The data processing link stress testing method provided in this disclosure acquires a first real-time data stream and its partition information in a data stream processing service. The first real-time data stream is then segmented and stored in a database according to the partition information to obtain historical simulation data. This historical simulation data is then read from the database and sent to a first topic of the data stream processing service to obtain a stress testing simulation data source. The stress testing simulation data source is then connected to the link under test for data processing and storage operations to obtain and analyze stress testing performance data. By segmenting and storing the real-time data stream in the real-time data stream processing service in a database before sending it to a data stream processing service topic to obtain a stress testing simulation data source, the simulation level of the stress testing data source can be improved, thereby enhancing the stress testing effect.

[0021] It should be understood that the above general description and the following detailed description are merely exemplary and do not limit this disclosure. Attached Figure Description

[0022] The above and other objects, features and advantages of this disclosure will become more apparent from a detailed description of exemplary embodiments thereof with reference to the accompanying drawings.

[0023] Figure 1 A flowchart of a data processing link stress testing method according to an embodiment of this disclosure is shown.

[0024] Figure 2 It shows Figure 1The step S104 shown is a schematic diagram of the processing procedure in one embodiment.

[0025] Figure 3 according to Figure 2 It shows Figure 1 The step S104 shown is a schematic diagram of the processing procedure in another embodiment.

[0026] Figure 4 It is based on Figure 2 and Figure 3 The diagram shown illustrates the number drop process.

[0027] Figure 5 It is based on Figure 4 The diagram shows the partition data and the corresponding written files.

[0028] Figure 6 It shows Figure 1 The step S106 shown is a schematic diagram of the processing procedure in one embodiment.

[0029] Figure 7 It shows Figure 1 The step S106 shown is a schematic diagram of the processing procedure in another embodiment.

[0030] Figure 8 It is based on Figure 6 and Figure 7 The diagram shows a processing flow diagram of a pressurization module.

[0031] Figure 9 It is based on Figure 8 The diagram shows a comparison of the effects of the two pressurization methods.

[0032] Figure 10 It shows Figure 1 The step S108 shown is a schematic diagram of the processing procedure in one embodiment.

[0033] Figure 11 It shows Figure 1 The step S108 shown is a schematic diagram of the processing procedure in another embodiment.

[0034] Figure 12 It shows Figure 1 The step S110 shown is a schematic diagram of the processing procedure in one embodiment.

[0035] Figure 13 It is based on Figures 10 to 12 The diagram illustrates a method for collecting and analyzing performance data during pressure testing.

[0036] Figure 14 It is based on Figures 1 to 13 The diagram shows an automated load testing system architecture for real-time links.

[0037] Figure 15 A block diagram of a data processing link stress testing device is shown in an embodiment of this disclosure.

[0038] Figure 16 A schematic diagram of the structure of an electronic device according to an embodiment of the present disclosure is shown. Detailed Implementation

[0039] Exemplary embodiments will now be described more fully with reference to the accompanying drawings. However, these exemplary embodiments can be implemented in many forms and should not be construed as limited to the examples set forth herein; rather, they are provided so that this disclosure will be more comprehensive and complete, and will fully convey the concept of the exemplary embodiments to those skilled in the art. The drawings are merely illustrative of this disclosure and are not necessarily drawn to scale. The same reference numerals in the drawings denote the same or similar parts, and therefore repeated descriptions of them will be omitted.

[0040] Furthermore, the described features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. Numerous specific details are provided in the following description to give a thorough understanding of embodiments of this disclosure. However, those skilled in the art will recognize that the technical solutions of this disclosure can be practiced with one or more of the specific details omitted, or other methods, apparatuses, steps, etc., can be employed. In other instances, well-known structures, methods, apparatuses, implementations, or operations are not shown or described in detail to avoid obscuring various aspects of this disclosure.

[0041] Furthermore, the terms "first," "second," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this disclosure, "multiple" means at least two, such as two, three, etc., unless otherwise explicitly specified. The symbol " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0042] In this disclosure, unless otherwise expressly specified and limited, the term "connection" and similar terms should be interpreted broadly, for example, it can refer to an electrical connection or the ability to communicate with each other; it can refer to a direct connection or an indirect connection through an intermediate medium. Those skilled in the art can understand the specific meaning of the above terms in this disclosure according to the specific circumstances.

[0043] The terminology used in the embodiments of this disclosure is explained below.

[0044] Kafka is an open-source stream processing platform developed by the Apache Software Foundation, written in Scala and Java. The project aims to provide a unified, high-throughput, low-latency platform for processing real-time data.

[0045] A topic in Kafka is a message queue, and each topic corresponds to a message queue.

[0046] Load testing: The process of testing and verifying the task's processing capacity by increasing the amount of data to simulate some data volume peaks.

[0047] Partitioning: Data in a topic is divided into groups according to certain rules.

[0048] Increase pressure: Increase the amount of data sent per unit time.

[0049] Computing engine: A program framework used to process real-time data. It is generally only responsible for computation and not for storage.

[0050] Storage engine: A medium used to store real-time data results obtained from calculations.

[0051] As mentioned above, in related technologies, manually stopping the consumption of the stream processing platform without stopping its upstream production results in the accumulation of data streams to simulate peak flows within the stream processing platform. However, the simulation level of the accumulated daily data is low, leading to poor load testing performance. Therefore, this disclosure provides a data processing link load testing method. By segmenting and storing the real-time data stream in the real-time data stream processing service in a database, and then sending it to the data stream processing service topic to obtain a load testing simulation data source, the simulation level of the load testing data source can be improved by using this load testing simulation data source to perform load testing on the link under test. This improves the load testing performance.

[0052] Figure 1 This is a flowchart illustrating a data processing link stress testing method according to an exemplary embodiment. (Reference) Figure 1 The method 10 provided in this embodiment may include the following steps.

[0053] In step S102, the first real-time data stream in the data stream processing service and its partition information in the data stream processing service are obtained.

[0054] In some embodiments, the data stream processing service can be a stream processing platform, such as Kafka or other stream processing platforms like Pulsar.

[0055] In some embodiments, the first real-time data stream includes high-frequency processing business data collected on a preset date. For example, the preset date may be a peak period for major promotional events, such as the day of the "618" or "Double Eleven" promotion. High-frequency processing business data may also include data from flash sales, limited-time offers, and popular products that are processed in large volumes within a short period.

[0056] In step S104, the first real-time data stream is divided and stored in the database according to the partition information to obtain historical simulation data.

[0057] In some embodiments, the partition information of the first real-time data stream in the data stream processing service may include the partition identifier of the first real-time data stream in the data stream processing service, so as to be stored separately in the database as historical simulation data corresponding to the partition identifier.

[0058] In some embodiments, the database can be a persistent storage database, which can permanently retain important, high-frequency processing business data for generating simulation data, which is then used in the subsequent pressure-up step. For example, the structure fields of the data storage can be described as shown in Table 1.

[0059] Table 1

[0060]

[0061]

[0062] In some embodiments, the simulation data writing function module can be used to connect the real-time data stream (i.e., the first real-time data stream) from the Kafka online network to the writing program. First, data cleaning can be performed according to user-defined rules. After cleaning, it can be determined whether to expand the number of writing partitions. If so, each partition will be expanded, meaning one partition will correspond to multiple files, and data from one partition in a Kafka topic will be written to different files. For detailed implementation methods, please refer to... Figure 2 .

[0063] In some embodiments, the partition information of the first real-time data stream in the data stream processing service may include the number of partitions of the first real-time data stream in the data stream processing service. After determining the partition expansion coefficient, multiple files can be created based on the number of partitions and the partition expansion coefficient for file counting. Specific implementation methods can be found in [reference needed]. Figures 3 to 5 .

[0064] In step S106, historical simulation data is read from the database and sent to the first topic of the data stream processing service to obtain the stress test simulation data source.

[0065] In some embodiments, historical simulation data can be sequentially read from the database by the pressure module and sent to the real-time stream of the Kafka-topic used for load testing simulation (i.e., the first topic of the data stream processing service) to obtain the load testing simulation data source so that the tasks on the real-time end-to-end chain under test can perform calculations.

[0066] In some embodiments, a pressure-increasing program can read a fixed number of real-time data streams per minute according to a preset data volume and send them to the first topic of the data stream processing service. Specific implementation details can be found in [reference needed]. Figure 6 .For example,

[0067] In other embodiments, starting from a preset initial read amount, data can be read at a preset read frequency with a preset increment step and sent to the first topic of the data stream processing service. Specific implementation details can be found in [reference needed]. Figure 7 .

[0068] In some embodiments, two pressurization methods can be provided, allowing for flexible selection of the appropriate pressurization method in practical applications. Specific implementation methods can be found in [reference needed]. Figure 8 and Figure 9 .

[0069] In step S108, the load test simulation data source is connected to the link under test for data processing and storage operations to obtain load test performance data.

[0070] In some embodiments, while connecting the simulated load test data source to the link under test, real online data can also be connected to the link under test and processed using the same logic. This allows for the updates of real-time data without interruption and without affecting service availability. For specific implementation details, please refer to [link to implementation details]. Figure 10 .

[0071] In some embodiments, the pipeline under test may include a task processing engine and a data storage engine. Performance data during the load test can be automatically obtained from the interface between the task processing engine (also known as the computing engine) and the data storage engine using a performance data collection and analysis program. Specific implementation details can be found in [reference needed]. Figure 11 .

[0072] In step S110, the pressure test performance data is analyzed.

[0073] In some embodiments, the stress test performance data includes stress test performance index values ​​and their corresponding times. The stress test performance data can be statistically analyzed to generate automated stress test reports. Specific implementation methods can be found in [reference needed]. Figure 12 .

[0074] According to the data processing link stress testing method provided in this disclosure, a first real-time data stream and its partition information in a data stream processing service are obtained. The first real-time data stream is then divided and stored in a database according to the partition information to obtain historical simulation data. This historical simulation data is then read from the database and sent to a first topic of the data stream processing service to obtain a stress testing simulation data source. This stress testing simulation data source is then connected to the link under test for data processing and storage operations to obtain and analyze stress testing performance data. This improves the simulation level of the stress testing data source, thereby enhancing the stress testing effect.

[0075] Figure 2 It shows Figure 1 The step S104 shown is a schematic diagram of the processing procedure in one embodiment. (See attached diagram.) Figure 2 As shown in the present embodiment, step S104 may further include the following steps.

[0076] Step S202: Determine whether to perform partition expansion.

[0077] Step S2042: If partition expansion is performed, obtain the partition expansion coefficient. This partition expansion coefficient is greater than 1.

[0078] Step S2044: If no partition expansion is performed, the partition expansion coefficient is obtained as 1.

[0079] Step S206: After reading data from each partition of the first real-time data stream, write the data into the partition expansion coefficient file corresponding to the partition identifier in the database. The historical simulation data includes the data in the partition expansion coefficient file corresponding to the partition identifier in the database.

[0080] According to the method provided in the embodiments of this disclosure, when the amount of data in each partition of some Kafka topics is particularly large, partition expansion can be configured to avoid the problem of slow message delivery speed caused by ensuring message order when only one file is written to each partition.

[0081] Figure 3 according to Figure 2 It shows Figure 1 The illustrated step S104 is a process diagram in another embodiment. (See diagram below.) Figure 3 As shown in the present embodiment, step S104 may further include the following steps.

[0082] Step S302: Determine whether to perform partition expansion.

[0083] Step S3042: If partition expansion is performed, obtain the partition expansion coefficient. This partition expansion coefficient is greater than 1.

[0084] Step S3044: If no partition expansion is performed, the partition expansion coefficient is obtained as 1.

[0085] Step S306: If the partition expansion coefficient is greater than 1, create multiple files in the database according to the number of partitions and the partition expansion coefficient. The number of files created is obtained by multiplying the number of partitions by the partition expansion coefficient.

[0086] In some embodiments, the number of partitions can be represented as N, and the partition expansion factor can be represented as M (M and N are both positive integers). Then the number of files created is N*M. For specific implementation details, please refer to [reference needed]. Figure 4 and Figure 5 .

[0087] Step S308: After reading data from each partition of the first real-time data stream, the data is concurrently written to multiple files in the database.

[0088] Figure 4 It is based on Figure 2 and Figure 3 The diagram shows the number drop process. (Example) Figure 4 As shown, the real-time link data streams 4002 from N partitions on the Kafka online are connected to the data delivery program (S402). The data delivery program first performs data cleaning according to the rules set by the user (S404). After cleaning, it can determine whether to expand the number of data delivery partitions (S406). If not, the partition expansion coefficient M = 1 (S4082); if partitioning, the partition expansion coefficient M > 1 (S4084). After setting the partition expansion coefficient M, N*M data delivery small files are created (S410). Then, each data delivery machine reads data from the corresponding partition and writes it concurrently (S412) to the corresponding file in the permanent repository 4004, improving the write speed and providing a foundation for subsequent load testing.

[0089] Figure 5 It is based on Figure 4 This diagram illustrates another type of partitioned data and its corresponding written file. (See example...) Figure 5 As shown, Figure 5 If the number of partitions is N+1 and the expansion factor is M+1, then the data in each partition from partition 0 to partition N corresponds to a group of (M+1) files. For example, the data in partition 0 corresponds to files 0-0, 0-1, ..., 0-M; the data in partition 1 corresponds to files 1-0, 1-1, ..., 1-M; ...; and the data in partition N corresponds to files N-0, N-1, ..., NM.

[0090] In some related technologies, when writing load test data, a single data entry is slightly modified, altering a few fields. Ten similar data entries based on this entry are then randomly written to a Kafka topic, effectively amplifying the data. However, this method of load testing by amplifying daily backlog data compromises fidelity and may not expose problems, thus failing to achieve the desired load testing effect.

[0091] This embodiment employs a permanent storage database to separate and store important simulation data for subsequent data compression, thus solving the problem of poor simulation results from stress test data.

[0092] Figure 6 It shows Figure 1 The step S106 shown in the diagram is a process flow diagram in one embodiment, illustrating a stable pressurization method. For example... Figure 6 As shown in the present embodiment, step S106 may further include the following steps.

[0093] Step S602: Obtain the preset unit read volume.

[0094] In some embodiments, the user can set a preset unit of reading volume, such as the amount of data that can be stably pressurized per minute, in units of 10,000 per minute.

[0095] Step S604: Read data of a preset unit read volume from historical simulation data at a preset read frequency and send it to the first topic of the data stream processing service to obtain the stress test simulation data source.

[0096] In some embodiments, before the transmission is completed, the pressure-boosting program can read a fixed number of real-time data entries per minute according to a preset unit read volume. After data cleaning and transformation, the data is sent to the downstream load testing simulation data source Kafka-topic for downstream real-time computing tasks to perform calculations.

[0097] Figure 7 It shows Figure 1 The step S106 shown in the diagram is a process flow diagram in another embodiment, illustrating an incremental pressurization method. (See diagram below.) Figure 7 As shown in the present embodiment, step S106 may further include the following steps.

[0098] Step S702: Obtain the preset starting read volume, the preset increment read volume, and the preset upper limit read volume.

[0099] In some embodiments, the user can set a preset starting read volume, a preset increment read volume, and a preset upper limit read volume, for example, the unit can be 10,000 / minute.

[0100] Step S704: Starting from a preset initial reading amount, read data from historical simulation data at a preset reading frequency with a preset incrementing step size and send it to the first topic of the data stream processing service to obtain the stress test simulation data source, wherein the amount of data read does not exceed a preset upper limit reading amount.

[0101] In some embodiments, after the pressure-increasing program starts, it first reads and sends data according to the preset initial read volume in the first minute; from the second minute onwards, it increases the corresponding preset increment every minute and reads and writes data according to the newly calculated value; until a certain minute, when the read volume per minute reaches or exceeds the preset upper limit read volume, the pressure-increasing program reads and writes data according to the preset upper limit read volume, and then does not change, until all data is pressure-increasing.

[0102] Figure 8 It is based on Figure 6 and Figure 7 The diagram illustrates the processing flow of a pressurization module. (For example...) Figure 8 As shown, after the pressure-increasing program starts (S802), it first reads the number of data drops from the permanent file library 8002 (S804), and then determines which pressure-increasing method to use (S806). If the stable pressure-increasing method is used, the pressure-increasing program first determines whether the data transmission has been completed (S8082). If it has, the process ends (S812). If it has not been completed, it can read a fixed number of real-time data drops per minute according to the user-set preset unit reading volume (S8084). After data cleaning and transformation, it is sent to the downstream load testing simulation data source Kafka-topic (S8086) for downstream real-time computing tasks to perform calculations. Then it returns to step S8082.

[0103] If incremental pressure is used, the pressure process first checks whether the data transmission has been completed (S8102). If it has, the process ends (S812). If it has not, it checks whether the current data read volume per minute exceeds the preset upper limit (S8104). If it does, it reads and writes data according to the preset upper limit (S8108) and writes data (S8110). If it does not exceed the limit, it increases the corresponding preset increment step size for reading (S8106) and then sends the data to the downstream load test simulation data source Kafka-topic (S8110). Then it returns to step S8102 to continue the judgment until all data has been pressure-pressurized.

[0104] Figure 9 It is based on Figure 8 The diagram shows a comparison of the effects of two pressurization methods. (See attached image.) Figure 9As shown, the horizontal axis represents time, for example, 20:00 to 20:10 in the figure; the vertical axis represents the amount of data read per minute, for example, 1 million to 8 million / minute in the figure. Among them, 902 is the relationship curve of incremental pressure mode, with a preset initial read volume of 1 million / minute, a preset increment read volume of 1 million / minute, and a preset upper limit read volume of 7 million / minute; 904 is the relationship curve of stable pressure mode, with a preset unit read volume of 5 million / minute.

[0105] Some related technologies employ a stress-inducing method that accumulates a certain amount of real-time messages before starting the real-time consumption program to simulate the stress test peak. This method results in a large number of messages arriving within a short period, leading to an excessively large stress test peak. When the simulated peak is reached, it puts significant pressure on the Flink computing task, potentially overwhelming downstream real-time tasks and causing them to crash, thus causing the stress test to fail. This poses a risk of overwhelming real-time computing tasks and negates the intended stress test effect. Furthermore, it makes it difficult to pinpoint the processing performance ceiling of real-time tasks.

[0106] According to the pressurization method adopted in the embodiments of this disclosure, the pressure can be gradually increased by incremental pressurization to avoid the excessive flow in the real-time stream from overwhelming the downstream real-time computing tasks. This can also help R&D personnel quickly locate the upper limit of the task's processing capacity. After locating the processing upper limit, a suitable stable reading volume can be selected to stably increase the pressure based on the actual situation.

[0107] Figure 10 It shows Figure 1 The step S108 shown is a schematic diagram of the processing procedure in one embodiment. (See attached diagram.) Figure 10 As shown in the present embodiment, step S108 may further include the following steps.

[0108] Step S1002: Connect the second real-time data stream and the stress test simulation data source in the data stream processing service to the link under test for data processing and storage operations, and obtain the corresponding real-time data processing results and stress test simulation data processing results.

[0109] In some embodiments, the second real-time data stream can be an online real-time data stream. In the Flink task of the real-time link under test, the online real-time data stream and the load test simulation data source can be processed using the same operators and the same processing logic. Then, the processing results can be distinguished by different version numbers or other mechanisms to avoid contaminating the accuracy of the online data.

[0110] Step S1004: Obtain performance data during the data processing and storage operations of the link under test on the second real-time data stream and the load test simulation data source, so as to obtain load test performance data.

[0111] The stress testing method using the aforementioned technologies requires pausing real-time computing tasks during the accumulation process, which will cause the online real-time data update to stop, resulting in data service unavailability. The timeliness and availability of the real-time link can only be restored after the data stress test is completed, which is very unfriendly to users.

[0112] The method used in this embodiment can process both online real-time data streams and load test simulation data sources simultaneously. Without compromising the accuracy of online data, it saves and updates data in real time during load testing, ensuring continuous service availability and improving user satisfaction and system availability.

[0113] Other related technologies involve setting up a simulated load testing environment with the same configuration as the production environment, and then separately testing the writing of load test data to Kafka topics within this simulated environment. Setting up a test environment requires significant resource investment, necessitating an identical set of machines and code redeployment, as well as substantial manpower. Furthermore, any modifications to the production environment code require corresponding adjustments to the test environment code, which is extremely time-consuming and labor-intensive. In some scenarios, although the test environment configuration is identical to the production environment, differences in network, data center, and deployment factors mean that the load testing performance of the test environment cannot be entirely equivalent to that of the production environment.

[0114] The method used in this embodiment utilizes an online environment as a load testing environment, which can reduce the machine and labor costs of building a load testing environment, while ensuring the effectiveness of load testing.

[0115] Figure 11 It shows Figure 1 The step S108 shown is a schematic diagram of the processing procedure in another embodiment. (See diagram below.) Figure 11 As shown in the present embodiment, step S108 may further include the following steps.

[0116] Step S1102: Connect the load test simulation data source to the task processing engine for data processing operations to obtain load test simulation data task processing information.

[0117] Step S1104: Send the stress test simulation data task processing information to the data storage engine for storage operation.

[0118] Step S1106: Using the Hyper Text Transfer Protocol (http), requests are made to call the interfaces of the task processing engine and the data storage engine respectively. According to the preset configuration, the performance data during the task processing engine's data processing operation on the load test simulation data source and the data storage engine's storage operation on the load test simulation data task processing information are obtained to obtain load test performance data.

[0119] In some embodiments, an Http-client can be used to call the data interface between the task processing engine and the data storage engine via an HTTP request. According to the information pre-configured by the user, the system requests and obtains performance monitoring data during the load test. The entire process simulates and replaces the manual process of requesting data from various system pages, thus realizing the automated acquisition of load test performance data.

[0120] Figure 12 It shows Figure 1 The step S110 shown is a schematic diagram of the processing procedure in one embodiment. (See attached diagram.) Figure 12 As shown in the embodiments of this disclosure, step S110 may further include the following steps.

[0121] Step S1202: Generate a pressure test performance curve with time as the horizontal axis and the pressure test performance index value corresponding to time.

[0122] Step S1204: Perform at least one of the following inspection or testing operations based on the pressure test performance data: compliance inspection, average value inspection, peak value detection, threshold inspection, and outlier detection.

[0123] Step S1206: Generate and output a pressure test performance report based on the pressure test performance curve and the results of at least one inspection or testing operation.

[0124] In some embodiments, after obtaining performance monitoring data from various engines such as the task processing engine and the data storage engine, the analysis system can perform at least one of the following tests or detection operations based on the performance monitoring data: compliance test, average value test, peak detection, threshold test, and outlier detection. Alternatively, it can perform all tests or detection operations sequentially. The implementation steps for each test or detection operation are shown in Table 2.

[0125] Table 2

[0126]

[0127] Figure 13 It is based on Figures 10 to 12 The diagram shown illustrates a method for collecting and analyzing pressure test performance data, which can be implemented using the pressure test performance data collection and analysis module 13006. For example... Figure 13 As shown, the core of the collection and analysis module 13006 is an HTTP client 130062, which can call the data interface between the computing engine monitoring system 13002 and the storage engine monitoring system 13004 via HTTP requests, and request and obtain the performance monitoring data (performance data) 13000 during the stress test according to the information pre-configured by the user.

[0128] After obtaining the performance monitoring data 13000 of each engine, the analysis system 130064 can perform the following steps in sequence based on the performance monitoring data 13000: whether the standard is met (S1302), average value test (S1304), peak detection (S1306), threshold test (S1308), and outlier detection (S1310), and draw the analysis conclusion 1300642.

[0129] After analysis, the plotting system 130066 can use JFreeChart to plot a line chart 1300662 based on the performance data curve, with time as the X-axis and performance data as the Y-axis. This chart, along with the analysis conclusions 1300642, is then sent to the report generation system 130068 to generate an automated stress test performance report 1300682 for output, making it easier for R&D personnel to review and locate problems.

[0130] In related technologies, performance data for the compute engine and data storage engine is scattered. Compute engine performance data is displayed on the Flink UI, while performance data for storage engines such as Redis, Elasticsearch, HBase, and Druid are scattered across their respective monitoring system pages. Furthermore, the data metrics are relatively fixed and lack features such as peak analysis and threshold-exceeding annotations. After load testing, developers need to manually calculate the average, peak, dispersion, and threshold levels of various metrics on the monitoring system pages of each compute and storage engine, identifying any abnormal behavior. This process is extremely labor-intensive and lacks automation; moreover, manual statistical analysis often results in inaccurate data.

[0131] The method used in this embodiment collects and analyzes the performance parameters of the computing engine and storage engine during stress testing, and generates a unified stress test report, which solves the problems of scattered performance data distribution and lack of automation in the statistical and analysis process.

[0132] Figure 14 It is based on Figures 1 to 13 The diagram illustrates a system architecture for automated load testing of real-time links. (See diagram for example.) Figure 14 As shown, the entire system can be divided into three major modules: simulation data drop area 14002, pressurization area 14004, and task performance statistics area 14006.

[0133] In the simulation data dropout area 14002, the real data 1402 in the real-time calculation pipeline Kafka for a specific date (e.g., the day of a major promotion) can be stored in the permanent storage library 1406 through the dropout program 1404, and separated according to the Kafka partition 14062 for later simulation pressure.

[0134] In pressure zone 14004, historical simulation data in the permanent storage library 1406 can be read through pressure program 1408. Data is simultaneously read from multiple pressure machines 14082 (e.g., pressure machines 0 to N corresponding one-to-one with partitions 0 to N) according to preset partitioning rules. After data cleaning and transformation, the data is sent to the Kafka-topic 1410 used for load testing simulation. The Flink computing task 1414 on the real-time link simultaneously connects to the online real-time Kafka data source 1412 and the data source of the load testing simulation Kafka-topic 1410. In the Flink technical task 1414 on the real-time link, the online real data and load testing data follow the same processing logic in the same operators, but the final results are distinguished through mechanisms such as different version numbers to avoid contaminating the accuracy of the online data.

[0135] In the task performance statistics area 14006, the performance data collection and analysis program 1418 first automatically obtains the performance data 14142 of the computing engine and the performance data 1416 of the storage engine during the load test from the interface between the computing engine 1414 and the storage engine 1416. Then, it performs statistics and analysis to generate an automated load test report 1420, which is convenient for developers to review and quickly locate problems.

[0136] The method provided in this disclosure automates real-time computing link stress testing through modules such as simulation data collection, multi-functional stress testing, and performance data collection and analysis. It connects the entire process, from data preparation, data stress testing, data isolation, performance data collection, and analysis reporting, reducing the investment of human and machine resources, maximizing the stress testing effect and simulation accuracy, and improving the overall automation level. In terms of data preparation, data collection ensures the simulation accuracy of the data; in terms of stress testing, it provides both stable stress testing and gradual stress testing methods to help locate the processing capacity limit; by modifying the task and simultaneously accessing two data sources, it uses an online environment to ensure the stress testing effect of the task processing capacity, avoiding the failure to expose problems due to different environments; in terms of data collection and analysis, it automatically collects performance data by calling HTTP interfaces, performs data analysis according to pre-defined rules, automatically labels problems, and generates a stress testing report.

[0137] Figure 15 This is a block diagram illustrating a data processing link stress testing apparatus according to an exemplary embodiment. Figure 15 The device shown can be applied, for example, to... Figure 14 The system shown.

[0138] refer to Figure 15The apparatus 150 provided in this embodiment may include an acquisition module 1502, a counting module 1504, a pressurization module 1506, a processing module 1508, and an analysis module 1510.

[0139] The acquisition module 1502 can be used to acquire the first real-time data stream in the data stream processing service and its partition information in the data stream processing service.

[0140] The partition information of the first real-time data stream in the data stream processing service may include the partition identifier of the first real-time data stream in the data stream processing service.

[0141] The partition information of the first real-time data stream in the data stream processing service may also include the number of partitions of the first real-time data stream in the data stream processing service.

[0142] The first real-time data stream may include high-frequency processed business data collected on a preset date.

[0143] The data acquisition module 1504 can be used to divide and store the first real-time data stream in the database according to the partition information to obtain historical simulation data.

[0144] The data acquisition module 1504 can also be used to: determine whether to perform partition expansion; if partition expansion is performed, obtain the partition expansion coefficient; read data from each partition of the first real-time data stream and write it into the database to the partition expansion coefficient file corresponding to the partition identifier; the historical simulation data includes the data in the partition expansion coefficient file corresponding to the partition identifier in the database.

[0145] The data entry module 1504 can also be used to: if partition expansion is performed, create multiple files in the database according to the number of partitions and the partition expansion coefficient, the number of files created being obtained by multiplying the number of partitions by the partition expansion coefficient; and read data from each partition of the first real-time data stream and write it concurrently to multiple files in the database.

[0146] The pressure module 1506 can be used to read historical simulation data from the database and send it to the first topic of the data stream processing service to obtain the stress test simulation data source.

[0147] The pressure module 1506 can also be used to: obtain a preset unit read volume; read data of the preset unit read volume from historical simulation data at a preset read frequency and send it to the first topic of the data stream processing service to obtain the stress test simulation data source.

[0148] The pressure module 1506 can also be used to: obtain a preset starting read value, a preset increment read value, and a preset upper limit read value; read data from historical simulation data starting from the preset starting read value, according to the preset increment read value and a preset read frequency, and send the data to the first topic of the data stream processing service to obtain the stress test simulation data source, wherein the amount of data read does not exceed the preset upper limit read value.

[0149] The processing module 1508 can be used to connect the load test simulation data source to the link under test for data processing and storage operations in order to obtain load test performance data.

[0150] The pipeline under test may include a task processing engine and a data storage engine.

[0151] The stress test performance data can include stress test performance index values ​​and their corresponding time.

[0152] The processing module 1508 can also be used to: connect the second real-time data stream and the stress test simulation data source in the data stream processing service to the link under test for data processing and storage operations, and obtain the corresponding real-time data processing results and stress test simulation data processing results; and obtain the performance data of the link under test during the data processing and storage operations of the second real-time data stream and the stress test simulation data source, so as to obtain stress test performance data.

[0153] The processing module 1508 can also be used to: connect the load testing simulation data source to the task processing engine for data processing operations to obtain load testing simulation data task processing information; send the load testing simulation data task processing information to the data storage engine for storage operations; and request the interfaces of the task processing engine and the data storage engine respectively through the Hypertext Transfer Protocol to obtain the performance data during the data processing operations of the task processing engine on the load testing simulation data source and the storage operations of the data storage engine on the load testing simulation data task processing information, respectively, according to the preset configuration, so as to obtain load testing performance data.

[0154] Analysis module 1510 can be used to analyze pressure test performance data.

[0155] Analysis module 1510 can also be used to generate a pressure test performance curve with time as the horizontal axis and pressure test performance index values ​​corresponding to time; perform at least one of the following inspection or testing operations based on the pressure test performance data: compliance inspection, average value inspection, peak value detection, threshold inspection, and outlier detection; and generate and output a pressure test performance report based on the pressure test performance curve and the results of at least one inspection or testing operation.

[0156] The specific implementation of each module in the apparatus provided in this embodiment can be referred to the content of the above method, and will not be repeated here.

[0157] Figure 16A schematic diagram of the structure of an electronic device according to an embodiment of this disclosure is shown. It should be noted that... Figure 16 The devices shown are merely examples of computer systems and should not be construed as limiting the functionality and scope of use of the embodiments disclosed herein.

[0158] like Figure 16 As shown, device 1600 includes a central processing unit (CPU) 1601, which can perform various appropriate actions and processes according to a program stored in read-only memory (ROM) 1602 or a program loaded from storage section 1608 into random access memory (RAM) 1603. RAM 1603 also stores various programs and data required for the operation of device 1600. CPU 1601, ROM 1602, and RAM 1603 are interconnected via bus 1604. Input / output (I / O) interface 1605 is also connected to bus 1604.

[0159] The following components are connected to I / O interface 1605: an input section 1606 including a keyboard, mouse, etc.; an output section 1607 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.; a storage section 1608 including a hard disk, etc.; and a communication section 1609 including a network interface card such as a LAN card, modem, etc. The communication section 1609 performs communication processing via a network such as the Internet. A drive 1610 is also connected to I / O interface 1605 as needed. Removable media 1611, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., are installed on drive 1610 as needed so that computer programs read from them can be installed into storage section 1608 as needed.

[0160] In particular, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 1609, and / or installed from removable medium 1611. When the computer program is executed by central processing unit (CPU) 1601, it performs the functions defined above in the system of this disclosure.

[0161] It should be noted that the computer-readable medium disclosed herein may be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium may be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this disclosure, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media can also be any computer-readable medium other than computer-readable storage media, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wireless, wire, optical fiber, RF, etc., or any suitable combination thereof.

[0162] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0163] The modules described in the embodiments of this disclosure can be implemented in software or hardware. The described modules can also be housed in a processor; for example, a processor can be described as including an acquisition module, a data acquisition module, a pressure processing module, a processing module, and an analysis module. The names of these modules do not necessarily limit the module itself; for example, the acquisition module can also be described as a "module for acquiring real-time data streams."

[0164] In another aspect, this disclosure also provides a computer-readable medium, which may be included in the device described in the above embodiments; or it may exist independently and not assembled into the device. The computer-readable medium carries one or more programs, which, when executed by the device, cause the device to include:

[0165] Acquire the first real-time data stream in the data stream processing service and its partition information in the data stream processing service; divide and store the first real-time data stream in the database according to the partition information to obtain historical simulation data; read the historical simulation data from the database and send it to the first topic of the data stream processing service to obtain the load test simulation data source; connect the load test simulation data source to the link under test for data processing and storage operations to obtain load test performance data; analyze the load test performance data.

[0166] Exemplary embodiments of this disclosure have been specifically shown and described above. It should be understood that this disclosure is not limited to the detailed structures, arrangements, or implementations described herein; rather, this disclosure is intended to cover various modifications and equivalent arrangements contained within the spirit and scope of the appended claims.

Claims

1. A data processing link stress testing method, characterized in that, include: Obtain a first real-time data stream in the data stream processing service and its partition information in the data stream processing service. The partition information of the first real-time data stream in the data stream processing service includes the partition identifier and the number of partitions N of the first real-time data stream in the data stream processing service, where N is a positive integer. The first real-time data stream is divided and stored in the database according to the partition information to obtain historical simulation data, which is then divided and stored in the database in accordance with the partition identifier as the historical simulation data. The historical simulation data is read from the database and sent to the first topic of the data stream processing service to obtain the stress test simulation data source; The simulated load test data source is connected to the link under test for data processing and storage to obtain load test performance data. The pressure test performance data was analyzed; The first real-time data stream is divided and stored in the database according to the partition information to obtain historical simulation data, including: Determine whether to perform partition expansion; If partition expansion is performed, obtain the partition expansion coefficient; After reading data from the N partitions of the first real-time data stream, the data is concurrently written to the partition expansion coefficient files corresponding to the partition identifiers in the N partitions of the database. The historical simulation data includes the data in the partition expansion coefficient files corresponding to the partition identifiers in the database.

2. The method according to claim 1, characterized in that, The process of dividing and storing the first real-time data stream in a database according to the partition information to obtain historical simulation data further includes: If partition expansion is performed, multiple files are created in the database based on the number of partitions and the partition expansion coefficient. The number of files created is obtained by multiplying the number of partitions by the partition expansion coefficient.

3. The method according to claim 1, characterized in that, The stress test simulation data source is connected to the link under test for data processing and storage to obtain stress test performance data, including: The second real-time data stream in the data stream processing service and the stress test simulation data source are connected to the link under test for data processing and storage operations to obtain the corresponding real-time data processing results and stress test simulation data processing results. The performance data of the link under test during the data processing and storage operations of the second real-time data stream and the stress test simulation data source are obtained to obtain the stress test performance data.

4. The method according to claim 1, characterized in that, The historical simulation data is read from the database and sent to the first topic of the data stream processing service to obtain the stress test simulation data source, including: Get the preset unit read volume; The preset unit read volume of data is read from the historical simulation data at a preset read frequency and sent to the first topic of the data stream processing service to obtain the stress test simulation data source.

5. The method according to claim 1, characterized in that, The historical simulation data is read from the database and sent to the first topic of the data stream processing service to obtain the stress test simulation data source, including: Get the preset starting read value, the preset increment read value, and the preset maximum read value; Starting from the preset initial reading amount, data is read from the historical simulation data at a preset reading frequency with a preset incrementing step size and sent to the first topic of the data stream processing service to obtain the stress test simulation data source, wherein the amount of data read does not exceed the preset upper limit reading amount.

6. The method according to claim 1, characterized in that, The link under test includes a task processing engine and a data storage engine; The stress test simulation data source is connected to the link under test for data processing and storage to obtain stress test performance data, including: The stress test simulation data source is connected to the task processing engine for data processing operations to obtain stress test simulation data task processing information. The stress test simulation data task processing information is sent to the data storage engine for storage operation; By requesting the interface of the task processing engine and the interface of the data storage engine through the Hypertext Transfer Protocol, the performance data during the time when the task processing engine performs data processing operations on the load test simulation data source and the time when the data storage engine performs storage operations on the load test simulation data task processing information are obtained according to the preset configuration, so as to obtain the load test performance data.

7. The method according to claim 1, characterized in that, The pressure test performance data includes pressure test performance index values ​​and their corresponding times; The stress test performance data is analyzed, including: A pressure test performance curve is generated with time as the horizontal axis and the pressure test performance index values ​​corresponding to time. Perform at least one of the following inspection or testing operations based on the pressure test performance data: compliance inspection, average value inspection, peak value detection, threshold inspection, and outlier detection. A pressure test performance report is generated and output based on the pressure test performance curve and the results of at least one of the inspection or testing operations.

8. The method according to claim 1, characterized in that, The first real-time data stream includes high-frequency processed business data collected on a preset date.

9. A data processing link stress testing device, characterized in that, include: The acquisition module is used to acquire a first real-time data stream in the data stream processing service and its partition information in the data stream processing service. The partition information of the first real-time data stream in the data stream processing service includes the partition identifier and the number of partitions N of the first real-time data stream in the data stream processing service, where N is a positive integer. The data entry module is used to divide and store the first real-time data stream in the database according to the partition information to obtain historical simulation data, and to divide and store the historical simulation data in the database in accordance with the partition identifier. The data entry module is also used to determine whether to perform partition expansion; if partition expansion is performed, the partition expansion coefficient is obtained; after reading data from the N partitions of the first real-time data stream, the data is concurrently written to the partition expansion coefficient files corresponding to the partition identifier in the N partitions of the database; the historical simulation data includes the data in the partition expansion coefficient files corresponding to the partition identifier in the database. The pressure module is used to read the historical simulation data from the database and send it to the first topic of the data stream processing service to obtain the stress test simulation data source; The processing module is used to connect the stress test simulation data source to the link under test for data processing and storage operations to obtain stress test performance data; The analysis module is used to analyze the pressure test performance data.

10. An electronic device, comprising: A memory, a processor, and executable instructions stored in the memory and executable in the processor, characterized in that the processor, when executing the executable instructions, implements the method as described in any one of claims 1-8.

11. A computer-readable storage medium having computer-executable instructions stored thereon, characterized in that, When the executable instructions are executed by the processor, they implement the method as described in any one of claims 1-8.

Images