Concurrent traffic monitoring method and apparatus, terminal device, and storage medium

Abstract

This application discloses a concurrent traffic monitoring method, apparatus, terminal device, and storage medium. The concurrent traffic monitoring method includes: acquiring application consumption logs; transforming and filtering the consumption logs to obtain corresponding event data streams; and performing second-level processing on the event data streams based on a preset second-level processing window to obtain target concurrent data. Based on this application's solution, using the consumption logs of microservice applications as a data source can solve the technical problem of low security in high-concurrency traffic monitoring, avoiding modification or intrusion into business code. By transforming and filtering the consumption logs to obtain event data streams, and then combining this with a second-level processing window to perform second-level processing, the second-level concurrent traffic of the microservice application can be obtained, solving the technical problem of poor timeliness in high-concurrency traffic monitoring and improving the response efficiency of high-concurrency traffic.

Description

Technical Field

[0001] This application relates to the field of traffic monitoring technology, and in particular to a concurrent traffic monitoring method, apparatus, terminal device, and storage medium. Background Technology

[0002] Currently, most microservice applications in production environments involve high-concurrency flash sales. In these scenarios, the traffic handled by microservices typically occurs within a limited timeframe of tens of seconds, presenting the following main challenges:

[0003] First, developers find it difficult to obtain real-time traffic data for microservice applications using existing organizational-level monitoring tools (such as Tianyan and Dayu). Second, existing similar technologies require receiving a pre-statistical total number of requests to the server cluster within a specified time period, and also need to calculate the proportion of requests received locally within that specified time period to the total number of requests to the server cluster. This results in a large amount of data to be collected, a cumbersome process, and intrusiveness to the code.

[0004] Therefore, in high-concurrency flash sale scenarios, microservice applications face extremely short traffic response time requirements and security requirements. Summary of the Invention

[0005] The main objective of this application is to provide a concurrent traffic monitoring method, device, terminal equipment, and storage medium, aiming to solve the technical problems of poor timeliness and low security in high-concurrency traffic monitoring.

[0006] To achieve the above objectives, this application provides a concurrent traffic monitoring method, the concurrent traffic monitoring method comprising:

[0007] Obtain the application's consumption logs;

[0008] The consumption logs are transformed and filtered to obtain the corresponding event data stream;

[0009] Based on a preset second-level processing window, the event data stream is processed in seconds to obtain the target concurrent data.

[0010] Optionally, the second-level processing window includes a second-level event time window and a second-level aggregation window. The step of performing second-level processing on the event data stream based on the preset second-level processing window to obtain the target concurrent data includes:

[0011] Based on a preset water level, the event data stream is calculated within the second-level event time window to obtain several concurrent traffic flows;

[0012] Based on the second-level aggregation window, the target concurrent data is obtained by selecting from the several concurrent traffic streams.

[0013] Optionally, the event data stream includes a first event data stream and / or a second event data stream, and the step of transforming and filtering the consumption logs to obtain the corresponding event data stream includes:

[0014] The type of the consumption log is converted and copied to obtain the initial event data stream and the copied event data stream, respectively.

[0015] Based on preset subscription conditions, the initial event data stream and the replicated event data stream are filtered respectively to obtain the first event data stream and the second event data stream.

[0016] Optionally, the water level line includes a low water level line and a high water level line, and the step of triggering the calculation of the event data stream based on the preset water level line and the second-level event time window to obtain several concurrent flows includes:

[0017] Based on the low water level, the second-level event time window is triggered to calculate the first event data stream, and a number of first concurrent data are obtained;

[0018] Based on the high water level, the second event data stream is calculated using the second-level event time window to obtain several second concurrent data.

[0019] Optionally, the target concurrent data includes target first concurrent data and target second concurrent data, and the step of selecting from the plurality of concurrent traffic based on the second-level aggregation window to obtain the target concurrent data includes:

[0020] Based on the preset first aggregation mechanism and the second-level aggregation window, the target first concurrent data is extracted from the plurality of first concurrent data;

[0021] Based on the preset second aggregation mechanism and the second-level aggregation window, the target second concurrent data is extracted from the plurality of second concurrent data.

[0022] Optionally, after the step of processing the event data stream in seconds based on a preset second-level processing window to obtain the target concurrent data, the method further includes:

[0023] The first concurrent data is written to a preset server cluster, and the second concurrent data is written to a preset database;

[0024] When the first concurrent data and / or the second concurrent data are detected to be higher than the preset historical information, the historical information is updated and / or compensated.

[0025] Optionally, after the steps of writing the first concurrent data into a preset server cluster and writing the second concurrent data into a preset database, the method further includes:

[0026] The first concurrent data and / or the second concurrent data are obtained from the server cluster and / or the database, respectively.

[0027] The application is adjusted based on the first concurrent data and / or the second concurrent data.

[0028] This application also proposes a concurrent traffic monitoring device, which includes:

[0029] The acquisition module is used to acquire the application's consumption logs;

[0030] The conversion module is used to convert and filter the consumption logs to obtain the corresponding event data stream;

[0031] The second-level processing module is used to process the event data stream in seconds based on a preset second-level processing window to obtain the target concurrent data.

[0032] This application also proposes a terminal device, which includes a memory, a processor, and a concurrent traffic monitoring program stored in the memory and executable on the processor. When the concurrent traffic monitoring program is executed by the processor, it implements the steps of the concurrent traffic monitoring method described above.

[0033] This application also proposes a computer-readable storage medium storing a concurrent traffic monitoring program, which, when executed by a processor, implements the steps of the concurrent traffic monitoring method described above.

[0034] The concurrent traffic monitoring method, apparatus, terminal device, and storage medium proposed in this application obtain application consumption logs; transform and filter the consumption logs to obtain corresponding event data streams; and process the event data streams at the second level based on a preset second-level processing window to obtain target concurrent data. Based on this application's solution, using the consumption logs of microservice applications as a data source can solve the technical problem of low security in high-concurrency traffic monitoring, avoiding modification or intrusion into business code; by transforming and filtering the consumption logs to obtain event data streams, and then combining this with a second-level processing window to perform second-level processing, the second-level concurrent traffic of the microservice application can be obtained, solving the technical problem of poor timeliness in high-concurrency traffic monitoring and improving the response efficiency of high-concurrency traffic. Attached Figure Description

[0035] Figure 1 This is a schematic diagram of the functional modules of the terminal equipment to which the concurrent traffic monitoring device of this application belongs;

[0036] Figure 2 This is a flowchart illustrating a first exemplary embodiment of the concurrent traffic monitoring method of this application;

[0037] Figure 3 This is a flowchart illustrating a second exemplary embodiment of the concurrent traffic monitoring method of this application;

[0038] Figure 4 This is a module architecture diagram of the concurrent traffic monitoring method in this application;

[0039] Figure 5 This is a flowchart illustrating a third exemplary embodiment of the concurrent traffic monitoring method of this application;

[0040] Figure 6 This is a screenshot of the interface for querying real-time concurrent data using the concurrent traffic monitoring method.

[0041] Figure 7 This is a screenshot of the interface for querying historical concurrent data using the concurrent traffic monitoring method.

[0042] The realization of the purpose, functional features and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0043] It should be understood that the specific embodiments described herein are merely illustrative of this application and are not intended to limit this application.

[0044] The main solution of this application embodiment is: to obtain the consumption logs of the application; to transform and filter the consumption logs to obtain the corresponding event data stream; and to perform second-level processing on the event data stream based on a preset second-level processing window to obtain the target concurrent data. Based on this application solution, obtaining the consumption logs of the microservice application as a data source can solve the technical problem of low security in high-concurrency traffic monitoring, avoiding modification or intrusion into business code. By transforming and filtering the consumption logs to obtain the event data stream, and then combining it with a second-level processing window to perform second-level processing, the second-level concurrent traffic of the microservice application can be obtained, solving the technical problem of poor timeliness in high-concurrency traffic monitoring and improving the response efficiency of high-concurrency traffic.

[0045] Technical terms used in the embodiments of this application:

[0046] QPS, Queries Per Second, refers to the number of queries or requests a system can process per unit of time. It is commonly used to measure a system's performance and throughput. A higher QPS indicates that the system can handle more queries or requests and has higher concurrency capabilities.

[0047] In the Kafka messaging system, a Topic represents a specific message category or topic, and can be viewed as a logical container for message publishing and subscription. Each Topic can have multiple producers (message publishers) and multiple consumers (message subscribers). Producers publish messages to the specified Topic, while consumers subscribe to and consume messages from the Topic. A Kafka Topic is a logical classification of messages within a Kafka cluster, used to organize and divide messages. It can be understood as a message queue where messages are stored and distributed according to certain rules. Each Topic has a unique name identifier used for sending and receiving messages within the Kafka cluster. By using Topics, different messages can be isolated and managed, allowing different applications or modules to independently publish and subscribe to specific message types. This logical classification and isolation helps to achieve a highly scalable messaging system while providing flexible message processing and flow mechanisms.

[0048] Flink, an open-source streaming framework, is designed to run in all common cluster environments, performing computations at in-memory speeds and arbitrary scales. It provides an efficient, scalable, and fault-tolerant way to process and analyze real-time data. It supports a high-throughput, low-latency, high-performance distributed processing framework.

[0049] This application's embodiments take into account that the business scenarios currently covered in production already include a large number of high-concurrency flash sale scenarios. In high-concurrency flash sale scenarios, the traffic handled by microservices typically occurs within a limited number of tens of seconds, resulting in significant instantaneous pressure and the following main pain points:

[0050] First, existing organizational-level monitoring systems (such as Tianyan and Dayu) cannot query real-time concurrent data at the second level, so developers cannot perceive the real-time traffic volume.

[0051] Second: Currently, existing organizational-level monitoring systems (such as Tianyan and Dayu) cannot query historical concurrent data at the second level. Therefore, developers cannot assess the capacity of production applications, predict activity traffic data, or perform operations such as capacity expansion in advance.

[0052] Third: Existing similar technologies require receiving a pre-statistical total number of server cluster requests within a specified time period, and also need to calculate the proportion of the number of requests received locally within the specified time period in the total number of server cluster requests. The process is cumbersome, requires a lot of data collection, and is intrusive to the code.

[0053] Therefore, the solution proposed in this application addresses the practical problems of timeliness and security of interface QPS data in high-concurrency scenarios. It adopts a solution that calculates interface QPS data through logs without intruding on the code, and designs an interface QPS data calculation method based on the Flink framework. This method can obtain real-time and historical concurrent data of the interface, avoid the intrusiveness of high-concurrency traffic monitoring on the code, and improve the response efficiency of high-concurrency traffic.

[0054] Specifically, refer to Figure 1 , Figure 1 This is a functional module diagram of the terminal device to which the concurrent traffic monitoring device of this application belongs. The concurrent traffic monitoring device can be an independent device capable of concurrent traffic monitoring, and it can be implemented on the terminal device in hardware or software form. The terminal device can be a smart mobile terminal with concurrent traffic monitoring capabilities, such as a mobile phone or tablet computer, or it can be a fixed terminal device or server with concurrent traffic monitoring capabilities.

[0055] In this embodiment, the terminal device to which the concurrent traffic monitoring device belongs includes at least an output module 110, a processor 120, a memory 130, and a communication module 140.

[0056] The memory 130 stores the operating system and concurrent traffic monitoring program. The concurrent traffic monitoring device can acquire the application's consumption logs; transform and filter the consumption logs to obtain the corresponding event data stream; process the event data stream at the second level based on a preset second-level processing window, and store the target concurrent data and other information in the memory 130. The output module 110 can be a display screen, etc. The communication module 140 can include a WIFI module, a mobile communication module, and a Bluetooth module, etc., and communicates with external devices or servers through the communication module 140.

[0057] The concurrent traffic monitoring program in memory 130, when executed by the processor, performs the following steps:

[0058] Obtain the application's consumption logs;

[0059] The consumption logs are transformed and filtered to obtain the corresponding event data stream;

[0060] Based on a preset second-level processing window, the event data stream is processed in seconds to obtain the target concurrent data.

[0061] Furthermore, when the concurrent traffic monitoring program in memory 130 is executed by the processor, it also performs the following steps:

[0062] Based on a preset water level, the event data stream is calculated within the second-level event time window to obtain several concurrent traffic flows;

[0063] Based on the second-level aggregation window, the target concurrent data is obtained by selecting from the several concurrent traffic streams.

[0064] Furthermore, when the concurrent traffic monitoring program in memory 130 is executed by the processor, it also performs the following steps:

[0065] The type of the consumption log is converted and copied to obtain the initial event data stream and the copied event data stream, respectively.

[0066] Based on preset subscription conditions, the initial event data stream and the replicated event data stream are filtered respectively to obtain the first event data stream and the second event data stream.

[0067] Furthermore, when the concurrent traffic monitoring program in memory 130 is executed by the processor, it also performs the following steps:

[0068] Based on the low water level, the second-level event time window is triggered to calculate the first event data stream, and a number of first concurrent data are obtained;

[0069] Based on the high water level, the second event data stream is calculated using the second-level event time window to obtain several second concurrent data.

[0070] Furthermore, when the concurrent traffic monitoring program in memory 130 is executed by the processor, it also performs the following steps:

[0071] Based on the preset first aggregation mechanism and the second-level aggregation window, the target first concurrent data is extracted from the plurality of first concurrent data;

[0072] Based on the preset second aggregation mechanism and the second-level aggregation window, the target second concurrent data is extracted from the plurality of second concurrent data.

[0073] Furthermore, when the concurrent traffic monitoring program in memory 130 is executed by the processor, it also performs the following steps:

[0074] The first concurrent data is written to a preset server cluster, and the second concurrent data is written to a preset database;

[0075] When the first concurrent data and / or the second concurrent data are detected to be higher than the preset historical information, the historical information is updated and / or compensated.

[0076] Furthermore, when the concurrent traffic monitoring program in memory 130 is executed by the processor, it also performs the following steps:

[0077] The first concurrent data and / or the second concurrent data are obtained from the server cluster and / or the database, respectively.

[0078] The application is adjusted based on the first concurrent data and / or the second concurrent data.

[0079] This embodiment, through the above-described scheme, specifically obtains the application's consumption logs; transforms and filters the consumption logs to obtain the corresponding event data stream; and processes the event data stream at the second level based on a preset second-level processing window to obtain the target concurrent data. Based on this application's scheme, using the consumption logs of the microservice application as a data source can solve the technical problem of low security in high-concurrency traffic monitoring, avoiding modification or intrusion into business code; by transforming and filtering the consumption logs to obtain the event data stream, and then combining it with the second-level processing window for second-level processing, the second-level concurrent traffic of the microservice application can be obtained, solving the technical problem of poor timeliness in high-concurrency traffic monitoring and improving the response efficiency of high-concurrency traffic.

[0080] Based on, but not limited to, the terminal device architecture described above, this application proposes method embodiments.

[0081] Reference Figure 2 , Figure 2 This is a flowchart illustrating a first exemplary embodiment of the concurrent traffic monitoring method of this application. The concurrent traffic monitoring method includes:

[0082] Step S210: Obtain the application's consumption logs;

[0083] The execution subject of the method in this embodiment can be a concurrent traffic monitoring device, a concurrent traffic monitoring terminal device, or a server. This embodiment takes a concurrent traffic monitoring device as an example. The concurrent traffic monitoring device can be integrated into a terminal device such as a smartphone or tablet computer with concurrent traffic monitoring function.

[0084] This application's embodiment addresses the practical problems of timeliness and security in high-concurrency traffic monitoring under high-concurrency scenarios. It primarily focuses on monitoring high-concurrency traffic, especially high-concurrency traffic, to improve image quality. A solution is adopted that calculates interface QPS data through logs without intruding on the code. A Flink-based method for calculating interface QPS data is designed, which can obtain real-time and historical concurrency data of the interface, avoiding the intrusiveness of high-concurrency traffic monitoring to the code and improving the response efficiency of high-concurrency traffic.

[0085] Specifically, consumption logs can be logs generated by multiple users using a microservice application, including but not limited to timestamps, log levels, log sources, log content, thread information, exception information, user information, system environment information, request and response information, performance metrics, etc. In this embodiment, by obtaining consumption logs generated by multiple users using a microservice application and sending them to a streaming processing platform for second-level processing, second-level concurrent data of the microservice application can be obtained.

[0086] Step S220: Transform and filter the consumption logs to obtain the corresponding event data stream;

[0087] Specifically, event data streams can be processed in the order in which events occur, and events can be processed and responded to in a low-latency manner. By converting consumption logs into event data streams, various data aggregation and transformation operations can be performed. For example, events can be filtered, mapped, windowed, and aggregated to extract useful information and metrics.

[0088] Step S230: Based on a preset second-level processing window, the event data stream is processed in seconds to obtain the target concurrent data.

[0089] It should be noted that the concurrent data in the embodiments of this application can be QPS data.

[0090] For example, considering that existing organizational-level monitoring (such as Tianyan and Dayu) is difficult to query real-time concurrent data at the second level, and requires receiving the total number of server cluster requests within a specified time period, as well as calculating the proportion of locally received requests within the specified time period to the total number of server cluster requests, the process is cumbersome, requires a large amount of data collection, and is intrusive to the code. Therefore, when multiple users operate on the microservice application simultaneously, the consumption logs of the microservice application can be obtained in real time through an interface. The consumption logs can be converted into different types, and the data to be aggregated can be dynamically filtered according to the information of the pre-recorded subscription nodes to obtain a real-time event data stream. Based on the preset second-level processing window, the real-time event data stream can be processed at the second level to obtain the real-time concurrent data of the microservice application in every second or any second. This allows developers to perform operation and maintenance management of the microservice application based on the real-time concurrent data.

[0091] This embodiment, through the above-described scheme, specifically obtains the application's consumption logs; transforms and filters the consumption logs to obtain the corresponding event data stream; and processes the event data stream at the second level based on a preset second-level processing window to obtain the target concurrent data. Based on this application's scheme, using the consumption logs of the microservice application as a data source can solve the technical problem of low security in high-concurrency traffic monitoring, avoiding modification or intrusion into business code; by transforming and filtering the consumption logs to obtain the event data stream, and then combining it with the second-level processing window for second-level processing, the second-level concurrent traffic of the microservice application can be obtained, solving the technical problem of poor timeliness in high-concurrency traffic monitoring and improving the response efficiency of high-concurrency traffic.

[0092] Reference Figure 3 , Figure 3 This is a flowchart illustrating a second exemplary embodiment of the concurrent traffic monitoring method of this application. Based on the above... Figure 2 In the illustrated embodiment, the event data stream includes a first event data stream and / or a second event data stream. Step S220 involves transforming and filtering the consumption log to obtain the corresponding event data stream, including:

[0093] Step S310: Convert the type of the consumption log and copy it to obtain the initial event data stream and the copied event data stream respectively;

[0094] Specifically, the initial event data stream refers to the event data stream obtained by converting the consumption log type, and the replicated event data stream refers to another event data stream obtained by replicating the consumption log. It is independent of the initial event data stream and has the same data content. In this embodiment, the initial event data stream and the replicated event data stream are used for parallel computing of real-time concurrency (QPS data) and historical concurrency (historical QPS data), respectively.

[0095] This embodiment uses logs as the primary data source for analysis. This achieves zero intrusion into the business code and leverages the streaming processing framework (Flink). By combining different window sizes (an operator function in Flink) with the log generation time (Event Time) and the application's processing time (Processing Time), it balances the high latency caused by out-of-order data and the accuracy of data in near real-time scenarios.

[0096] Reference Figure 4 , Figure 4This is a module architecture diagram of the concurrent traffic monitoring method in this application. The module architecture diagram for concurrent traffic monitoring in this embodiment includes: log consumption (Source), data transformation, and data distribution (Sink). The KafkaTopic for consuming logs is considered the primary data source. To process this log data, the Kafka consumer interface provided by the Flink framework is used to transform the log data into an event data stream. The main data source in the system is one or more KafkaTopics, which store the generated log data. The Flink framework provides functionality for integration with Kafka, including the Kafka consumer interface. This interface allows data to be consumed from Kafka Topics and passed as an event data stream to the Flink application for processing.

[0097] Step S320: According to preset subscription conditions, filter the initial event data stream and the replicated event data stream respectively to obtain the first event data stream and the second event data stream.

[0098] Specifically, this example uses the first event data stream as the real-time event data stream and the second event data stream as the historical event data stream.

[0099] In the data transformation task, the initial event data stream and the replication event data stream are respectively coupled with the configuration data stream (such as...). Figure 4 The parsing unit / filtering unit is connected, and according to the information of the subscription node recorded in the configuration data stream, the data that needs to be aggregated in the initial event data stream and the replicated event data stream are dynamically filtered out to form a new first event data stream and a second event data stream. In this way, the event data to be processed can be flexibly selected according to the needs of the subscription node and distributed to the second-level processing window for processing.

[0100] Further, the second-level processing window includes a second-level event time window and a second-level aggregation window. Step S230 involves performing second-level processing on the event data stream based on the preset second-level processing window to obtain the target concurrent data, including:

[0101] Step S330: Based on the preset water level, the event data stream is calculated by triggering the second-level event time window to obtain several concurrent traffic flows;

[0102] Specifically, the second-level event time window is used to calculate concurrent data per second (QPS data); the second-level aggregation window is used to aggregate concurrent data over several seconds, selecting the highest concurrent data value from the aggregation window as the target concurrent data; to ensure data real-time performance and accuracy, the watermark represents the progress of event time, ensuring that all events in the time data stream have reached or exceeded a certain time point, and is used to control the triggering timing of window calculations. The setting of the watermark depends on the real-time performance of the data and the required data accuracy.

[0103] When processing event time data streams, in order to perform refined monitoring and statistics for specific business needs, the log timestamps in the logs are first used as the event time. Then, a watermark is added to ensure that when the current second-level event time window is triggered, the data within the second-level event time window can be collected. Then, based on different logs, the concurrent data (QPS data) under different interfaces or different transaction codes can be calculated.

[0104] It's important to note that the calculation of the second-level event time window is based on the event time dimension, meaning that a calculation operation is triggered every second. For each interface or transaction code, its corresponding concurrent data can be calculated independently. This allows for monitoring and analysis across different business dimensions, understanding the request frequency and load of each interface or transaction code. By grouping log data according to interface or transaction code and aggregating it during the second-triggered calculation process, real-time concurrent data can be obtained. This provides a better understanding of the performance of microservice applications, enabling timely problem detection and corresponding optimizations and adjustments to provide a better user experience and system reliability.

[0105] Further, the water level line includes a low water level line and a high water level line. In step S330, based on the preset water level line, the event data stream is calculated using the second-level event time window to obtain several concurrent flows, including:

[0106] Step A331: Based on the low water level, trigger the second-level event time window to calculate the first event data stream and obtain a number of first concurrent data.

[0107] Step A332: Based on the high water level, trigger the calculation of the second event data stream within the second-level event time window to obtain several second concurrent data.

[0108] It should be noted that, in the embodiments of this application, steps A331 and A332 may be executed simultaneously, steps A331 may be executed before steps A332, or steps A331 may be executed after steps A332.

[0109] Specifically, in Figure 4After the Source outputs the event time data stream, different watermark sizes are set for the real-time data stream and the historical data stream. This can control the triggering timing of window calculations. The low watermark configuration ensures the real-time performance of the data stream; the high watermark configuration ensures the accuracy of the data stream, and the high watermark data stream is used to compensate for and update the highest data of the day and / or the highest historical data, and records the historical QPS data.

[0110] Step S340: Based on the second-level aggregation window, select from the several concurrent traffic streams to obtain the target concurrent data.

[0111] Further, the target concurrent data includes target first concurrent data and target second concurrent data. Step S340 involves selecting from the plurality of concurrent traffic flows based on the second-level aggregation window to obtain the target concurrent data, including:

[0112] Step A341: Based on the preset first aggregation mechanism and the second-level aggregation window, extract the target first concurrent data from the plurality of first concurrent data;

[0113] It should be noted that, in the embodiments of this application, steps A341 and A342 may be executed simultaneously, steps A341 may be executed before steps A342, or steps A341 may be executed after steps A342.

[0114] Specifically, the first concurrent data can be real-time concurrent data. By aggregating concurrent data over several seconds using a second-level aggregation window, the highest concurrent data value within that window can be extracted and output. This reduces write operations to downstream systems while simultaneously obtaining the highest real-time concurrent data over several seconds.

[0115] For example, concurrent data corresponding to 1-3 seconds is aggregated through a second-level aggregation window, and the highest concurrent data value is extracted from the 1-3 seconds to obtain the target concurrent data for 1-3 seconds; then, concurrent data corresponding to 4-6 seconds is aggregated through a second-level aggregation window, and the highest concurrent data value is extracted from the 4-6 seconds to obtain the target concurrent data for 4-6 seconds.

[0116] Step A342: Based on the preset second aggregation mechanism and the second-level aggregation window, extract the target second concurrent data from the plurality of second concurrent data.

[0117] Specifically, the second concurrent data can be historical concurrent data. For historical event data streams, the highest QPS value is selected and output every hour. An output operation is triggered every hour, saving the highest daily QPS value and the highest historical QPS value. This allows for compensation and updates to the highest QPS value of the day or in history.

[0118] Further, in step S230, after processing the event data stream at the second level based on a preset second-level processing window to obtain the target concurrent data, the process further includes:

[0119] Step S350: Write the first concurrent data into a preset server cluster and write the second concurrent data into a preset database;

[0120] Specifically, the first and second concurrent data are written to a pre-defined server cluster and a database, respectively. By storing data on different storage media, redundant backup and fault tolerance can be achieved, improving data reliability and availability.

[0121] Step S360: When the first concurrent data and / or the second concurrent data are detected to be higher than the preset historical information, update and / or compensate by updating the historical information.

[0122] Specifically, historical information can be the highest historical information of the day and / or the highest historical information of the month and / or the highest historical information of the year and / or the highest historical information.

[0123] When the system detects that the first concurrent data and / or the second concurrent data are higher than the historical information, the historical information is updated and compensated accordingly. That is, if the new concurrent data exceeds the previously recorded historical information, the system will update the historical information to reflect the latest data status, improving the accuracy and real-time performance of the historical information and maintaining consistency with the real-time data.

[0124] This embodiment, through the aforementioned scheme, specifically uses log timestamps as event timestamps, adds watermarks to trigger window calculations, and employs different window sizes, aggregation strategies, and watermark sizes to achieve real-time calculation and historical statistics of concurrent data, ensuring data accuracy and timeliness. This helps the system promptly detect anomalies, adjust resource allocation in a timely manner, and provide real-time business metrics for decision-making and optimization. Furthermore, event-time-based calculations ensure data accuracy and consistency, avoiding calculation errors caused by the uncertainty of data arrival order.

[0125] Reference Figure 5 , Figure 5 This is a flowchart illustrating a third exemplary embodiment of the concurrent traffic monitoring method of this application. Based on the above... Figure 2 , 3 In the illustrated embodiment, step S350, writing the first concurrent data into a preset server cluster and writing the second concurrent data into a preset database, includes:

[0126] Step S510: Obtain the first concurrent data and / or the second concurrent data from the server cluster and / or the database, respectively;

[0127] Step S520: Adjust the application based on the first concurrent data and / or the second concurrent data.

[0128] Specifically, the first and second concurrent data can be displayed separately to facilitate developers' real-time monitoring and analysis of concurrent traffic. (See reference...) Figure 6 and Figure 7 , Figure 6 This is a screenshot of the interface for querying real-time concurrent data using the concurrent traffic monitoring method. The real-time concurrent data query interface can include one or more of the following: product number, microservice name, interface name, alias, type, concurrent response time, concurrent volume, current time of concurrent volume, highest concurrent volume of the day, highest concurrent volume of the day time, and historical highest concurrent volume. It can also be sorted for easy user querying. Figure 7 This is an interface diagram for querying historical concurrent data using the concurrent traffic monitoring method. Users can select one or more of the following: ID, microservice name, interface, and time to query the corresponding historical concurrent data.

[0129] Based on the results of real-time and / or historical concurrency data queries, applications can be adjusted. By analyzing concurrency data through models, potential problems or bottlenecks can be identified, and corresponding measures can be taken to improve application performance and efficiency. For example, system configuration parameters can be adjusted, algorithms optimized, or resources added to meet high concurrency requirements.

[0130] This embodiment, through the above-described scheme, specifically obtains the first concurrent data and / or the second concurrent data from the server cluster and / or the database respectively; and adjusts the application based on the first concurrent data and / or the second concurrent data, thereby achieving system performance optimization, efficiency improvement, and user experience enhancement, and thus improving the overall quality and value of microservice applications.

[0131] Furthermore, this application also proposes a concurrent traffic monitoring device, which includes:

[0132] The acquisition module is used to acquire the application's consumption logs;

[0133] The conversion module is used to convert and filter the consumption logs to obtain the corresponding event data stream;

[0134] The second-level processing module is used to process the event data stream in seconds based on a preset second-level processing window to obtain the target concurrent data.

[0135] The principle and implementation process of concurrent traffic monitoring in this embodiment are explained in the above embodiments and will not be repeated here.

[0136] Furthermore, this application also proposes a terminal device, which includes a memory, a processor, and a concurrent traffic monitoring program stored in the memory and executable on the processor. When the concurrent traffic monitoring program is executed by the processor, it implements the steps of the concurrent traffic monitoring method described above.

[0137] Since this concurrent traffic monitoring program employs all the technical solutions of all the aforementioned embodiments when executed by the processor, it possesses at least all the beneficial effects brought about by all the technical solutions of all the aforementioned embodiments, which will not be elaborated upon here.

[0138] Furthermore, embodiments of this application also propose a computer-readable storage medium storing a concurrent traffic monitoring program, which, when executed by a processor, implements the steps of the concurrent traffic monitoring method described above.

[0139] Since this concurrent traffic monitoring program employs all the technical solutions of all the aforementioned embodiments when executed by the processor, it possesses at least all the beneficial effects brought about by all the technical solutions of all the aforementioned embodiments, which will not be elaborated upon here.

[0140] Compared to existing technologies, the concurrent traffic monitoring method, apparatus, terminal device, and storage medium proposed in this application obtain application consumption logs; transform and filter the consumption logs to obtain corresponding event data streams; and process the event data streams at the second level based on a preset second-level processing window to obtain target concurrent data. Based on this application's solution, using the consumption logs of microservice applications as a data source solves the technical problem of low security in high-concurrency traffic monitoring, avoiding modification or intrusion into business code; by transforming and filtering the consumption logs to obtain event data streams, and then combining this with a second-level processing window to perform second-level processing, the second-level concurrent traffic of the microservice application is obtained, solving the technical problem of poor timeliness in high-concurrency traffic monitoring and improving the response efficiency of high-concurrency traffic.

[0141] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or system that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or system. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or system that includes that element.

[0142] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0143] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) as described above, and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, controlled terminal, or network device, etc.) to execute the methods of each embodiment of this application.

[0144] The above are merely preferred embodiments of this application and do not limit the patent scope of this application. Any equivalent structural or procedural transformations made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.

Claims

1. A concurrent traffic monitoring method, characterized in that, The concurrent traffic monitoring method includes the following steps: Obtain the application's consumption logs; The consumption logs are transformed and filtered to obtain the corresponding event data stream; The event data stream includes a first event data stream and / or a second event data stream, and the step of transforming and filtering the consumption logs to obtain the corresponding event data stream includes: The type of the consumption log is converted and copied to obtain the initial event data stream and the copied event data stream, respectively. Based on preset subscription conditions, the initial event data stream and the replicated event data stream are filtered respectively to obtain the first event data stream and the second event data stream; Based on a preset second-level processing window, the event data stream is processed in seconds to obtain the target concurrent data; The second-level processing window includes a second-level event time window and a second-level aggregation window. The step of processing the event data stream at the second level based on the preset second-level processing window to obtain the target concurrent data includes: Based on a preset water level, the event data stream is calculated within the second-level event time window to obtain several concurrent traffic flows; The water level line includes a low water level line and a high water level line. The step of triggering the calculation of the event data stream based on the preset water level line and obtaining several concurrent traffic flows through the second-level event time window includes: Based on the low water level, the second-level event time window is triggered to calculate the first event data stream, and a number of first concurrent data are obtained; Based on the high water level, the second event data stream is calculated using the second-level event time window to obtain several second concurrent data. Based on the second-level aggregation window, the target concurrent data is obtained by selecting from the several concurrent traffic streams; The target concurrent data includes target first concurrent data and target second concurrent data. The step of selecting from the plurality of concurrent traffic based on the second-level aggregation window to obtain the target concurrent data includes: Based on the preset first aggregation mechanism and the second-level aggregation window, the target first concurrent data is extracted from the plurality of first concurrent data, wherein the plurality of first concurrent data are real-time concurrent data; Based on the preset second aggregation mechanism and the second-level aggregation window, the target second concurrent data is extracted from the plurality of second concurrent data, wherein the plurality of second concurrent data are historical concurrent data.

2. The concurrent traffic monitoring method as described in claim 1, characterized in that, After the step of processing the event data stream at the second level based on a preset second-level processing window to obtain the target concurrent data, the method further includes: The first concurrent data is written to a preset server cluster, and the second concurrent data is written to a preset database; When the first concurrent data and / or the second concurrent data are detected to be higher than the preset historical information, the historical information is updated and / or compensated.

3. The concurrent traffic monitoring method as described in claim 2, characterized in that, After the steps of writing the first concurrent data into a preset server cluster and writing the second concurrent data into a preset database, the method further includes: The first concurrent data and / or the second concurrent data are obtained from the server cluster and / or the database, respectively. The application is adjusted based on the first concurrent data and / or the second concurrent data.

4. A concurrent traffic monitoring device, characterized in that, The concurrent traffic monitoring device is used to implement the concurrent traffic monitoring method as described in claim 1, and the concurrent traffic monitoring device includes: The acquisition module is used to acquire the application's consumption logs; The conversion module is used to convert and filter the consumption logs to obtain the corresponding event data stream; The second-level processing module is used to process the event data stream in seconds based on a preset second-level processing window to obtain the target concurrent data.

5. A terminal device, characterized in that, The terminal device includes a memory, a processor, and a concurrent traffic monitoring program stored in the memory and executable on the processor. When the concurrent traffic monitoring program is executed by the processor, it implements the steps of the concurrent traffic monitoring method as described in any one of claims 1-3.

6. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a concurrent traffic monitoring program, which, when executed by a processor, implements the steps of the concurrent traffic monitoring method as described in any one of claims 1-3.

Images