Monitoring data processing method and equipment based on distributed time window adaptive alignment

Abstract

The invention discloses a monitoring data processing method and equipment based on distributed time window self-adaptive alignment, and relates to data processing, a node statistics time window is configured for each distributed node, and the window can be self-adaptively adjusted based on a service statistics rule and is aligned with a service statistics time window. And each distributed node counts network data key indexes based on respective node statistical time window, and uniformly reports the network data key indexes to a key index data aggregation module. And the key index data aggregation module aggregates the key index data of each node based on the service statistical time window. Through the aggregation operation, the key indexes of the data flowing through the whole distributed cluster network in the specified service statistical time window can be accurately obtained. According to the method, the accuracy of key index statistics is improved, and powerful support is provided for distributed system performance monitoring and optimization.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a monitoring data processing method and device based on distributed time window adaptive alignment. Background Technology

[0002] With the development of internet technology, network data, as the core carrier of information transmission, is increasingly exhibiting high dynamism and unpredictability. Network data changes rapidly, not only differing across different time periods but also varying in scale and characteristics across various application scenarios. In the field of network security, real-time monitoring and analysis of network data is particularly crucial.

[0003] For example, real-time parsing of network data content helps to quickly identify and respond to potential network attacks, while continuous monitoring of network data size can effectively prevent network congestion and failures. Time-window-based statistical techniques provide strong support for the accurate statistics and analysis of real-time network data. This technique achieves real-time monitoring of network data by precisely capturing and aggregating data. As a flexible and configurable framework, the time window allows us to set different time spans according to actual needs, thereby accurately capturing and calculating network data within a specified window.

[0004] However, in a distributed environment, the statistical time windows of each distributed node are often difficult to keep aligned due to complex factors such as node failure and restart, and clock synchronization anomalies, which in turn affects the accuracy of the aggregated results of key indicators. Summary of the Invention

[0005] This application provides a monitoring data processing method and device based on distributed time window adaptive alignment, which is used to realize distributed statistics and unified aggregation of key indicators on the basis of distributed time window adaptive alignment statistical mechanism, and significantly improve the accuracy of statistical results through innovative adaptive alignment mechanism.

[0006] This application provides a monitoring data processing method based on distributed time window adaptive alignment, including: After startup, each message queue node obtains unified business statistics rules and adjusts its node statistics time window based on the business statistics rules so that the node statistics time window is aligned with the business statistics time window defined by the business statistics rules in time. The node statistics time window is defined by the start time and the window length. Each message queue node starts a timed statistics task and, within each task cycle, collects key indicator values ​​of the network data flowing through this node. When each scheduled statistical task is triggered, each message queue node sequentially writes the key indicator data obtained by the node within the current node's statistical time window into its local log file in a specified data format. The log files of each message queue node are parsed in parallel to obtain the key indicator data. Cluster number, node number and key indicator identifier are added to each data entry to form a processed statistical record. The processed statistical record is then reported uniformly. Based on the processed statistical records reported by each node, the records are filtered according to the cluster number, node number, and key indicator identifier to obtain a target record set. Then, based on the start and end times of each record in the target record set, the records are aggregated, and the statistical value fields are summed to obtain the overall statistical results of the distributed system within the specified business statistical time window.

[0007] This application also proposes a monitoring data processing system based on distributed time window adaptive alignment, applied to the management end of a distributed system, including: The business statistics rule monitoring module is used to periodically query and retrieve currently effective business statistics rules; The time window adaptive alignment module is used to distribute the business statistics rules to each message queue node and instruct each node to adaptively adjust its node statistics time window at startup according to the business statistics rules so as to align with the business statistics time window. The key indicator aggregation module is used to receive processed statistical records reported from each message queue node. The processed statistical records include at least the cluster number, node number, key indicator identifier, start time, end time, and statistical value. The key indicator aggregation module is used to filter records according to the cluster number, node number, and key indicator identifier, and to aggregate and sum the statistical values ​​of the filtered records according to the start time and end time, and output the overall statistical results of the distributed system.

[0008] This application provides an electronic device, characterized in that it includes a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, it implements the steps of the aforementioned monitoring data processing method based on distributed time window adaptive alignment.

[0009] This application's embodiments, based on a distributed time window adaptive alignment statistical mechanism, achieve distributed statistics and unified aggregation of key indicators. The method not only supports flexible setting of business statistical time windows to meet diverse analytical needs, but also significantly improves the accuracy of statistical results through an innovative adaptive alignment mechanism. This innovation provides strong technical support for real-time monitoring and analysis of network data in a distributed environment.

[0010] The above description is only an overview of the technical solution of this application. In order to better understand the technical means of this application and to implement it in accordance with the contents of the specification, and to make the above and other objects, features and advantages of this application more obvious and understandable, the following are specific embodiments of this application. Attached Figure Description

[0011] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the scope of this application. Furthermore, the same reference numerals denote the same parts throughout the drawings. In the drawings: Figure 1 This is a schematic diagram of the basic process of the monitoring data processing method based on distributed time window adaptive alignment according to an embodiment of this application; Figure 2 This is a schematic diagram of the overall process of the monitoring data processing method based on distributed time window adaptive alignment according to an embodiment of this application; Figure 3 This is a schematic diagram of a time-window-based distributed statistical network data according to an embodiment of this application; Figure 4 This is a schematic diagram of the monitoring data aggregation and statistics system based on distributed time window adaptive alignment, as described in an embodiment of this application. Detailed Implementation

[0012] Exemplary embodiments of the present disclosure will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.

[0013] This application provides a monitoring data processing method based on distributed time window adaptive alignment, such as... Figure 1 As shown, it includes the following steps: In steps S101 and S102, after startup, each message queue node obtains a unified business statistics rule and adjusts its node statistics time window based on the business statistics rule, including the start time and window length of the node statistics time window, so that the node statistics time window is aligned with the business statistics time window defined by the business statistics rule in time. The node statistics time window is defined by the start time and the window length.

[0014] In a specific example, the business statistics rule monitoring module is responsible for monitoring business statistics rules, such as the requirement to statistically analyze the size of imported network data at granular levels like daily / hourly / minute, and forwarding these business statistics rules to the statistical time window adaptive alignment module. For example... Figure 2 As shown, each distributed message queue node starts a timed statistics task. The timed statistics task is embedded in the distributed message queue program and has the same lifecycle as the distributed message queue program. Therefore, the timed statistics task can be started in the following scenarios: the initial startup of the distributed message queue program and the restart of the distributed message queue program after a failure. Each timed statistics task uses the local time and executes the periodic task every specified time period RT, where RT is the length of the node statistics time window. The local time of each distributed node is kept consistent through clock synchronization technology.

[0015] Unified business statistics rules support user customization. Common business statistics rules include compiling and displaying key indicator data by day, hour, or minute. Here, day, hour, and minute all refer to the hourly time. Scheduled statistics tasks will further determine the node statistics time window based on the business statistics rules. Specifically, for business statistics rules with a granularity of day, hour, or minute, the node statistics time window length is 1 minute, and the start and end times are all minute-level. If the scheduled task starts at a time other than the hourly minute, such as... Figure 3 As shown in (a), the length of the first node's statistical time window needs to be automatically adjusted. Specifically, the end time of the first node's statistical time window is set to the nearest whole minute. After the node's statistical time window is aligned with the business statistical time window, the length of the node's statistical time window is restored to 1 minute.

[0016] In step S103, each message queue node starts a timed statistics task and, within each task cycle, counts the key indicator values ​​of the network data flowing through this node.

[0017] In step S104, each message queue node, upon triggering each scheduled statistical task, sequentially writes the key indicator data collected by the node within its current statistical time window into its local log file using a specified data format. In a specific example, each scheduled statistical task writes the key indicator data sequentially into the local log file F according to a pre-defined output format.

[0018] In step S105, each scheduled statistical task parses the log files of each message queue node in parallel to obtain the key indicator data. A cluster number, node number, and key indicator identifier are added to each data entry to form a processed statistical record. This processed statistical record is then uniformly reported, for example, to the key indicator aggregation module. During the processing, additional metadata information is added to the agreed-upon format <start time, end time, statistical value>. The format of the processed key indicator data is <cluster number, node number, key indicator, start time, end time, statistical value>.

[0019] The key indicator aggregation module persistently stores and aggregates the key indicator data reported by each node's timed statistical task based on the business statistical time window, and persistently stores the aggregation results in an agreed format for use by downstream visualization tasks. In step S106, based on the processed statistical records reported by each node, the records are filtered according to the cluster number, node number, and key indicator identifier to obtain a target record set. The target record set is then aggregated based on the start and end times of each record, and the statistical value fields are summed to obtain the overall statistical results of the distributed system within the specified business statistical time window.

[0020] Specifically, a candidate key indicator data list can be obtained by filtering according to cluster number, node number, and key indicator combination conditions. Further, the sum of key indicator data of each node within the range [ST,ET] is obtained by aggregating based on start time ST and end time ET, which gives the key indicator data related to network data flowing through the distributed message queue cluster within the time period [ST,ET].

[0021] For example, firstly, a candidate set of node statistical information records, denoted as Set_R, is filtered out using three combined conditions: cluster number, node number, and key indicators. Further, Set_R is aggregated based on the start and end times, and the statistical value fields are summed to obtain the sum of statistical values ​​for each distributed node within a specified time window.

[0022] This application's embodiments, based on a distributed time window adaptive alignment statistical mechanism, achieve distributed statistics and unified aggregation of key indicators. The method not only supports flexible setting of business statistical time windows to meet diverse analytical needs, but also significantly improves the accuracy of statistical results through an innovative adaptive alignment mechanism. This innovation provides strong technical support for real-time monitoring and analysis of network data in a distributed environment.

[0023] The clock synchronization module is responsible for synchronizing the local time of each distributed node in the distributed network environment. The time window adaptive alignment module is responsible for automatically adjusting and aligning the node statistical time window according to the received business statistical rules. If a deviation occurs between the node statistical time window and the business statistical time window due to reasons such as node service recovery or restart, the module automatically adjusts the size of the node statistical time window to align it with the business statistical time window in the next statistical time window period. The node statistical time window information automatically adjusted by the time window adaptive alignment module is forwarded to the network data key indicator statistics module. In some embodiments, adjusting the node statistical time window based on the business statistical rules includes: If, when a node starts, the current time T_start of this node is not an integer multiple of the start point of the business statistics time window, then the start time of the current node's statistics time window is set to T_start, the end time is set to the next integer multiple of the end point T_end of the business statistics time window closest to T_start, and the window length is adjusted to T_end-T_start. In the next statistical period, the start time of the node statistical time window is adjusted to T_end, and the window length is restored to the window length defined by the business statistical rules to align with the business statistical time window.

[0024] The process by which the node statistics time window component adaptively aligns the time window according to business statistics rules is as follows: Figure 3 As shown in (b), and These represent the start and end times of a business statistics time window, and the length of the business statistics time window is... - Each message queue node maintains a node statistics time window. Normally, the node statistics time window is aligned with the business statistics time window, as shown in message queue node 1. In abnormal situations, such as when the node statistics time windows of message queue nodes 2 and 3 intersect with the business statistics time windows, the node statistics time window will adaptively adjust to align with the business statistics time window. Taking message queue node 3 as an example, the scheduled statistics task is performed at time... Upon startup, the node statistics time window component will adjust the node statistics time window to node statistics time window 1 based on the business statistics time window information, i.e., the start time is... The end time is Window length is - In the next statistical period, the node statistical time window will automatically change to node statistical time window 2, aligning with the business statistical time window.

[0025] In some embodiments, the specified data format includes start time, end time, and statistical value. For example, the output format is specified as <start time, end time, statistical value>, where the start time and end time are local time in the form of timestamps.

[0026] The log files written sequentially to the local machine include log files in which the key indicator data is written to the storage medium in chronological order and in an append-only manner, for the purpose of persistent data storage and data recovery after node failure.

[0027] In some embodiments, each message queue node further includes the following before startup: The local clocks of all message queue nodes in the distributed system are synchronized using the network time protocol to ensure that the time of each node is consistent.

[0028] In some embodiments, the parallel parsing of the log files on each message queue node's local machine is performed by an independent parsing and forwarding process deployed on each message queue node, with each process running independently.

[0029] In some embodiments, aggregation based on the start and end times of each record in the target record set includes: determining records with identical start and end times as belonging to the same statistical period, and grouping them together for subsequent statistical value summation.

[0030] This application presents a method for real-time network data monitoring, proposing a monitoring data aggregation and statistical scheme based on distributed time window adaptive alignment. Furthermore, this application creates a monitoring data aggregation and statistical device based on distributed time window adaptive alignment. The method presented in this application has strong practicality and wide applicability in the field of internet data monitoring, and has broad application prospects.

[0031] This application also proposes a monitoring data processing system based on distributed time window adaptive alignment, applied to the management end of a distributed system, including: The business statistics rule monitoring module is used to periodically query and retrieve currently effective business statistics rules; The time window adaptive alignment module is used to distribute the business statistics rules to each message queue node and instruct each node to adaptively adjust its node statistics time window at startup according to the business statistics rules so as to align with the business statistics time window. The key indicator aggregation module is used to receive processed statistical records reported from each message queue node. The processed statistical records include at least the cluster number, node number, key indicator identifier, start time, end time, and statistical value. The key indicator aggregation module is used to filter records according to the cluster number, node number, and key indicator identifier, and to aggregate and sum the statistical values ​​of the filtered records according to the start time and end time, and output the overall statistical results of the distributed system.

[0032] The application example of this system includes the following steps: S100: Message queue node service starts.

[0033] S101: Obtain business statistics rules R, and automatically adjust the node statistics time window based on R.

[0034] Specifically, the process of automatically adjusting the node statistical time window is as follows: Figure 3 As shown in (b), this has already been explained above and will not be repeated here.

[0035] S102: Start the node timed statistics task. Specifically, the task period of the node timed statistics task is the length of the node statistics time window, and the start time of the node statistics time window is the task start time.

[0036] S103: Key metrics of network data flowing through this node within the statistical time window of the computing node.

[0037] S104: The scheduled statistics task is triggered, and statistical information is written to the local log file in sequence according to the agreed output format. Specifically, the agreed output format is <start time, end time, statistical value>, where the start time and end time are the start and end times of the statistical time window for a certain node, respectively.

[0038] S105: Parse the node log files, process the key node indicator data, and report it to the key indicator data aggregation module.

[0039] Specifically, the processing of node key indicator data mainly involves adding additional metadata information, including cluster number, node number, key indicators, etc., that is, processing the node key indicator data into <cluster number, node number, key indicator, start time, end time, statistical value>, and uniformly reporting it to the key indicator data aggregation module.

[0040] S106: Persistent storage node key indicator data, aggregated node key indicator data based on start time and end time.

[0041] Specifically, the key indicator data aggregation module persistently stores the key indicator statistics reported by each node. Then, it filters out the candidate set of node statistical information records by combining three conditions: cluster number, node number, and key indicator. It aggregates the candidate set of node statistical information records based on the two attributes of start time and end time, and sums the statistical value field to obtain the sum of the statistical values ​​of each distributed node within the specified time window.

[0042] Figure 4 This is a schematic diagram of a monitoring data aggregation and statistics system based on distributed time window adaptive alignment. Specifically, the business statistics rule monitoring module obtains the current business statistics rule information by periodically querying the business statistics rule information data table. For example, the business needs to count the size of network data imported at granularity such as day / hour / minute, and then reports the business statistics rule information to the time window adaptive alignment module via HTTP request.

[0043] The time window adaptive alignment module automatically adjusts the node statistical time window in real time based on the received business statistical rule information. Specifically, if there is a deviation between the node statistical time window and the business statistical time window, that is, if the start time of the node statistical time window WST is not a minute past the hour, the size of the node statistical time window is automatically adjusted, that is, the time window length is set to WET-WST, where WET is the nearest minute past the hour to WST, so as to align with the business statistical time window in the next time window period, and the node time window information obtained by automatic adjustment is forwarded to the network data key indicator statistics module via HTTP request.

[0044] The clock synchronization module cleverly utilizes the NTP (Network Time Protocol) time synchronization service to accurately and efficiently synchronize the local time of each distributed node in a complex distributed network environment, ensuring that the time consistency of the entire system reaches a highly accurate state.

[0045] The network data key indicator statistics module systematically collects and statistically analyzes key indicator data related to network data based on the node statistical time window, such as the peak value, average value, and total value of network data, and writes the key indicator statistics data sequentially into the local log file.

[0046] The key indicator parsing and forwarding module is responsible for parsing the log files generated by the network data key indicator statistics module to obtain network data key indicators in the format of <start time, end time, statistical value>. It then processes and forwards the key indicator statistics data to the statistical key indicator aggregation module. The processed data is in the format of <cluster number, node number, key indicator, start time, end time, statistical value>.

[0047] The key indicator aggregation module is responsible for collecting key indicator statistics reported by each node, and aggregating the key indicator statistics of each node according to the start time and end time to obtain the overall network data flowing through each node within a specified time window in a distributed environment.

[0048] This application also proposes an electronic device, including a memory and a processor. The memory stores a computer program, which, when executed by the processor, implements the steps of the aforementioned monitoring data processing method based on distributed time window adaptive alignment.

[0049] It should be noted that, in the embodiments of this application, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0050] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0051] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0052] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims. All of these forms are within the protection scope of this application.

Claims

1. A monitoring data processing method based on distributed time window adaptive alignment, characterized in that, include: After startup, each message queue node obtains unified business statistics rules and adjusts its node statistics time window based on the business statistics rules so that the node statistics time window is aligned with the business statistics time window defined by the business statistics rules in time. The node statistics time window is defined by the start time and the window length. Each message queue node starts a timed statistics task and, within each task cycle, collects key indicator values ​​of the network data flowing through this node. When each scheduled statistical task is triggered, each message queue node sequentially writes the key indicator data obtained by the node within the current node's statistical time window into its local log file in a specified data format. The log files of each message queue node are parsed in parallel to obtain the key indicator data. Cluster number, node number and key indicator identifier are added to each data entry to form a processed statistical record. The processed statistical record is then reported uniformly. Based on the processed statistical records reported by each node, the records are filtered according to the cluster number, node number, and key indicator identifier to obtain a target record set. Then, based on the start and end times of each record in the target record set, the records are aggregated, and the statistical value fields are summed to obtain the overall statistical results of the distributed system within the specified business statistical time window.

2. The monitoring data processing method based on distributed time window adaptive alignment as described in claim 1, characterized in that, The task period of the timed statistics task is equal to the window length of the node statistics time window.

3. The monitoring data processing method based on distributed time window adaptive alignment as described in claim 1, characterized in that, Adjusting the node statistics time window based on the aforementioned business statistics rules includes: If, when a node starts, the current time T_start of this node is not an integer multiple of the start point of the business statistics time window, then the start time of the current node's statistics time window is set to T_start, the end time is set to the next integer multiple of the end point T_end of the business statistics time window closest to T_start, and the window length is adjusted to T_end-T_start. In the next statistical period, the start time of the node statistical time window is adjusted to T_end, and the window length is restored to the window length defined by the business statistical rules to align with the business statistical time window.

4. The monitoring data processing method based on distributed time window adaptive alignment as described in claim 2, characterized in that, The specified data format includes start time, end time, and statistical values; The log files written sequentially to the local machine include: The key indicator data is written to the log file of the storage medium in chronological order and appended to achieve persistent data storage and data recovery after node failure.

5. The monitoring data processing method based on distributed time window adaptive alignment as described in claim 1, characterized in that, Each message queue node also includes the following before startup: The local clocks of all message queue nodes in the distributed system are synchronized using the network time protocol to ensure that the time of each node is consistent.

6. The monitoring data processing method based on distributed time window adaptive alignment as described in claim 1, characterized in that, Parallel parsing of the log files on each message queue node's local machine is performed by independent parsing and forwarding processes deployed on each message queue node, with each process running independently.

7. The monitoring data processing method based on distributed time window adaptive alignment as described in claim 1, characterized in that, Aggregation based on the start and end times of each record in the target record set includes: determining records with identical start and end times as belonging to the same statistical period, grouping them together, and then summing the subsequent statistical values.

8. A monitoring data processing system based on distributed time window adaptive alignment, characterized in that, Applications include the management interface for distributed systems, including: The business statistics rule monitoring module is used to periodically query and retrieve currently effective business statistics rules; The time window adaptive alignment module is used to distribute the business statistics rules to each message queue node and instruct each node to adaptively adjust its node statistics time window at startup according to the business statistics rules so as to align with the business statistics time window. The key indicator aggregation module is used to receive processed statistical records reported from each message queue node. The processed statistical records include at least the cluster number, node number, key indicator identifier, start time, end time, and statistical value. The key indicator aggregation module is used to filter records according to the cluster number, node number, and key indicator identifier, and to aggregate and sum the statistical values ​​of the filtered records according to the start time and end time, and output the overall statistical results of the distributed system.

9. An electronic device, characterized in that, It includes a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, it implements the steps of the monitoring data processing method based on distributed time window adaptive alignment as described in any one of claims 1 to 7.

Images