A data monitoring and dynamic alarm method based on probe and machine learning

Abstract

The present invention proposes a data monitoring and dynamic alarm method based on probes and machine learning, which relates to the field of computer operation and maintenance monitoring technology. The method determines the key nodes of the network topology structure; configures probes on each key node, and calculates the total processing time of the probes for the detection amount of all target positions in the coverage area within T time periods; according to the machine learning model, a dynamic alarm threshold is output based on real-time traffic data and the total processing time of the probes for all target positions in the coverage area within T time periods. When the total processing time is higher than the dynamic alarm threshold, dynamic alarm processing is performed.

Description

Technical Field

[0001] The present invention proposes a method for realizing data monitoring and dynamic alarm based on probes and machine learning, and relates to the technical field of computer operation and maintenance monitoring. Background Art

[0002] In existing monitoring systems, data collection usually relies on statically configured probes, which are deployed on key nodes of the network to collect information such as system performance indicators, network traffic, and log files. These data are transmitted to the central monitoring system for analysis and processing. The architecture of the monitoring system usually includes the data collection layer, data transmission layer, data processing layer, and alarm management layer.

[0003] In the data collection layer, the probes are responsible for collecting various monitoring indicators, such as CPU usage, memory usage, network traffic, etc. The data transmission layer is responsible for transmitting the collected data to the data processing layer, which analyzes and summarizes the data, and finally the alarm management layer generates alarms based on the analysis results.

[0004] The defects of the existing monitoring system are mainly manifested in the following aspects:

[0005] Lack of adaptability: Statically configured probes cannot dynamically adjust data collection strategies based on real-time changes in network status, which may result in missing critical data during peak network traffic or collecting too much irrelevant data when traffic is low.

[0006] False alarm and missed alarm problems: The alarm system based on fixed thresholds cannot accurately identify complex abnormal behaviors, resulting in false alarms and missed alarms, which increases the burden on operation and maintenance personnel.

[0007] Resource consumption: When processing large-scale data, traditional monitoring systems may consume a lot of computing resources, affecting system performance.

[0008] Lack of intelligent analysis: The existing system lacks in-depth intelligent analysis of collected data and cannot effectively predict potential security threats.

[0009] The present invention aims to solve the following technical problems:

[0010] How to implement adaptive configuration of monitoring probes to dynamically respond to changes in the network environment.

[0011] How to reduce false alarms and missed alarms and improve the accuracy of the alarm system.

[0012] How to optimize resource consumption and improve the efficiency of monitoring systems.

[0013] How to achieve intelligent analysis of monitoring data to predict and identify potential security threats. Summary of the invention

[0014] In order to solve the above technical problems, the present invention proposes a method for data monitoring and dynamic alarm based on probes and machine learning, comprising the following steps:

[0015] Step 1: Determine the key nodes of the network topology;

[0016] Step 2: configure probes at each key node and calculate the total processing time of the probes for all target locations in the coverage area within T time periods;

[0017] Step 3: According to the machine learning model, a dynamic alarm threshold is output based on real-time traffic data and the total processing time of probes for all target locations in the coverage area in T time periods. When the total processing time is higher than the dynamic alarm threshold, dynamic alarm processing is performed.

[0018] In the preferred embodiment, in step 2, the detection requirement of the i-th target position in time period t is ;definition represents the number of hops between the i-th target location and the j-th probe ;

[0019] The sum of the detection measurements of all N target positions by the jth probe in time period t is L j (t):

[0020] ;

[0021] Then, in T time periods, the total processing time of the jth probe for the detection of all N target positions is for: ;

[0022] Among them, the processing rate of the jth probe in time period t is .

[0023] In a preferred embodiment, in step three, a neural network architecture of a machine learning model is set, and a parameter optimizer and a cross entropy loss function are used to perform parameter optimization and probability calculation of the neural network architecture.

[0024] In a preferred embodiment, the parameter optimization steps of the parameter optimizer are as follows:

[0025] Computing Gradients :

[0026] ;

[0027] in, is the loss function at time step t Parameters The gradient of

[0028] Update the first-order moment estimate m t :

[0029] ;

[0030] in, is the decay rate of the first-order moment;

[0031] Update the second-order moment estimate v t :

[0032] ;

[0033] in, is the decay rate of the second-order moment;

[0034] Perform bias correction to obtain the corrected first-order moment estimate and corrected second moment estimates :

[0035] ;

[0036] Update Parameters :

[0037] ;

[0038] in, is the learning rate, is a constant that prevents the denominator from being zero.

[0039] In a preferred embodiment, the predicted value L[ The formula for ] is as follows:

[0040] L[ ]=-yln[ ]-(1-y)ln(1- );

[0041] Where: y is real-time traffic data; It is the total processing time of the detection data of all N target positions in the coverage area by the jth probe in T time periods.

[0042] In a preferred embodiment, the dynamic alarm threshold Th is:

[0043] Th=threshold{ };

[0044] Alarm situation A for the jth probe detection data i for:

[0045] ;

[0046] Among them, A i Indicates the alarm status of the jth probe detection data, 1 indicates an alarm and 0 indicates normal.

[0047] In a preferred embodiment, the 95% quantile of the mean L of the historical predicted values ​​is used as the threshold, and the dynamic alarm threshold Th is:

[0048] Th=percentile(L,95%).

[0049] In a preferred embodiment, the dynamic alarm method adopts a parallel pipeline architecture to process the pulled data in parallel through a thread pool; asynchronously monitor the alarm detection queue to monitor in real time whether the data exceeds the dynamic threshold.

[0050] In a preferred embodiment, the data collection frequency and range of the probe are dynamically adjusted according to the network traffic in the hot spot area and the load conditions of the key nodes.

[0051] Compared with the prior art, the present invention has the following advantages:

[0052] Identify the key nodes of the network topology; configure probes on each key node, and calculate the total processing time from the probe to all target locations in the hot spot area where the key node is located in each time period; output dynamic alarm thresholds based on real-time traffic data and total processing time according to the machine learning model, and perform dynamic alarm processing when the total processing time is higher than the dynamic alarm threshold. Improve adaptability: The probe can adjust the data collection strategy according to the real-time network status, which improves the flexibility of the dynamic alarm system.

[0053] The present invention improves the accuracy of the alarm system and reduces false alarms and missed alarms through intelligent analysis of machine learning models; resource optimization algorithms ensure that resource consumption of the monitoring system is minimized when processing large-scale data; optimizes resource consumption; predictive analysis can identify potential security threats in advance, improve the security of the system; and enhance predictive capabilities. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 A flow chart of a method for realizing data monitoring and dynamic alarm based on probe and machine learning of the present invention;

[0055] Figure 2 Schematic diagram of the parallel pipeline architecture. DETAILED DESCRIPTION

[0056] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0057] In the drawings of the specific embodiments of the present invention, in order to better and more clearly describe the working principles of the various components in the system, the connection relationship of the various parts in the device is shown, which only clearly distinguishes the relative position relationship between the various components, and cannot constitute a limitation on the signal transmission direction, connection sequence and structural size, size and shape of each part within the component or structure.

[0058] Secondly, the term "one embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The term "in one embodiment" that appears in different places in this specification does not necessarily refer to the same embodiment, nor does it refer to a separate or selective embodiment that is mutually exclusive with other embodiments.

[0059] Example 1

[0060] like Figure 1 FIG. 1 is a flow chart of a method for realizing data monitoring and dynamic alarm based on probe and machine learning of the present invention, which includes the following steps:

[0061] Step 1: Determine the key nodes of the network topology.

[0062] Conduct a detailed analysis of the network topology to understand the network's hierarchical structure, the connection relationships between nodes, and the link relationships.

[0063] Scan the network to obtain information such as the IP address and MAC address of network devices, and use network visualization tools to draw a network topology diagram to display the nodes of the network topology structure and the link relationships between nodes.

[0064] Analyze the time series data of traffic in each area to identify traffic peaks and valleys. Identify hot spots in the network based on the traffic in each area and identify key nodes in the hot spots.

[0065] Key nodes in hot spots include core routers, important servers, etc. The data status of these key nodes is monitored in real time, including latency, packet loss rate, system load, etc.

[0066] Step 2: Configure probes at each key node and calculate the total processing time of the probes' detection measurements of all target locations in the coverage area within T time periods.

[0067] Probes play an important role in network performance monitoring and are usually designed to be lightweight so as to collect runtime performance data without significantly affecting the performance of application devices.

[0068] Probes can be loaded dynamically when the application is started or when it is running, and they can be integrated into the application without requiring extensive modifications to the application code.

[0069] In the network topology, each application device can be considered as a node. If multiple sub-application devices are running on a device, these application devices can be distinguished by adding a number to the unique identification code of their parent node, thereby ensuring that each sub-node has a unique identification code.

[0070] Calculate the total processing time from the probe to all target locations in the hotspot area in each time period. The specific method is as follows:

[0071] Assume that the detection demand of the i-th target position in time period t is expressed as ;definition represents the number of hops between the i-th target location and the j-th probe , indicating whether the i-th target position is successfully scanned by the j-th probe within the time period t.

[0072] The sum of the detection measurements of all N target positions in the hotspot area by the jth probe in time period t is L j (t):

[0073] ;

[0074] Then, in T time periods, the total processing time of the jth probe for the detection of all N target locations in the coverage area is for: ;

[0075] Among them, the processing rate of the jth probe in time period t is .

[0076] Through such a network topology and probe deployment, the network infrastructure can be effectively monitored and managed to ensure business continuity and data security. This structure also supports rapid response and resolution of performance issues, because the probes can collect and report the performance indicators of application devices in real time, and the probes are evenly distributed on various key nodes to ensure the coverage of the probes in the hotspot areas.

[0077] Step 3: According to the machine learning model, a dynamic alarm threshold is output based on real-time traffic data and the total processing time of probes for all target locations in the coverage area in T time periods. When the total processing time is higher than the dynamic alarm threshold, dynamic alarm processing is performed.

[0078] According to the machine learning model, the alarm threshold is automatically adjusted based on the real-time traffic data and the total processing time output of the probe for all target locations in the coverage area within T time periods. This alarm threshold will be adjusted according to the difference in the input data of the machine learning model. If the total processing time of the probe for all target locations in the coverage area is significantly different from the distribution of historical data, the dynamic alarm reflects the current alarm status.

[0079] If the total processing time of the probe for all target locations in the coverage area exceeds the dynamic alarm threshold, an alarm is triggered. When an alarm is triggered, relevant alarm information is captured, including alarm type, occurrence time, and system status, and the alarm information is classified into urgent, important, or minor. If the alarm information is too frequent, the alarm threshold is adjusted dynamically to reduce false alarms. The machine learning model once again analyzes the real-time traffic data and the total processing time of the probe for all target locations in the coverage area within T time periods and the total processing time of the probe for all target locations in the coverage area within T time periods. If the model's prediction is more accurate and the alarm frequency is controlled, the improvement is effective and the optimized alarm threshold is updated to the alarm system.

[0080] In a preferred embodiment, the alarm system's database contains historical traffic data and average processing time of probes for the past month, and uses machine learning models to identify input big data and predict alarms.

[0081] First, the neural network architecture of the machine learning model is set, including the input layer, hidden layer, and output layer. A hidden layer of one layer is selected, and 100 hidden units are set. The parameter optimizer and the cross entropy loss function are used to optimize the parameters of the neural network architecture and calculate the probability. The number of hidden layers and the number of units in each hidden layer can be adjusted according to the complexity of the actual problem. If the model performs well on the training set but poorly on the validation set (overfitting), the complexity of the model needs to be reduced; conversely, if both perform poorly, the complexity of the model may need to be increased.

[0082] The parameter optimizer can provide good convergence speed and is not very sensitive to the choice of initial learning rate. At the same time, the parameter optimization steps of the parameter optimizer are as follows:

[0083] Computing Gradients :

[0084] ;

[0085] in, is the loss function at time step t Parameters gradient.

[0086] Update the first-order moment estimate m t :

[0087] ;

[0088] in, is the decay rate of the first-order moment, preferably set to 0.9.

[0089] Update the second-order moment estimate v t :

[0090] ;

[0091] in, is the decay rate of the second-order moment, preferably set to 0.999.

[0092] After bias correction, we get and :

[0093] ;

[0094] The purpose of bias correction is to correct the error in the early stage of training due to m t and v t All start from 0 to avoid these estimates being underestimated.

[0095] Update Parameters :

[0096] ;

[0097] in, is the learning rate, is a constant used to prevent the denominator from being zero.

[0098] The predicted value L[ The formula for ] is as follows:

[0099] L[ ]=-yln[ ]-(1-y)ln(1- );

[0100] Where: y is real-time traffic data; It is the total processing time of the detection data of all N target positions in the coverage area by the jth probe in T time periods.

[0101] Capture network traffic data in real time and calculate the total processing time of the jth probe for all N target locations in the coverage area within T time periods. Use the above two types of data as input data for the machine learning model and load the trained machine learning model into the network environment. Use the trained machine learning model to predict the input data and obtain the dynamic alarm threshold and the current alarm situation.

[0102] The dynamic alarm threshold output by the machine learning model can be used to determine whether the current network traffic is abnormal. Based on the output of the machine learning model, a predicted value is generated.

[0103] Assume that the output of the machine learning model is a predicted value, indicating whether the detection data of the current key node is abnormal.

[0104] The dynamic alarm threshold Th is:

[0105] Th=threshold{ };

[0106] The dynamic alarm threshold can be determined by a fixed method or a dynamically adjusted method. For example, the 95% quantile of the mean L of the historical forecast value can be used as the threshold:

[0107] Th = percentile (L, 95%);

[0108] Alarm situation A for the jth probe detection data j for:

[0109] ;

[0110] Among them, 1 indicates an alarm and 0 indicates normal.

[0111] Example 2

[0112] Dynamic Alert System: Develop a dynamic alert system that automatically adjusts alert thresholds based on the output of the machine learning model to reduce false positives and false negatives.

[0113] The dynamic alarm system adopts a parallel pipeline architecture. Figure 2 As shown:

[0114] Step 1: Data processing module

[0115] Actively pull data streams: ensure that the latest performance indicators are obtained in the shortest time;

[0116] Multi-threaded data preprocessing and conditional filtering: The pulled data is processed in parallel through the thread pool. If the indicator storage conditions are met, the indicator storage event is published and the task is submitted to the indicator storage queue for subsequent batch processing. If the alarm detection conditions are met, the alarm detection event is published and the task is submitted to the alarm detection queue for subsequent analysis.

[0117] The pulled data is processed in parallel through the thread pool to fully utilize system resources, ensure high throughput, and quickly respond to potential alarm events. The Unsafe operation class is introduced in parallel processing to optimize remote IO operations.

[0118] When multiple threads request the same database query operation at the same time, only the first requesting thread actually executes the operation, and the remaining threads enter a blocked state, waiting for data to be returned. After the first thread successfully obtains the data, it wakes up the other threads to share the data.

[0119] Through this design, the system can gracefully handle concurrent requests, reduce resource consumption, ensure data consistency, avoid repeated remote IO requests, reduce network burden, thereby achieving efficient thread coordination and improving overall performance, response speed and resource utilization.

[0120] Step 2: Indicator Storage Module

[0121] Asynchronous monitoring indicator storage queue: When a new task enters, the batch storage operation is triggered, and an efficient batch insertion mechanism is adopted to ensure the rapid persistence of data.

[0122] Step 3: Alarm Detection Module

[0123] Asynchronous monitoring alarm detection queue: real-time monitoring and processing of tasks.

[0124] Alarm logic judgment: The system has built-in intelligent alarm strategy to monitor data in real time to see if it exceeds the dynamic threshold. Once an abnormal situation is found, an alarm event is immediately issued and the relevant tasks are placed in the blocking queue to ensure that the alarm information can quickly trigger subsequent processing.

[0125] Technical summary: To avoid blocking the main thread, the alarm task is asynchronously monitored and processed by an independent thread. This mechanism achieves efficient task distribution and ensures that the system can continuously pull data and respond to alarms.

[0126] Step 4: Alarm storage module

[0127] Asynchronous monitoring alarm storage queue: When a task enters, the storage of alarm information is executed immediately.

[0128] Synchronous backup mechanism: If you need to synchronize the alarm levels of the resource table and the alarm table, publish the task to the synchronous backup task queue.

[0129] Alarm notification processing: If an alarm notification needs to be sent, publish the event, submit the task to the alarm notification queue, and prepare for subsequent notification operations.

[0130] Step 5: Synchronize the backup module.

[0131] Asynchronous monitoring synchronous backup queue: handles key synchronization tasks to ensure that the alarm levels of the alarm table and resource table are consistent.

[0132] Step 6: Alarm notification module.

[0133] Asynchronous monitoring of alarm notification queue: executes alarm notification logic to ensure that alarm information is delivered to relevant personnel and systems in a timely manner to achieve rapid response.

[0134] The monitoring module adopts an advanced parallel pipeline architecture. During the whole process, the main thread continues to pull data from Kafka immediately after publishing the event. At the same time, other task listening threads are processing tasks in the blocking queue. Through this design, the system forms a parallel pipeline: the main thread is responsible for data pulling, while multiple threads process other tasks in parallel, improving the response speed and processing power of the system. At the same time, combined with the technology of Unsafe atomic operations, the monitoring module implements an effective remote data access strategy in a high-concurrency environment. This design not only optimizes performance and saves CPU resources. It also achieves a high degree of coordination among various processing links to ensure seamless connection between alarm response and data processing.

[0135] Example 3

[0136] Dynamically adjust the probe data collection frequency and scope based on the network traffic in the hotspot area and the load conditions at the target location.

[0137] Load perception at the target location: Determine the load status of the current key nodes by monitoring indicators such as the CPU, memory usage, and network bandwidth at the target location.

[0138] Traffic analysis: Identify the network status of hot spots based on historical data and real-time traffic patterns in hot spots, such as distinguishing between peak hours and off-peak hours.

[0139] Probe placement strategy adjustment: Automatically adjust the frequency and range of probe data collection based on the above analysis results.

[0140] Build an evaluation model: Use a statistical model to evaluate the current state of network traffic in hot spots, preferably using linear regression to predict network traffic, or using a decision tree to classify the state type of network traffic. Set normal range thresholds for key nodes, and trigger adaptive adjustments when these thresholds are exceeded.

[0141] The probe adjustment rules are dynamically formulated according to the evaluation results of the evaluation model. For example, the data collection frequency of the probe is increased when the load at the target location is high, and vice versa.

[0142] The frequency index of the probe's data collection frequency is obtained in the following way:

[0143] Get the maximum load and minimum load from all target locations to the probe. The maximum load describes the most unbalanced load situation of all target locations, and the minimum load describes the load difference of key nodes under ideal conditions.

[0144] Calculate the ratio of the maximum to the minimum load difference in the current probe coverage area :

[0145] ;

[0146] in, is the probe load, and are the maximum load and minimum load respectively.

[0147] At this time, the frequency index f of the probe is:

[0148] ;

[0149] Among them, T P It is the maximum probe sending interval set by the system. Delay time for probe detection.

[0150] In a preferred embodiment, in order to prevent instability caused by frequent switching, a certain buffer zone may be set or a gradual adjustment method may be adopted.

[0151] In a preferred embodiment, a probe resource optimization algorithm is designed to ensure that resource consumption of the alarm system is kept to a minimum when processing large-scale data.

[0152] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited to this.

[0153] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0154] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A method for data monitoring and dynamic alarm based on probes and machine learning, characterized in that: The steps include: Step 1: Determine the key nodes of the network topology; Step 2: configure probes at each key node and calculate the total processing time of the probes for all target locations in the coverage area within T time periods; Assume that the detection demand of the i-th target position in time period t is ;definition represents the number of hops between the i-th target location and the j-th probe ; The sum of the detection measurements of all N target positions by the jth probe in time period t is L j (t): ； Then, in T time periods, the total processing time of the jth probe for the detection of all N target positions is for: ； Among them, the processing rate of the jth probe in time period t is ; Step 3: According to the machine learning model, a dynamic alarm threshold is output based on the real-time traffic data and the total processing time of the probes for all target locations in the coverage area in T time periods. When the total processing time is higher than the dynamic alarm threshold, dynamic alarm processing is performed; Set the neural network architecture of the machine learning model, and use the parameter optimizer and cross entropy loss function to optimize the parameters of the neural network architecture and calculate probabilities; The parameter optimization steps of the parameter optimizer are as follows: Computing Gradients : ； in, is the loss function at time step t Parameters The gradient of Update the first moment estimate : ； in, is the decay rate of the first-order moment; Update the second moment estimate : ； in, is the decay rate of the second-order moment; Perform bias correction to obtain the corrected first-order moment estimate and corrected second moment estimates : ； Update Parameters : ； in, is the learning rate, is a constant that prevents the denominator from being zero; The predicted value calculated by the binary cross entropy loss function The formula is as follows: ； Among them: y is the real-time traffic data; is the total processing time of the detection data of all N target positions in the coverage area by the jth probe in T time periods; The dynamic alarm threshold Th is: ； Alarm status for the jth probe detection data for: ； in, Indicates the alarm status of the jth probe detection data, 1 indicates an alarm and 0 indicates normal.

2. The method for realizing data monitoring and dynamic alarm based on probe and machine learning according to claim 1 is characterized in that: The 95% quantile of the mean L of the historical forecast value is used as the threshold, and the dynamic alarm threshold Th is: Th=percentile(L,95%).

3. The method for realizing data monitoring and dynamic alarm based on probe and machine learning according to claim 1 is characterized in that: The dynamic alarm method adopts a parallel pipeline architecture to process the pulled data in parallel through a thread pool; asynchronously monitor the alarm detection queue to monitor in real time whether the data exceeds the dynamic threshold.

4. The method for realizing data monitoring and dynamic alarm based on probe and machine learning according to claim 1 is characterized in that: Dynamically adjust the probe's data collection frequency and range based on the network traffic in the hotspot area and the load conditions of key nodes.

Images