A network anomaly monitoring processing system

Abstract

The application relates to a network exception monitoring processing system and belongs to the technical field of network monitoring, which solves the problem that existing systems cannot adaptively correct monitoring parameters. The system comprises the following modules: a data acquisition module which is used for periodically acquiring monitoring data of front-end services and rear-end networks according to various monitoring indexes of a parameter configuration module; an exception detection module which is used for periodically detecting the monitoring data of the front-end services and the rear-end networks according to various thresholds of the parameter configuration module, and sequentially identifying whether network exceptions exist according to detection results, and if network exceptions exist, a fault processing module is notified; the fault processing module is used for determining fault processing measures and processing faults according to analysis results of the exception detection module, and calculating fault processing scores according to fault recovery time lengths; and the parameter configuration module is used for presetting various indexes and various thresholds, and adaptively correcting a plurality of thresholds according to monitoring data, detection results and fault recovery time lengths. The system realizes accurate monitoring and adaptive adjustment of network exceptions.

Description

Technical Field

[0001] This invention relates to the field of network monitoring technology, and in particular to a network anomaly monitoring and processing system. Background Technology

[0002] Accurate detection of network anomalies is crucial for maintaining the performance and stability of distributed storage clusters. However, network anomalies such as network latency, packet loss and errors, and network congestion occur randomly and intermittently, requiring a comprehensive and robust monitoring system to ensure the continuous and efficient operation of the storage cluster.

[0003] Existing network anomaly monitoring systems typically use underlying network latency and packet loss rate as monitoring indicators. They identify whether the monitoring indicators are normal based on pre-set thresholds, thereby determining whether there are network anomalies in the entire distributed storage cluster.

[0004] Existing monitoring systems are prone to misjudgments. For example, a 3ms increase in network latency between distributed nodes may have no impact on some business applications, but it could render other applications essentially unusable. Such anomalies cannot be detected by relying solely on thresholds. For bandwidth-intensive business systems, network bandwidth can easily reach the physical limit of the network card, resulting in a high packet loss rate, but without a significant impact on business storage. Relying on thresholds could lead to misjudgments of network anomalies during troubleshooting, causing human error.

[0005] Moreover, the monitoring thresholds in existing monitoring systems cannot be adaptively adjusted, rely too much on manual intervention, cannot flexibly adapt to different network and cluster environments, and have low operational efficiency. Summary of the Invention

[0006] Based on the above analysis, the embodiments of the present invention aim to provide a network anomaly monitoring and processing system to solve the problems of inaccurate monitoring and the inability to adaptively correct monitoring parameters in existing systems.

[0007] This invention provides a network anomaly monitoring and processing system, comprising: a data acquisition module, an anomaly detection module, a fault handling module, and a parameter configuration module, wherein...

[0008] The data acquisition module is used to periodically collect monitoring data from the front-end services and back-end network according to the various monitoring indicators of the parameter configuration module.

[0009] The anomaly detection module is used to periodically detect the monitoring data of the front-end business and the back-end network according to the thresholds of the parameter configuration module. It identifies whether there is a network anomaly based on the detection results. If a network anomaly is found, the fault handling module is notified.

[0010] The fault handling module is used to determine fault handling measures based on the analysis results of the anomaly detection module and to handle the fault, and to calculate the fault handling score based on the fault recovery time.

[0011] The parameter configuration module is used to preset various indicators and thresholds, and adaptively correct several thresholds based on monitoring data, detection results and fault recovery time.

[0012] Based on further improvements to the above system, fault handling measures are determined according to the detection results of the anomaly detection module, including:

[0013] If the anomaly coefficient in the detection result is less than the fault coefficient threshold, a network anomaly alarm is sent; otherwise, when the network card binding mode of the server in the distributed storage is primary / backup mode, the fault handling measure is to switch the network card; when the network card binding mode of the server in the distributed storage is load balancing mode, the fault handling measure is to isolate the server nodes and the number of isolated server nodes does not exceed the maximum number of isolated nodes, which is obtained based on the total number of server nodes and the proportional threshold in the parameter configuration module.

[0014] Based on further improvements to the above system, the fault recovery time is obtained by acquiring the start time of the detection period corresponding to the detection result that identifies network anomalies, and the start time of the detection period corresponding to the detection result that does not identify network anomalies after fault handling, and calculating the difference between the two start times; the fault handling score is calculated based on the fault recovery time, including: if the fault recovery time is less than or equal to the expected recovery time, it is full score; otherwise, the score corresponding to the excess time is deducted from the full score.

[0015] Based on further improvements to the above system, the detection results in the anomaly detection module include: a business analysis result sequence, a network analysis result sequence, and anomaly coefficients. The business analysis result sequence is a sequence obtained by periodically analyzing the monitoring data of the front-end business based on multiple business thresholds among various anomaly scenario modes, and using 0 and 1 as normal and anomaly identifiers, respectively. The network analysis result sequence is a sequence obtained by periodically analyzing the monitoring data of the back-end network based on multiple network thresholds among various thresholds and clustering algorithms, and using 0 and 1 as normal and anomaly identifiers, respectively.

[0016] Based on further improvements to the above system, if the anomaly coefficient is greater than or equal to the similarity threshold, then a network anomaly exists; the anomaly coefficient is obtained through the following steps:

[0017] After extracting the test sequences from the business analysis result sequence and the network analysis result sequence according to the length of the test sequence, if the number of abnormal markers at the same position in the two test sequences is greater than or equal to the number of normal markers, the Pearson correlation coefficient of the two test sequences is calculated, and multiple Pearson correlation coefficients are calculated after shifting any test sequence multiple times over time. If the number of Pearson correlation coefficients greater than or equal to the similarity threshold is greater than half, the largest Pearson correlation coefficient is taken as the abnormality coefficient; otherwise, the abnormality coefficient is 0.

[0018] Based on further improvements to the above system, several thresholds are adaptively adjusted according to monitoring data, detection results, and fault recovery time, including: adjusting network thresholds and adjusting service thresholds; each time an adjustment is made, one of the thresholds is selected.

[0019] Based on further improvements to the above system, the parameter configuration module identifies when the fault handling score is less than the score threshold, and the correction direction is to reduce it. The time when the fault recovery time exceeds the expected recovery time is taken as the advance time. The correction period is obtained based on the advance time and the detection cycle corresponding to the detection result. The network threshold or the service threshold is corrected based on the detection result of the correction period.

[0020] Further improvements to the above system include revising network thresholds or service thresholds based on detection results during the revision period, including:

[0021] The new threshold is obtained by adjusting the corresponding threshold proportionally according to the correction direction. When the detection results within the correction period obtained based on the new threshold identify a network anomaly, the new threshold is used as the quasi-threshold.

[0022] When the anomaly detection module performs subsequent periodic checks, it obtains the corresponding detection results and their anomaly coefficients based on the default threshold and the quasi-threshold, respectively. The threshold corresponding to the high anomaly coefficient is used as the decision threshold, and the corresponding fault handling score is recorded.

[0023] If, within a preset time period, there are a large number of quasi-thresholds used as decision thresholds, and the corresponding fault handling scores are greater than or equal to the score thresholds, then the default threshold will be modified to a quasi-threshold.

[0024] Based on further improvements to the above system, the detection results within the correction period are obtained according to the new threshold, including:

[0025] Based on the new threshold, the monitoring data of the backend network / frontend services within the correction period are re-detected to obtain new sequences to be detected; historical sequences to be detected within the correction period are obtained from the service / network analysis result sequences; the anomaly coefficients of the new sequences to be detected and the historical sequences to be detected are calculated to obtain the detection results within the correction period.

[0026] Based on further improvements to the above system, the network threshold is a network latency threshold, and the service threshold is a combination of multiple drop thresholds, multiple time thresholds, and multiple frequency thresholds in various abnormal scenario modes.

[0027] Compared with the prior art, the present invention can achieve at least one of the following beneficial effects:

[0028] 1. It covers the entire process from data acquisition to anomaly detection, fault handling, and parameter correction, ensuring comprehensive monitoring of network status; it adaptively corrects parameters based on monitoring data and fault handling results, and the continuous feedback loop forms a closed-loop system, promoting the system's self-adjustment and optimization, improving the accuracy of anomaly detection and the timeliness of fault handling, and enhancing the system's flexibility and adaptability in the face of different network environments and conditions.

[0029] 2. Based on the analysis results of the backend network and the analysis results of the damage to the frontend business, by analyzing the correlation between the frontend and backend, minor network anomalies that do not affect the business and network anomalies caused by changes in the normal business model are eliminated, thereby improving the accuracy of network anomaly detection.

[0030] 3. Adaptive threshold adjustment and decision-making processes reduce reliance on manual intervention and improve operational efficiency; timely and accurate detection and handling of network anomalies enhance the network experience for end users.

[0031] In this invention, the above-described technical solutions can be combined with each other to achieve more preferred combinations. Other features and advantages of this invention will be set forth in the following description, and some advantages may become apparent from the description or be learned by practicing the invention. The objects and other advantages of this invention can be realized and obtained from what is particularly pointed out in the description and drawings. Attached Figure Description

[0032] The accompanying drawings are for illustrative purposes only and are not intended to limit the invention. Throughout the drawings, the same reference numerals denote the same parts.

[0033] Figure 1 This is a schematic diagram of the structure of a network anomaly monitoring and processing system according to an embodiment of the present invention. Detailed Implementation

[0034] Preferred embodiments of the present invention will now be described in detail with reference to the accompanying drawings, which form part of this application and are used together with the embodiments of the present invention to illustrate the principles of the present invention, but are not intended to limit the scope of the present invention.

[0035] A specific embodiment of the present invention discloses a network anomaly monitoring and processing system, such as... Figure 1As shown, it includes: a data acquisition module, an anomaly detection module, a fault handling module, and a parameter configuration module, wherein,

[0036] The data acquisition module is used to periodically collect monitoring data from the front-end services and back-end network according to the various monitoring indicators of the parameter configuration module.

[0037] The anomaly detection module is used to periodically detect the monitoring data of the front-end business and the back-end network according to the thresholds of the parameter configuration module. It identifies whether there is a network anomaly based on the detection results. If a network anomaly is found, the fault handling module is notified.

[0038] The fault handling module is used to determine fault handling measures based on the analysis results of the anomaly detection module and to handle the fault, and to calculate the fault handling score based on the fault recovery time.

[0039] The parameter configuration module is used to preset various indicators and thresholds, and adaptively correct several thresholds based on monitoring data, detection results and fault recovery time.

[0040] In practice, the monitoring and processing system of this embodiment monitors the distributed storage cluster in real time. The continuous internal feedback loop forms a closed loop, which promotes the system's self-adjustment and optimization, improves the accuracy of anomaly detection and the timeliness of fault handling, and enhances the system's flexibility and adaptability in the face of different network environments and conditions.

[0041] (1) Data acquisition module

[0042] It should be noted that a distributed storage cluster includes multiple storage nodes and server nodes. Storage nodes store front-end business data by creating storage pools and different types of storage services, including block storage, object storage, and file storage. Server nodes interact with data by building a back-end physical network. Each server node includes at least one network interface card (NIC), and the NIC bonding mode is either master-slave mode or load balancing mode.

[0043] The data acquisition module obtains the configured monitoring metrics from the parameter configuration module, collects different monitoring metrics for each storage node and each server node, samples according to the set data sampling interval, and reports to the time-series database. For example, the sampling interval is 3 seconds. If the disk space where the time-series database is located is large enough, the sampling interval can be reduced to 1 second to improve data accuracy and make data analysis more accurate.

[0044] It should be noted that the monitoring metrics include business monitoring metrics for front-end business monitoring objects and network monitoring metrics for back-end network monitoring objects.

[0045] Specifically, the front-end business monitoring objects include each block storage volume, object storage bucket, and file storage directory in each storage pool across all storage nodes. Business monitoring metrics include IOPS (Input / Output Operations Per Second), storage bandwidth, and storage latency for each business monitoring object. These metrics can be obtained by calling the respective storage service's API.

[0046] The backend network monitoring targets include each network interface card (NIC) on each server node. Network monitoring metrics include network latency, number of packets sent and received, network bandwidth, network packet loss rate, network error rate, and NIC status for each monitoring target. These metrics can be obtained by analyzing the returned results after executing system commands.

[0047] For example, the ping command can be used to test the round-trip time of the network and calculate the latency; when the server is a Linux system, the ifconfig or ip command can be used to view the statistics of the network interface, including the number of packets sent and received; or the ethtool command can be used to view more detailed network interface statistics, such as the number of lost packets and the number of corrupted packets.

[0048] (2) Anomaly Detection Module

[0049] It should be noted that the anomaly detection module periodically detects the monitoring data of the front-end business and the back-end network based on the thresholds of the parameter configuration module. The detection results include: business analysis result sequence, network analysis result sequence, and anomaly coefficient.

[0050] ① Business Analysis Result Sequence

[0051] The business analysis result sequence is a sequence obtained by periodically analyzing the monitoring data of the front-end business based on multiple abnormal scenario patterns and various business thresholds, and using 0 and 1 as normal and abnormal identifiers, respectively.

[0052] It's important to note that while changes in IOPS and storage bandwidth in business monitoring metrics are strongly correlated with business pressure and scenarios, a sudden drop in these two metrics could also be caused by network anomalies. Storage latency, on the other hand, is a relatively stable metric. During normal operation, latency for different services typically remains within a certain range with very little fluctuation. A sudden increase in latency is very likely due to network anomalies. Therefore, when analyzing front-end business monitoring data, first identify changes in storage latency. Once storage latency is within the normal range, then identify changes in IOPS and storage bandwidth.

[0053] Specifically, based on various abnormal scenario patterns, and according to multiple business thresholds within each threshold, the monitoring data of the front-end business is analyzed periodically to obtain a business analysis result set, including:

[0054] Within each business analysis cycle, identify whether the storage latency of any business monitoring object exceeds the maximum latency threshold. If it does, the first analysis result is an anomaly. Otherwise, based on multiple business thresholds and various anomaly scenario patterns, identify whether the time-series data of IOPS and storage bandwidth of any business monitoring object matches any one of the anomaly scenario patterns, and obtain the respective identification results. Based on the identification results of multiple consecutive business analysis cycles, obtain the business analysis results. Place the business analysis results into the business analysis result set.

[0055] It is important to emphasize that, in order to reduce random fluctuations in the actual raw time-series data and make potential trends and patterns more obvious, when identifying whether the time-series data matches any abnormal scenario pattern, we analyze both the raw time-series data of IOPS and storage bandwidth of each business monitoring object within the business analysis cycle and the smoothed time-series data obtained by averaging each raw time-series data in equal amounts.

[0056] It should be noted that taking the average of the same quantity includes: taking the average of non-overlapping same quantities in sequence, for example, calculating an average of every 3 data points; and taking the average of the data within a window in the form of a sliding window, for example, the length of the sliding window is 3 and the sliding step is 1.

[0057] Furthermore, this embodiment primarily identifies three abnormal scenario patterns for IOPS and storage bandwidth: a precipitous drop, a continuous drop, and a recurring drop. If the amount of change in data rises or falls within a certain threshold range, the data is considered to be in a stable state.

[0058] Specifically, the cliff-like drop pattern is characterized by a sudden drop in data that lasts for a long time. That is, the current data drops by a magnitude greater than or equal to the second drop threshold compared to the previous data, and the duration of the data after the drop is greater than or equal to the first time threshold. For example, when the second drop threshold is 30% and the first time threshold is 120 seconds, the data drop magnitude is greater than or equal to 30% of the previous data, and the duration is greater than or equal to 120 seconds.

[0059] The characteristic of the continuous decline pattern is that there is a continuous downward trend, and finally it stabilizes at a very low level. That is, the data continues to decline compared with adjacent data, but the decline is less than the second decline threshold. When the data after the decline is in a stable state, the total decline is greater than or equal to the first decline threshold, and the duration is greater than or equal to the first time threshold. For example, when the first decline threshold is 70%, the value that continues to decline to the stable state is 70% of the data before the decline, and the duration is greater than or equal to 120 seconds.

[0060] The characteristic of the repeated decline pattern is that after a sudden drop, it lasts for a very short time and then recovers, and there are multiple sudden drop points repeatedly. That is, the current data drops by a magnitude greater than or equal to the third decline threshold compared to the previous data. The duration of the data after the drop is less than the second time threshold before it recovers. The number of times this occurs within the first time threshold range is greater than or equal to the number threshold. For example, when the third decline threshold is 20%, the second time threshold is 10 seconds, and the number threshold is 3, the data after each drop is 20% of the data before the drop, the duration is less than 10 seconds before it recovers to the data before the drop, and this occurs at least 3 times within 120 seconds.

[0061] Preferably, the first time threshold is greater than the second time threshold, the first drop threshold is greater than the second drop threshold, and the second drop threshold is greater than the third drop threshold.

[0062] This embodiment takes into account that fluctuations in indicators may be due to changes in business models, such as an IOPS-type service suddenly becoming a bandwidth-type service, or the IO size of an IOPS-type service changing irregularly. In order to accurately identify whether the service is damaged, multiple rounds of analysis are conducted to obtain the final business analysis results, which are then placed into the business analysis result set.

[0063] Specifically, based on the identification results from multiple consecutive business analysis cycles, business analysis results are obtained, including:

[0064] By setting the identification result to 1 for any abnormal scenario pattern and 0 otherwise, the identification result sequence of IOPS and storage bandwidth for the same business monitoring object is obtained. If the number of 1s in the same position of the two identification result sequences is greater than or equal to the number of 0s, and the Pearson correlation coefficient of the two identification result sequences is greater than or equal to the similarity threshold, then the business analysis result is abnormal; otherwise, the business analysis result is normal.

[0065] It should be noted that when the number of 1s at the same position in two identification result sequences is greater than or equal to 0, the Pearson correlation coefficient ranges from -1 to 1. A value of 0 indicates no linear correlation, a value greater than 0 indicates a positive correlation, and a value less than 0 indicates a negative correlation. The probability of service impairment is relatively high when both IOPS and storage bandwidth decrease. Therefore, this embodiment classifies Pearson correlation coefficients greater than 0 as follows: 0.8 to 1.0 indicates extremely high correlation; 0.6 to 0.8 indicates strong correlation; 0.4 to 0.6 indicates moderate correlation; 0.2 to 0.4 indicates weak correlation; and 0 to 0.2 indicates very weak correlation or no correlation. Preferably, the similarity threshold is set to 0.7. That is, when the Pearson correlation coefficient of the identification result sequences of IOPS and storage bandwidth for the same service monitoring object is greater than 0.7, it indicates a strong correlation between the two, indicating service impairment and an abnormal service analysis result; otherwise, the service analysis result is normal.

[0066] Finally, the business analysis result set is divided into two parts: 0 is used as the normal indicator and 1 is used as the abnormal indicator to obtain the business analysis result sequence.

[0067] ② Network analysis result sequence

[0068] The network analysis result sequence is obtained by periodically analyzing the monitoring data of the backend network based on multiple network thresholds and clustering algorithms, and using 0 and 1 as normal and abnormal indicators, respectively.

[0069] It should be noted that existing technologies typically only use thresholds to judge network latency. If the threshold is configured too low, it may already be affecting services, leading to missed detections. If the threshold is configured too high, it may lead to false positives under heavy service load. To address both missed and false positives, this embodiment primarily uses clustering algorithms to analyze dynamic noise, supplemented by static network thresholds, combining dynamic and static methods to improve the accuracy of network anomaly identification.

[0070] Furthermore, in network monitoring metrics, the network interface card (NIC) is enabled when its status is normal and disabled when its status is abnormal. Once the NIC is disabled, there is no need to judge other metrics. Therefore, the NIC status is identified first. When the NIC is in normal status, any abnormalities in network packet transmission / reception, network bandwidth, network packet loss rate, and network packet error rate will be reflected in the network latency metrics. Therefore, in order to improve the analysis speed, this embodiment uses NIC status and network latency as the main analysis metrics to analyze the monitoring data of the backend network.

[0071] Specifically, based on multiple network thresholds and clustering algorithms, the monitoring data of the backend network is analyzed periodically to obtain network analysis results, which are then placed into a network analysis result set, including:

[0072] Within each network analysis cycle, the network interface card (NIC) status of any network monitoring object is checked to see if it is disabled. If it is, the network analysis result is abnormal. Otherwise, the DBSCAN clustering algorithm is used to cluster the network latency of each network monitoring object. The algorithm checks whether the network latency in the noise category is greater than the network latency threshold, or whether the difference between the network latency threshold and the network latency threshold is less than the latency difference threshold. If so, the network analysis result is abnormal; otherwise, the network analysis result is normal. The network analysis results are then placed into the network analysis result set.

[0073] It should be noted that when using the DBSCAN clustering algorithm, the settings of the neighborhood radius (eps) and the minimum number of points (min_samples) directly affect the analysis results. After extensive experimental testing, in this embodiment, the neighborhood radius is set to 1.5 times the normal network latency value, and the minimum number of points is 1 / 3 of the number of network latency samples. After clustering, the network latency data is divided into several clusters and noise (data that does not belong to any cluster). In this embodiment, the network latency classified as noise is considered as data that may be abnormal. These data are further compared with the network latency threshold. If any network latency is greater than the network latency threshold, or any difference obtained from the comparison is less than the latency difference threshold (i.e., very close to the network latency threshold), the network analysis result is abnormal, and the corresponding network monitoring object (network card) and its network monitoring indicators are recorded; otherwise, the network analysis result is normal.

[0074] Finally, the network analysis result set is divided into two parts: 0 is used as the normal indicator and 1 is used as the abnormal indicator to obtain the network analysis result sequence.

[0075] ③ Anomaly coefficient

[0076] The anomaly coefficient is used to represent the degree of correlation between "business impairment" and "network anomaly". By analyzing network anomalies that can lead to business impairment, the accuracy of network anomaly problem judgment is greatly improved.

[0077] Considering that network anomalies usually precede business disruptions, and there is a certain time lag between the business analysis result sequence and the network analysis result sequence, this embodiment ensures the accuracy of anomaly coefficient acquisition by shifting data and integrating multiple correlation detection results.

[0078] Specifically, the anomaly coefficient is obtained through the following steps:

[0079] After extracting the test sequences from the business analysis result sequence and the network analysis result sequence according to the length of the test sequence, if the number of abnormal markers at the same position in the two test sequences is greater than or equal to the number of normal markers, the Pearson correlation coefficient of the two test sequences is calculated, and multiple Pearson correlation coefficients are calculated after shifting any test sequence multiple times over time. If the number of Pearson correlation coefficients greater than or equal to the similarity threshold is greater than half, the largest Pearson correlation coefficient is taken as the abnormality coefficient; otherwise, the abnormality coefficient is 0, indicating that there is no correlation and no network abnormality exists.

[0080] It should be noted that shifting the time point of the sequence to be detected means shifting the sequence corresponding to the service forward or shifting the sequence corresponding to the network backward. The number of shifts can be set in advance. For example, shifting the sequence corresponding to the network backward means shifting the data at time point 1 to time point 2, and so on for other data.

[0081] If the anomaly coefficient is greater than or equal to the similarity threshold, a network anomaly is detected. The fault handling module is then notified to make a fault handling decision and take corresponding action.

[0082] The anomaly handling module records each detection result and the thresholds used in the detection process to a log. For example, the record format is: {Detection Time 1, Detection Result 1, {Business Threshold 1, Network Threshold 1, Similarity Threshold 1}}, {Detection Time 2, Detection Result 2, {Business Threshold 2, Network Threshold 2, Similarity Threshold 2}}.

[0083] (3) Fault handling module

[0084] Troubleshooting measures include: alarms, network card switching, and server node isolation.

[0085] Specifically, if the anomaly coefficient in the detection result is less than the fault coefficient threshold in the parameter configuration module, such as 0.8, a network anomaly alarm is sent; otherwise, when the network interface card (NIC) bonding mode of the server in the distributed storage is primary / standby mode, the fault handling measure is to switch NICs; when the NIC bonding mode of the server in the distributed storage is load balancing mode, the fault handling measure is to isolate server nodes, and the number of isolated server nodes does not exceed the maximum number of isolated nodes. This maximum number of isolated nodes is obtained based on the total number of server nodes and the ratio threshold in the parameter configuration module. This ratio threshold is configured and adjusted according to the actual business pressure and availability requirements. For example, the ratio threshold is configured to 0.3, that is, the maximum number of isolated nodes is 30% of the total number of server nodes.

[0086] Based on the network analysis result sequence and the detection period corresponding to the detection results, the network monitoring object that has an anomaly, namely the network card, is obtained, and then the server information where the network card is located is obtained.

[0087] Understandably, in primary / standby mode, if the fault is not resolved after switching network interface cards (NICs), the server node containing the NIC will be isolated from the cluster. In load balancing mode, considering that too many isolated server nodes would significantly reduce the cluster's processing capacity and also affect front-end services, when the number of servers in the cluster is relatively small, a NIC is randomly switched first, and if the fault is not resolved, another NIC is switched.

[0088] It should be noted that after switching network cards or isolating server nodes, the distributed storage cluster will not experience any abnormalities for a period of time and will restore the network card status or access the isolated server node that was previously handled for the fault.

[0089] After switching network cards or isolating server nodes according to the fault handling measures, the data acquisition module and the anomaly detection module run regularly and continuously obtain new detection results based on the newly collected monitoring data. When no network anomaly is identified based on the detection results, the fault handling score is calculated based on the fault recovery time.

[0090] It should be noted that the fault recovery time is obtained by acquiring the start time of the detection period corresponding to the detection result that identifies network anomalies, and the start time of the detection period corresponding to the detection result that does not identify network anomalies after fault handling, and calculating the difference between the two start times; then, the fault handling score is calculated based on the fault recovery time, including: if the fault recovery time is less than or equal to the expected recovery time, it is full marks; otherwise, the score corresponding to the excess time is deducted from the full marks.

[0091] For example, the expected recovery time is 100 seconds, but the actual fault recovery time is 110 seconds, which is 10% longer than expected. Therefore, 10 points are subtracted from the full score of 100, and the final result is 90 points.

[0092] The fault handling module writes the calculated fault handling score into the log record containing the corresponding detection time and detection result.

[0093] (4) Parameter configuration module

[0094] The parameter configuration module has multiple pre-set monitoring indicators, network thresholds, service thresholds and other system parameters. It not only provides manual configuration, but also allows for adaptive correction of several thresholds based on monitoring data, detection results and fault recovery time.

[0095] When the parameter configuration module detects that the fault handling score is lower than the score threshold, it triggers adaptive correction of several thresholds. Each correction selects one of these thresholds: either the network threshold or the service threshold. The network threshold is a network latency threshold, while the service threshold comprises multiple decrease thresholds, multiple time thresholds, and multiple occurrence thresholds across various abnormal scenario modes.

[0096] It's important to note that a fault handling score below the threshold indicates a prolonged fault recovery time. This might be due to high network or service thresholds, leading to inaccurate network or service analysis results and failure to promptly flag anomalies. Even after shifting the threshold, the correlation between the two may not improve, resulting in missed detections of network anomalies. In this case, the correction strategy is to reduce the time difference. The time between the fault recovery time and the expected recovery time is used as the advance time. The correction period is derived by subtracting the advance time from the start and end times of the detection period corresponding to the detection results. The purpose of adjusting the threshold is to detect network anomalies within the advance correction period and resolve faults as early as possible.

[0097] Specifically, network thresholds or service thresholds are adjusted based on the detection results during the adjustment period, including:

[0098] ① Adjust the corresponding threshold proportionally according to the correction direction to obtain a new threshold, for example, the proportion is 5%. When the detection results within the correction period obtained based on the new threshold identify the existence of network anomalies, the new threshold is used as the quasi-threshold.

[0099] It should be noted that the detection results obtained based on the new threshold within the correction period include:

[0100] Based on the new threshold, the monitoring data of the backend network / frontend services within the correction period are re-detected to obtain new sequences to be detected; historical sequences to be detected within the correction period are obtained from the service / network analysis result sequences; the anomaly coefficients of the new sequences to be detected and the historical sequences to be detected are calculated to obtain the detection results within the correction period.

[0101] If no network anomaly is detected based on the corrected detection results, the new threshold is adjusted proportionally according to the correction direction, provided that the maximum number of adjustments is not exceeded, until a new threshold that can detect network anomalies is obtained. If a new threshold that can detect network anomalies is still not obtained after reaching the maximum number of adjustments, another threshold is switched to for correction.

[0102] ② When the anomaly detection module performs subsequent periodic checks, it obtains the corresponding detection results and their anomaly coefficients based on the default threshold and the quasi-threshold, respectively. The threshold corresponding to the high anomaly coefficient is used as the decision threshold, and the corresponding fault handling score is recorded.

[0103] ③ If, within a preset time period, there are a large number of quasi-thresholds used as decision thresholds, and the corresponding fault handling scores are greater than or equal to the score thresholds, then the default threshold will be modified to a quasi-threshold.

[0104] It should be noted that if repeated instances of network anomalies being recovered occur within a short period after the network and service thresholds are lowered through adaptive correction, it may be because the network or service thresholds are relatively low, leading to misjudgments of network anomalies. In this case, the corresponding consecutive detection periods are used as correction periods, with the correction direction being to increase. The network or service thresholds are adjusted proportionally to obtain new thresholds. It is then determined whether the detection results within the correction period indicate the absence of network anomalies. If not, the new threshold is used as the quasi-threshold, and the default threshold and the quasi-threshold are further compared using steps ② and ③ above to determine whether the default threshold should be corrected.

[0105] After running for a period of time, the thresholds in the parameter configuration module will become more accurate, and the network anomaly monitoring of the distributed storage cluster will become more and more accurate.

[0106] Compared to existing technologies, the network anomaly monitoring and processing system provided in this embodiment covers the entire process from data acquisition to anomaly detection, fault handling, and parameter correction, ensuring comprehensive monitoring of network status. Based on monitoring data and fault handling results, parameters are adaptively corrected, and a continuous feedback loop forms a closed-loop system, promoting system self-adjustment and optimization, improving the accuracy of anomaly detection and the timeliness of fault handling, and enhancing the system's flexibility and adaptability in the face of different network environments and conditions. Based on the analysis results of the backend network and combined with the analysis results of the damage to frontend services, by analyzing the correlation between the frontend and backend, minor network anomalies that do not affect services, as well as network anomalies caused by changes in normal business models, are excluded, improving the accuracy of network anomaly detection. The adaptive threshold adjustment and decision-making process reduces reliance on manual intervention, improving operational efficiency; timely and accurate detection and handling of network anomalies improve the network experience for end users.

[0107] Those skilled in the art will understand that all or part of the processes of the methods described in the above embodiments can be implemented by a computer program instructing related hardware, and the program can be stored in a computer-readable storage medium. The computer-readable storage medium may be a disk, optical disk, read-only memory, or random access memory, etc.

[0108] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention.

Claims

1. A network anomaly monitoring and processing system, characterized in that, include: The module includes a data acquisition module, an anomaly detection module, a fault handling module, and a parameter configuration module. The data acquisition module is used to periodically collect monitoring data of the front-end services and the back-end network according to the monitoring indicators of the parameter configuration module. The anomaly detection module is used to periodically detect the monitoring data of the front-end service and the back-end network according to the thresholds of the parameter configuration module, and sequentially identify whether there are network anomalies based on the detection results. If a network anomaly is found, the fault handling module is notified. The detection results in the anomaly detection module include: a service analysis result sequence, a network analysis result sequence, and anomaly coefficients. The service analysis result sequence is obtained by periodically analyzing the monitoring data of the front-end service based on multiple service thresholds among the thresholds, using 0 and 1 as normal and abnormal indicators, respectively. Specifically, changes in storage latency are identified first, and when the storage latency is within the normal range, changes in IOPS and storage bandwidth are then identified. The network analysis result sequence is obtained by periodically analyzing the back-end network based on multiple network thresholds and a clustering algorithm. The monitoring data is used to obtain a sequence, with 0 and 1 used as normal and abnormal identifiers, respectively. Priority is given to identifying network interface card (NIC) status, and when the NIC status is normal, network latency is the primary analysis indicator. If the anomaly coefficient is greater than or equal to the similarity threshold, a network anomaly exists. The anomaly coefficient is obtained through the following steps: After extracting the test sequence from the business analysis result sequence and the network analysis result sequence according to the length of the test sequence, if the number of anomaly identifiers at the same position in the two test sequences is greater than or equal to the number of normal identifiers, the Pearson correlation coefficient of the two test sequences is calculated. Multiple Pearson correlation coefficients are also calculated by shifting any test sequence multiple times over time. If the number of Pearson correlation coefficients greater than or equal to the similarity threshold is greater than half, the largest Pearson correlation coefficient is taken as the anomaly coefficient; otherwise, the anomaly coefficient is 0. The fault handling module is used to determine fault handling measures and handle faults based on the analysis results of the anomaly detection module, and to calculate a fault handling score based on the fault recovery time. The parameter configuration module is used to preset various indicators and thresholds, and to adaptively correct several thresholds based on the monitoring data, detection results and fault recovery time.

2. The network anomaly monitoring and processing system according to claim 1, characterized in that, Based on the detection results of the anomaly detection module, fault handling measures are determined, including: If the anomaly coefficient in the detection result is less than the fault coefficient threshold, a network anomaly alarm is sent; otherwise, when the network card binding mode of the server in the distributed storage is primary / backup mode, the fault handling measure is to switch the network card; when the network card binding mode of the server in the distributed storage is load balancing mode, the fault handling measure is to isolate the server nodes and the number of isolated server nodes does not exceed the maximum number of isolated nodes, which is obtained based on the total number of server nodes and the proportional threshold in the parameter configuration module.

3. The network anomaly monitoring and processing system according to claim 1, characterized in that, The fault recovery time is obtained by acquiring the start time of the detection period corresponding to the detection result that identifies a network anomaly, and the start time of the detection period corresponding to the detection result that does not identify a network anomaly after fault handling, and calculating the difference between the two start times. The calculation of the fault handling score based on the fault recovery time includes: if the fault recovery time is less than or equal to the expected recovery time, then it is a full score; otherwise, the score is obtained by deducting the score corresponding to the percentage of time exceeding the expected recovery time from the full score.

4. The network anomaly monitoring and processing system according to claim 1, characterized in that, The method involves adaptively adjusting several thresholds based on monitoring data, detection results, and fault recovery time, including adjusting network thresholds and adjusting service thresholds; each time an adjustment is made, one of the thresholds is selected.

5. The network anomaly monitoring and processing system according to claim 4, characterized in that, When the parameter configuration module identifies that the fault handling score is less than the score threshold, the correction direction is to reduce it. The time when the fault recovery time exceeds the expected recovery time is taken as the advance time. The correction period is obtained based on the advance time and the detection period corresponding to the detection result. The network threshold or the service threshold is corrected based on the detection result of the correction period.

6. The network anomaly monitoring and processing system according to claim 5, characterized in that, The step of correcting the network threshold or service threshold based on the detection results during the correction period includes: The new threshold is obtained by adjusting the corresponding threshold proportionally according to the correction direction. When the detection results within the correction period obtained based on the new threshold identify a network anomaly, the new threshold is used as the quasi-threshold. When the anomaly detection module performs subsequent periodic detection, it obtains the corresponding detection results and their anomaly coefficients according to the default threshold and the quasi-threshold, respectively, and uses the threshold corresponding to the high anomaly coefficient as the decision threshold, and records the corresponding fault handling score. If, within a preset time period, there are a large number of quasi-thresholds used as decision thresholds, and the corresponding fault handling scores are greater than or equal to the score thresholds, then the default threshold will be modified to a quasi-threshold.

7. The network anomaly monitoring and processing system according to claim 6, characterized in that, The detection results within the corrected time period are obtained based on the new threshold, including: Based on the new threshold, the monitoring data of the backend network / frontend services within the correction period are re-detected to obtain new sequences to be detected; historical sequences to be detected within the correction period are obtained from the service / network analysis result sequences; the anomaly coefficients of the new sequences to be detected and the historical sequences to be detected are calculated to obtain the detection results within the correction period.

8. The network anomaly monitoring and processing system according to claim 4, characterized in that, The network threshold is a network latency threshold, and the service threshold is a combination of multiple drop thresholds, multiple time thresholds, and multiple frequency thresholds in various abnormal scenario modes.

Images