Network anomaly monitoring and processing system

Abstract

The invention relates to a network anomaly monitoring and processing system, belongs to the technical field of network monitoring, and solves the problem that monitoring parameters cannot be adaptively corrected in the prior art. The system comprises a data acquisition module which is used for regularly acquiring monitoring data of a front-end service and a back-end network according to each monitoring index of a parameter configuration module; the anomaly detection module is used for regularly detecting the monitoring data of the front-end service and the rear-end network according to the thresholds of the parameter configuration module, identifying whether network anomaly exists or not according to detection results in sequence, and if the network anomaly exists, informing the fault processing module; the fault processing module is used for determining a fault processing measure according to an analysis result of the anomaly detection module, processing the fault and calculating a fault processing score according to the fault recovery duration; and the parameter configuration module is used for presetting each index and each threshold value, and adaptively correcting the plurality of threshold values according to the monitoring data, the detection result and the fault recovery duration. Accurate monitoring and self-adaptive adjustment of network abnormity are realized.

Description

Technical Field

[0001] The present invention relates to the technical field of network monitoring, and in particular to a network abnormality monitoring and processing system. Background Art

[0002] Accurately detecting network anomalies is crucial to maintaining the performance and stability of distributed storage clusters. However, network anomalies such as network latency, packet loss and errors, and network congestion on network links occur randomly and intermittently, requiring a comprehensive and complete monitoring system to ensure that the storage cluster can continue to operate efficiently.

[0003] In existing network anomaly monitoring systems, the underlying network delay and packet loss rate are usually used as monitoring indicators. The monitoring indicators are identified to be normal based on pre-set thresholds, thereby determining whether there are network anomalies in the entire distributed storage cluster.

[0004] The results of the existing monitoring system are prone to misjudgment. For example, when the network delay between distributed nodes increases by 3ms, it has no impact on some business applications, but for other business applications, the business may be basically unavailable. If only the threshold is used, this anomaly cannot be monitored. For bandwidth-based business systems, the network bandwidth can easily reach the physical upper limit of the network card. At this time, the packet loss rate is very high, but there is no obvious impact on business storage. If the threshold is used, it will be misjudged as a network anomaly and fault handling will be performed, which will cause human impact.

[0005] Moreover, the monitoring thresholds in the existing monitoring system cannot be adaptively modified, rely too much on manual intervention, cannot be flexibly adapted to different network and cluster environments, and have low operation and maintenance efficiency. Summary of the invention

[0006] In view of the above analysis, an embodiment of the present invention aims to provide a network abnormality monitoring and processing system to solve the problems of inaccurate monitoring and inability to adaptively correct monitoring parameters in the prior art.

[0007] The embodiment of the present invention provides a network anomaly monitoring and processing system, including: a data acquisition module, an anomaly detection module, a fault processing module and a parameter configuration module, wherein:

[0008] The data collection module is used to regularly collect monitoring data of the front-end business and the back-end network according to the monitoring indicators of the parameter configuration module;

[0009] The anomaly detection module is used to regularly detect the monitoring data of the front-end business and the back-end network according to the thresholds of the parameter configuration module, and identify whether there is a network anomaly according to the detection results. If there is a network anomaly, the fault handling module is notified;

[0010] A fault handling module is used to determine fault handling measures and handle the fault according to the analysis results of the anomaly detection module, and calculate the fault handling score according to the fault recovery time;

[0011] The parameter configuration module is used to preset various indicators and thresholds, and to adaptively modify several thresholds based on monitoring data, detection results and fault recovery time.

[0012] Based on the further improvement of the above system, the fault handling measures are determined according to the detection results of the anomaly detection module, including:

[0013] If the abnormality coefficient in the detection result is less than the fault coefficient threshold, a network abnormality alarm is sent; otherwise, when the network card binding mode of the server in the distributed storage is the master-slave mode, the fault handling measure is to switch the network card; when the network card binding mode of the server in the distributed storage is the load balancing mode, the fault handling measure is to isolate the server nodes and the number of isolated server nodes does not exceed the maximum number of isolated nodes, and the maximum number of isolated nodes is obtained based on the total number of server nodes and the proportional threshold in the parameter configuration module.

[0014] Based on further improvements of the above system, the fault recovery time is obtained by obtaining the start time of the detection period corresponding to the detection result that identifies the existence of network anomalies, and the start time of the detection period corresponding to the detection result that does not identify the existence of network anomalies after fault processing, and calculating the difference between the two start times; the fault processing score is calculated according to the fault recovery time, including: if the fault recovery time is less than or equal to the expected recovery time, it is full marks, otherwise, it is obtained by deducting the score corresponding to the proportion of the excess time from the full mark.

[0015] Based on further improvements of the above system, the detection results in the anomaly detection module include: a business analysis result sequence, a network analysis result sequence and anomaly coefficient; the business analysis result sequence is based on multiple abnormal scenario modes, and regularly analyzes the monitoring data of the front-end business according to multiple business thresholds in each threshold, and uses 0 and 1 as normal and abnormal identifications respectively to obtain the sequence; the network analysis result sequence is based on multiple network thresholds in each threshold and a clustering algorithm to regularly analyze the monitoring data of the back-end network, and uses 0 and 1 as normal and abnormal identifications respectively to obtain the sequence.

[0016] Based on the further improvement of the above system, if the anomaly coefficient is greater than or equal to the similarity threshold, there is a network anomaly; the anomaly coefficient is obtained by the following steps:

[0017] After taking out the sequences to be detected from the business analysis result sequence and the network analysis result sequence according to the length of the detection sequence, if the number of abnormal identifiers at the same position in the two sequences to be detected is greater than or equal to the number of normal identifiers, the Pearson correlation coefficient of the two sequences to be detected is calculated, and multiple Pearson correlation coefficients are calculated after multiple translations of the time points of any sequence to be detected; if the number of Pearson correlation coefficients greater than or equal to the similarity threshold is greater than half, the largest Pearson correlation coefficient is taken as the abnormal coefficient, otherwise, the abnormal coefficient is 0.

[0018] Based on further improvements to the above system, several thresholds are adaptively modified according to monitoring data, detection results and fault recovery time, including: modifying the network threshold and modifying the service threshold; any one of the thresholds is selected for each modification.

[0019] Based on further improvements of the above system, when the parameter configuration module identifies that the fault processing score is less than the score threshold, the correction direction is to reduce it. The time by which the fault recovery time exceeds the expected recovery time is used as the advance time. The correction period is obtained according to the advance time and the detection period corresponding to the detection result. The network threshold or the service threshold is corrected according to the detection result of the correction period.

[0020] Based on further improvements of the above system, the network threshold or the service threshold is modified according to the detection result of the modification period, including:

[0021] A new threshold is obtained by proportionally adjusting the corresponding threshold according to the correction direction. When the detection result within the correction period obtained according to the new threshold identifies the existence of network anomaly, the new threshold is used as the quasi-threshold;

[0022] When the anomaly detection module performs subsequent regular detection, it obtains the corresponding detection results and their anomaly coefficients according to the default threshold and the quasi-threshold, takes the threshold corresponding to the high anomaly coefficient as the decision threshold, and records the corresponding fault handling score;

[0023] If the number of quasi-thresholds used as decision thresholds is relatively large within a preset period, and the corresponding fault processing scores are greater than or equal to the score threshold, the default threshold is modified to the quasi-threshold.

[0024] Based on the further improvement of the above system, the detection results within the correction period are obtained according to the new threshold, including:

[0025] According to the new threshold, the monitoring data of the back-end network / front-end business in the correction period is re-detected to obtain a new sequence to be detected; the historical sequence to be detected in the correction period is obtained from the business / network analysis result sequence; the abnormal coefficients of the new sequence to be detected and the historical sequence to be detected are calculated to obtain the detection results in the correction period.

[0026] Based on further improvement of the above system, the network threshold is a network delay threshold, and the service threshold is multiple drop thresholds, multiple time thresholds and number thresholds in multiple abnormal scenario modes.

[0027] Compared with the prior art, the present invention can achieve at least one of the following beneficial effects:

[0028] 1. It covers the entire process from data collection to anomaly detection, fault handling and parameter correction, ensuring comprehensive monitoring of the network status; it adaptively corrects parameters based on monitoring data and fault handling results, and forms a closed-loop system through continuous feedback loops, which promotes self-regulation and optimization of the system, improves the accuracy of anomaly detection and the timeliness of fault handling, and enhances the flexibility and adaptability of the system in the face of different network environments and conditions.

[0029] 2. Based on the analysis results of the back-end network and combined with the analysis results of the degree of damage to the front-end business, by analyzing the correlation between the front-end and the back-end, minor network anomalies that do not affect the business and network anomalies caused by changes in normal business models are eliminated, thereby improving the accuracy of network anomaly detection.

[0030] 3. Adaptive threshold adjustment and decision-making process reduce reliance on manual intervention and improve operation and maintenance efficiency; timely and accurate detection and processing of network anomalies improves the network experience of end users.

[0031] In the present invention, the above-mentioned technical solutions can also be combined with each other to achieve more preferred combination solutions. Other features and advantages of the present invention will be described in the subsequent description, and some advantages can become obvious from the description, or can be understood by practicing the present invention. The purpose and other advantages of the present invention can be realized and obtained through the contents particularly pointed out in the description and the drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] The accompanying drawings are only used for the purpose of illustrating specific embodiments and are not to be considered as limiting the present invention. In the entire drawings, the same reference symbols represent the same components;

[0033] Figure 1 The present invention is a schematic diagram of the structure of a network anomaly monitoring and processing system in an embodiment of the present invention. DETAILED DESCRIPTION

[0034] The preferred embodiments of the present invention are described in detail below in conjunction with the accompanying drawings, wherein the accompanying drawings constitute a part of this application and are used together with the embodiments of the present invention to illustrate the principles of the present invention, but are not used to limit the scope of the present invention.

[0035] A specific embodiment of the present invention discloses a network anomaly monitoring and processing system, such as Figure 1As shown, it includes: a data acquisition module, an anomaly detection module, a fault processing module and a parameter configuration module, wherein:

[0036] The data collection module is used to regularly collect monitoring data of the front-end business and the back-end network according to the monitoring indicators of the parameter configuration module;

[0037] The anomaly detection module is used to regularly detect the monitoring data of the front-end business and the back-end network according to the thresholds of the parameter configuration module, and identify whether there is a network anomaly according to the detection results. If there is a network anomaly, the fault handling module is notified;

[0038] A fault handling module is used to determine fault handling measures and handle the fault according to the analysis results of the anomaly detection module, and calculate the fault handling score according to the fault recovery time;

[0039] The parameter configuration module is used to preset various indicators and thresholds, and to adaptively modify several thresholds based on monitoring data, detection results and fault recovery time.

[0040] During implementation, the monitoring and processing system of this embodiment monitors the distributed storage cluster in real time, and the internal continuous feedback loop forms a closed loop, which promotes the self-regulation and optimization of the system, improves the accuracy of anomaly detection and the timeliness of fault handling, and enhances the flexibility and adaptability of the system when facing different network environments and conditions.

[0041] (1) Data acquisition module

[0042] It should be noted that the distributed storage cluster includes multiple storage nodes and server nodes. The storage nodes store front-end business data by creating storage pools and different types of storage services. The storage services include block storage, object storage, and file storage. The server nodes interact with data by building a back-end physical network. Each server node includes at least one network card, and the network card binding mode is either active-standby mode or load balancing mode.

[0043] The data collection module obtains the configured monitoring indicators from the parameter configuration module, collects different monitoring indicators for each storage node and each server node, samples according to the set data sampling interval, and reports to the time series database. For example, the sampling interval is 3 seconds. If the disk space where the time series database is located is large enough, the sampling interval can be reduced to 1 second to improve data accuracy and make data analysis more accurate.

[0044] It should be noted that the monitoring indicators include business monitoring indicators for front-end business monitoring objects and network monitoring indicators for back-end network monitoring objects.

[0045] Specifically, the front-end business monitoring objects include each block storage volume, object storage bucket, and file storage directory in each storage pool in all storage nodes, and the business monitoring indicators include IOPS (Input / Output Operations Per Second), storage bandwidth, and storage latency of each business monitoring object. These indicator data can be obtained by calling the API of each storage service.

[0046] The backend network monitoring objects include each network card in each server node, and the network monitoring indicators include network latency, number of network sent and received packets, network bandwidth, network packet loss rate, network error rate, and network card status for each network monitoring object. These indicator data can be analyzed from the returned results after executing system commands.

[0047] For example, the ping command can be used to test the round-trip time of the network and calculate the latency; when the server is a Linux system, the ifconfig or ip command can be used to view the statistics of the network interface, including the number of packets sent and received; or the ethtool command can be used to view more detailed network interface statistics, such as the number of lost packets and error packets.

[0048] (2) Anomaly Detection Module

[0049] It should be noted that the anomaly detection module regularly detects the monitoring data of the front-end business and the back-end network according to the thresholds of the parameter configuration module, and the obtained detection results include: business analysis result sequence, network analysis result sequence and anomaly coefficient.

[0050] ① Business analysis result sequence

[0051] The business analysis result sequence is based on multiple abnormal scenario modes, regularly analyzes the monitoring data of the front-end business according to multiple business thresholds in each threshold, and uses 0 and 1 as normal and abnormal identifications respectively to obtain the sequence.

[0052] It should be noted that although the changes in IOPS and storage bandwidth in the business monitoring indicators are strongly related to business pressure and business scenarios, the sudden drop in these two indicators may also be caused by network abnormalities; storage latency is a relatively stable indicator. During normal operation of different businesses, the latency is usually within a certain range and the fluctuation is very small; the sudden increase in latency is likely to be caused by network abnormalities. Therefore, when analyzing the monitoring data of the front-end business, first identify the changes in storage latency, and when the storage latency is within the normal range, then identify the changes in IOPS and storage bandwidth.

[0053] Specifically, based on multiple abnormal scenario modes, the monitoring data of the front-end business is regularly analyzed according to multiple business thresholds in each threshold to obtain a business analysis result set, including:

[0054] In each business analysis cycle, identify whether there is data with storage latency greater than the maximum latency threshold for any business monitoring object. If so, the first analysis result is abnormal. Otherwise, based on multiple business thresholds and multiple abnormal scenario modes, identify whether the timing data of IOPS and storage bandwidth of any business monitoring object matches any abnormal scenario mode to obtain respective identification results. Based on the identification results of multiple consecutive business analysis cycles, obtain the business analysis results. Put the business analysis results into the business analysis result set.

[0055] It should be emphasized that in order to reduce the random fluctuations in the original time series data actually collected and make the potential trends and patterns more obvious, when identifying whether the time series data matches any abnormal scenario mode, both the original time series data of IOPS and storage bandwidth of each business monitoring object during the business analysis cycle are analyzed, and the smoothed time series data obtained by taking the average of the same number of original time series data are analyzed.

[0056] It should be noted that taking the average of the same number includes: taking the average of the same number of non-overlapping numbers in sequence, for example, calculating an average for every 3 data; and taking the average of the data in the window in a sliding window manner, for example, the length of the sliding window is 3 and the sliding step is 1.

[0057] Furthermore, in this embodiment, three abnormal scenario modes are mainly identified for IOPS and storage bandwidth: a cliff-like drop mode, a continuous drop mode, and a repeated drop mode. If the amount of change in data increase or decrease is within the change threshold range, the data is considered to be in a stable state.

[0058] Specifically, the cliff-like decline mode is characterized by a sudden drop in data that lasts for a long time, that is, the current data drops by a magnitude greater than or equal to a second decline threshold compared with the previous data, and the duration of the data after the decline is greater than or equal to the first time threshold; exemplarily, when the second decline threshold is 30% and the first time threshold is 120 seconds, the data drops by a magnitude greater than or equal to 30% of the previous data, and the duration is greater than or equal to 120 seconds.

[0059] The characteristic of the continuous decline mode is that there is a continuous downward trend and finally stabilizes at a very low level, that is, the data continues to decline compared with adjacent data but the decline is less than the second decline threshold. When the data after the decline is in a stable state, the total decline is greater than or equal to the first decline threshold, and the duration is greater than or equal to the first time threshold; exemplarily, when the first decline threshold is 70%, the value that continues to decline to the stable state is 70% of the data before the decline, and the duration is greater than or equal to 120 seconds.

[0060] The repeated decline mode is characterized by a sudden decline that lasts for a very short time and then recovers, and there are multiple sudden drop points that are repeated repeatedly, that is, the decline of the current data compared with the previous data is greater than or equal to the third decline threshold, the data after the decline lasts for less than the second time threshold and then recovers, and the number of occurrences within the first time threshold is greater than or equal to the number threshold; exemplarily, the third decline threshold is 20%, the second time threshold is 10 seconds, and the number threshold is 3, the data after each decline is 20% of the data before the decline, lasts for less than 10 seconds and then recovers to the data before the decline, and it occurs at least 3 times within 120 seconds.

[0061] Preferably, the first time threshold is greater than the second time threshold, the first falling threshold is greater than the second falling threshold, and the second falling threshold is greater than the third falling threshold.

[0062] This embodiment takes into account that the fluctuation of indicators may be a change in business model, such as IOPS-type business suddenly becomes bandwidth-type business, or the IO size of IOPS-type business changes irregularly. In order to accurately identify whether the business is damaged, the final business analysis result is obtained through multiple rounds of analysis and put into the business analysis result set.

[0063] Specifically, based on the identification results of multiple consecutive business analysis cycles, business analysis results are obtained, including:

[0064] By setting the recognition result that matches any abnormal scenario pattern to 1, otherwise setting the recognition result to 0, the recognition result sequence of IOPS and storage bandwidth of the same business monitoring object is obtained; if the number of 1s in the same position of the two recognition result sequences is greater than or equal to the number of 0s, and the Pearson correlation coefficient of the two recognition result sequences is greater than or equal to the similarity threshold, the business analysis result is abnormal, otherwise the business analysis result is normal.

[0065] It should be noted that when the number of 1s at the same position of two recognition result sequences is greater than or equal to 0, the numerical range of the Pearson correlation coefficient is [-1,1]. When it is 0, it means there is no linear correlation, greater than 0 means positive correlation, and less than 0 means negative correlation. When both IOPS and storage bandwidth decrease, the probability of business damage is relatively high. Therefore, this embodiment classifies the Pearson correlation coefficient greater than 0: when the correlation coefficient is 0.8 to 1.0, it is extremely correlated; when it is 0.6 to 0.8, it is strongly correlated; when it is 0.4 to 0.6, it is moderately correlated; when it is 0.2 to 0.4, it is weakly correlated; and when it is 0 to 0.2, it is extremely weakly correlated or uncorrelated. Preferably, the similarity threshold is set to 0.7. That is, when the Pearson correlation coefficient of the recognition result sequence of IOPS and storage bandwidth of the same business monitoring object is greater than 0.7, it means that the two are strongly correlated. At this time, the business is damaged and the business analysis result is abnormal. Otherwise, the business analysis result is normal.

[0066] Finally, for the business analysis result set, 0 is used as a normal identifier and 1 is used as an abnormal identifier to obtain a business analysis result sequence.

[0067] ② Network analysis result sequence

[0068] The network analysis result sequence is a sequence obtained by regularly analyzing the monitoring data of the backend network according to multiple network thresholds in each threshold and a clustering algorithm, and using 0 and 1 as normal and abnormal identifiers respectively.

[0069] It should be noted that in the prior art, only the threshold is usually used to judge the network delay. If the threshold is configured too low, it may have affected the business, and it is easy to miss the judgment; if the threshold is configured too high, it is easy to make a wrong judgment when the business pressure is high. In order to solve the problems of missed judgment and wrong judgment, this embodiment mainly uses the clustering algorithm to analyze the dynamic noise points, supplemented by the static network threshold, and combines the dynamic and static to improve the accuracy of network anomaly identification.

[0070] Furthermore, in the network monitoring indicators, the network card status is enabled when it is normal and disabled when it is abnormal. Once the disabled status appears, there is no need to judge other indicators. Therefore, the network card status is identified first. When the network card status is normal, the number of network packets received and sent, network bandwidth, network packet loss rate and network packet error rate will be reflected in the network delay indicator once abnormalities occur. Therefore, in order to improve the analysis speed, this embodiment uses the network card status and network delay as the main analysis indicators to analyze the monitoring data of the back-end network.

[0071] Specifically, the monitoring data of the backend network is regularly analyzed according to the multiple network thresholds in each threshold and the clustering algorithm to obtain the network analysis results, which are put into the network analysis result set, including:

[0072] In each network analysis cycle, identify whether the network card status of any network monitoring object is disabled. If so, the network analysis result is abnormal; otherwise, use the DBSCAN clustering algorithm to cluster the network delay of each network monitoring object to identify whether the network delay classified as noise is greater than the network delay threshold, or the difference with the network delay threshold is less than the delay difference threshold. If so, the network analysis result is abnormal, otherwise the network analysis result is normal; put the network analysis result into the network analysis result set.

[0073] It should be noted that when the DBSCAN clustering algorithm is used, the setting of the neighborhood radius (eps) and the minimum number of points (min_samples) directly affects the analysis results. After a large number of experimental tests, the neighborhood radius in this embodiment is set to 1.5 times the normal value of the network delay, and the minimum number of points is 1 / 3 of the number of network delay samples. After clustering, the network delay data is divided into several clusters and noise (data that does not belong to any cluster). In this embodiment, the network delay classified as noise is used as data that may cause network anomalies. These data are further compared with the network delay threshold. If any network delay is greater than the network delay threshold, or any difference obtained by comparison is less than the delay difference threshold, that is, very close to the network delay threshold, the network analysis result is abnormal, and the corresponding network monitoring object (network card) and its network monitoring indicators are recorded, otherwise the network analysis result is normal.

[0074] Finally, for the network analysis result set, 0 is used as the normal mark and 1 is used as the abnormal mark to obtain the network analysis result sequence.

[0075] ③Abnormal coefficient

[0076] The anomaly coefficient is used to indicate the correlation between “business damage” and “network anomaly”, and to analyze network anomalies that may cause business damage, thus greatly improving the accuracy of determining network anomaly problems.

[0077] Considering that network anomalies usually occur first and business damage occurs later, there is a certain time difference between the business analysis result sequence and the network analysis result sequence. In this embodiment, data is shifted and multiple correlation detection results are integrated to ensure the accuracy of the anomaly coefficient acquisition.

[0078] Specifically, the anomaly coefficient is obtained by the following steps:

[0079] After taking out the sequences to be detected from the business analysis result sequence and the network analysis result sequence according to the length of the detection sequence, if the number of abnormal identifiers at the same position in the two sequences to be detected is greater than or equal to the number of normal identifiers, the Pearson correlation coefficient of the two sequences to be detected is calculated, and multiple Pearson correlation coefficients are calculated after multiple translations of the time points of any sequence to be detected; if the number of Pearson correlation coefficients greater than or equal to the similarity threshold is greater than half, the largest Pearson correlation coefficient is taken as the abnormal coefficient, otherwise, the abnormal coefficient is 0, indicating that there is no correlation and no network abnormality.

[0080] It should be noted that the time point shift of the sequence to be detected is to shift the sequence to be detected corresponding to the service forward, or to shift the sequence to be detected corresponding to the network backward, and the number of shifts can be set by pre-configuration. Exemplarily, the backward shift of the sequence to be detected corresponding to the network means to shift the data at time point 1 backward to time point 2, and the same applies to other data.

[0081] If the anomaly coefficient is greater than or equal to the similarity threshold, there is a network anomaly, and the fault handling module is notified to make a decision on the fault handling measures and perform corresponding processing.

[0082] The exception handling module records each detection result and each threshold used in the detection process in a log. Exemplarily, the record format is: {detection time 1, detection result 1, {business threshold 1, network threshold 1, similarity threshold 1}}, {detection time 2, detection result 2, {business threshold 2, network threshold 2, similarity threshold 2}}.

[0083] (3) Fault handling module

[0084] Troubleshooting measures include: alarm, network card switching and server node isolation.

[0085] Specifically, if the abnormal coefficient in the detection result is less than the fault coefficient threshold in the parameter configuration module, such as 0.8, a network abnormality alarm is sent; otherwise, when the network card binding mode of the server in the distributed storage is the master-slave mode, the fault handling measure is to switch the network card; when the network card binding mode of the server in the distributed storage is the load balancing mode, the fault handling measure is server node isolation and the number of isolated server nodes does not exceed the maximum number of isolated nodes, which is obtained based on the total number of server nodes and the ratio threshold in the parameter configuration module. The ratio threshold is configured and adjusted according to the actual business pressure and the requirements for availability. Exemplarily, the ratio threshold is configured to 0.3, that is, the maximum number of isolated nodes is 30% of the total number of server nodes.

[0086] According to the network analysis result sequence and the detection period corresponding to the detection result, the network monitoring object where the abnormality occurs, that is, the network card, is obtained, and then the server information where the network card is located is obtained.

[0087] It is understandable that in the active-standby mode, if the fault is not restored after switching the network card, the server node where the network card is located will be isolated from the cluster. In the load balancing mode, if too many server nodes are isolated, the processing capacity of the cluster will be greatly reduced, which will also affect the front-end business. Therefore, when the number of servers in the cluster is relatively small, a network card will be randomly switched first. If the fault is not restored, another network card will be switched.

[0088] It should be noted that after the network card is switched or the server node is isolated, the distributed storage cluster will not have any abnormalities for a period of time, and will restore the network card status of the previously fault-handled network card or access the isolated server node.

[0089] After switching the network card or isolating the server node according to the fault handling measures, the data collection module and the anomaly detection module are run regularly, and new detection results are continuously obtained based on the newly collected monitoring data. When no network anomaly is identified based on the detection results, the fault handling score is calculated based on the fault recovery time.

[0090] It should be noted that the fault recovery time is obtained by obtaining the start time of the detection cycle corresponding to the detection result that identifies the existence of network anomalies, and the start time of the detection cycle corresponding to the detection result that does not identify the existence of network anomalies after fault processing, and calculating the difference between the two start times; then, the fault processing score is calculated according to the fault recovery time, including: if the fault recovery time is less than or equal to the expected recovery time, it is full marks; otherwise, the score corresponding to the proportion of the excess time is deducted from the full mark.

[0091] For example, the expected recovery time is 100 seconds, but the actual failure recovery time is 110 seconds, which is 10% longer than expected. Therefore, 10 points are subtracted from the full score of 100, and the final result is 90 points.

[0092] The fault handling module writes the calculated fault handling score into the log record where the corresponding detection time and detection result are located.

[0093] (4) Parameter configuration module

[0094] The parameter configuration module presets multiple monitoring indicators, network thresholds, business thresholds and other system parameters. It not only provides manual configuration methods, but also can adaptively modify several thresholds based on monitoring data, detection results and fault recovery time.

[0095] When the parameter configuration module identifies that the fault handling score is less than the score threshold, it triggers adaptive correction of several thresholds. Each time the correction is made, one of the thresholds is selected, that is, the adaptive correction network threshold or service threshold. The network threshold is the network delay threshold, and the service threshold is multiple drop thresholds, multiple time thresholds, and number thresholds in multiple abnormal scenario modes.

[0096] It should be noted that if the fault handling score is less than the score threshold, it means that the fault recovery time is relatively long. This may be because the network threshold or business threshold is relatively high, resulting in inaccurate network analysis results or business analysis results, which cannot be marked as abnormal in time. Even after translation, the correlation between the two is not improved, and the network abnormality is missed. At this time, the correction direction is to reduce, and the time when the fault recovery time exceeds the expected recovery time is used as the advance time. The correction period is obtained according to the advance time and the detection period corresponding to the detection result, that is, the start time and end time of the detection period corresponding to the detection result are subtracted from the advance time to obtain the correction period. The purpose of correcting the threshold is to detect network abnormalities within the advance correction period and resolve the fault as soon as possible.

[0097] Specifically, the network threshold or the service threshold is modified according to the detection result of the modification period, including:

[0098] ① According to the correction direction, the corresponding threshold is adjusted proportionally to obtain a new threshold, for example, the ratio is 5%. When the detection result within the correction period obtained according to the new threshold identifies the existence of network anomalies, the new threshold is used as the quasi-threshold.

[0099] It should be noted that the detection results within the revised period are obtained according to the new threshold, including:

[0100] According to the new threshold, the monitoring data of the back-end network / front-end business in the correction period is re-detected to obtain a new sequence to be detected; the historical sequence to be detected in the correction period is obtained from the business / network analysis result sequence; the abnormal coefficients of the new sequence to be detected and the historical sequence to be detected are calculated to obtain the detection results in the correction period.

[0101] If the network anomaly is not identified according to the corrected detection result, the new threshold is adjusted again in proportion according to the correction direction without exceeding the maximum number of adjustments until a new threshold that can identify the network anomaly is obtained. If the new threshold that can identify the network anomaly is still not obtained after the maximum number of adjustments is reached, switch to another threshold for correction.

[0102] ② When the anomaly detection module performs subsequent regular detection, it obtains the corresponding detection results and their anomaly coefficients according to the default threshold and quasi-threshold, takes the threshold corresponding to the high anomaly coefficient as the decision threshold, and records the corresponding fault handling score.

[0103] ③ If the number of quasi-thresholds used as decision thresholds is relatively large within the preset time period, and the corresponding fault processing scores are greater than or equal to the score threshold, the default threshold is modified to the quasi-threshold.

[0104] It should be noted that after the network threshold and the service threshold are lowered through adaptive correction, if the network anomaly is repeatedly restored in a short period of time, it may be because the network threshold or the service threshold is relatively low and it is misjudged as a network anomaly. At this time, the corresponding consecutive detection cycles are used as the correction period, and the correction direction is to increase. The network threshold or the service threshold is adjusted proportionally to obtain a new threshold. It is identified whether the detection result within the correction period is that there is no network anomaly. If it is not, the new threshold is used as the quasi-threshold, and the default threshold is further compared with the quasi-threshold in the manner of the above steps ②③ to determine whether the default threshold is corrected.

[0105] After a period of operation, the thresholds in the parameter configuration module will be more accurate, and the network anomaly monitoring of the distributed storage cluster will become more and more accurate.

[0106] Compared with the prior art, the network anomaly monitoring and processing system provided in this embodiment covers the entire process from data collection to anomaly detection, fault handling and parameter correction, ensuring comprehensive monitoring of the network status; the parameters are adaptively corrected according to the monitoring data and fault handling results, and the continuous feedback loop forms a closed-loop system, which promotes the self-regulation and optimization of the system, improves the accuracy of anomaly detection and the timeliness of fault handling, and enhances the flexibility and adaptability of the system in the face of different network environments and conditions. On the basis of the analysis results of the back-end network, combined with the analysis results of the degree of damage to the front-end business, by analyzing the correlation between the front and back ends, minor network anomalies that do not affect the business, as well as network anomalies caused by changes in normal business models, improve the accuracy of network anomaly detection. The adaptive threshold adjustment and decision-making process reduce the reliance on manual intervention and improve the efficiency of operation and maintenance; timely and accurate detection and processing of network anomalies improves the network experience of end users.

[0107] Those skilled in the art will appreciate that all or part of the processes of the above-mentioned embodiments can be implemented by instructing related hardware through a computer program, and the program can be stored in a computer-readable storage medium, wherein the computer-readable storage medium is a disk, an optical disk, a read-only storage memory, or a random access memory, etc.

[0108] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by any technician familiar with the technical field within the technical scope disclosed by the present invention should be covered within the protection scope of the present invention.

Claims

1. A network anomaly monitoring and processing system, characterized in that: include: Data acquisition module, anomaly detection module, fault handling module and parameter configuration module, among which, The data acquisition module is used to regularly collect monitoring data of the front-end business and the back-end network according to the monitoring indicators of the parameter configuration module; The anomaly detection module is used to regularly detect the monitoring data of the front-end service and the back-end network according to the threshold values ​​of the parameter configuration module, and identify whether there is a network anomaly according to the detection results, and if there is a network anomaly, notify the fault handling module; The fault handling module is used to determine the fault handling measures and handle the fault according to the analysis results of the anomaly detection module, and calculate the fault handling score according to the fault recovery time; The parameter configuration module is used to preset various indicators and thresholds, and to adaptively modify several thresholds according to the monitoring data, detection results and fault recovery time.

2. The network anomaly monitoring and processing system according to claim 1, characterized in that: Determining fault handling measures according to the detection result of the abnormality detection module includes: If the abnormality coefficient in the detection result is less than the fault coefficient threshold, a network abnormality alarm is sent; otherwise, when the network card binding mode of the server in the distributed storage is the master-slave mode, the fault handling measure is to switch the network card; when the network card binding mode of the server in the distributed storage is the load balancing mode, the fault handling measure is to isolate the server nodes and the number of isolated server nodes does not exceed the maximum number of isolated nodes, and the maximum number of isolated nodes is obtained based on the total number of server nodes and the proportional threshold in the parameter configuration module.

3. The network anomaly monitoring and processing system according to claim 1, characterized in that: The fault recovery duration is obtained by obtaining the start time of the detection period corresponding to the detection result that identifies the existence of the network anomaly, and the start time of the detection period corresponding to the detection result that does not identify the existence of the network anomaly after the fault is handled, and calculating the difference between the two start times; The fault handling score is calculated based on the fault recovery time, including: if the fault recovery time is less than or equal to the expected recovery time, it is full marks, otherwise, the score is obtained by deducting the score corresponding to the proportion of the excess time from the full mark.

4. The network anomaly monitoring and processing system according to claim 1, characterized in that: The detection results in the anomaly detection module include: a business analysis result sequence, a network analysis result sequence and an anomaly coefficient; the business analysis result sequence is based on multiple abnormal scenario modes, regularly analyzes the monitoring data of the front-end business according to multiple business thresholds in each threshold, and uses 0 and 1 as normal and abnormal identifications respectively to obtain the sequence; the network analysis result sequence is based on multiple network thresholds in each threshold and a clustering algorithm regularly analyzes the monitoring data of the back-end network, and uses 0 and 1 as normal and abnormal identifications respectively to obtain the sequence.

5. The network anomaly monitoring and processing system according to claim 4, characterized in that: If the anomaly coefficient is greater than or equal to the similarity threshold, there is a network anomaly; the anomaly coefficient is obtained by the following steps: After extracting the sequences to be detected from the business analysis result sequence and the network analysis result sequence respectively according to the length of the detection sequence, if the number of abnormal identifiers at the same position in the two sequences to be detected is greater than or equal to the number of normal identifiers, the Pearson correlation coefficient of the two sequences to be detected is calculated, and multiple Pearson correlation coefficients are calculated after multiple translations of the time points of any sequence to be detected; If the number of Pearson correlation coefficients greater than or equal to the similarity threshold is greater than half, the largest Pearson correlation coefficient is taken as the abnormal coefficient, otherwise, the abnormal coefficient is 0.

6. The network anomaly monitoring and processing system according to claim 4, characterized in that: The adaptive correction of several thresholds according to monitoring data, detection results and fault recovery time includes: correcting the network threshold and correcting the service threshold; selecting any one of the thresholds each time the correction is made.

7. The network anomaly monitoring and processing system according to claim 6, characterized in that: When the parameter configuration module identifies that the fault processing score is less than the score threshold, the correction direction is to reduce, and the time by which the fault recovery time exceeds the expected recovery time is used as the advance time. The correction time period is obtained according to the advance time and the detection cycle corresponding to the detection result, and the network threshold or the service threshold is corrected according to the detection result of the correction time period.

8. The network anomaly monitoring and processing system according to claim 7, characterized in that: The modifying of the network threshold or the service threshold according to the detection result of the modification period includes: A new threshold is obtained by proportionally adjusting the corresponding threshold according to the correction direction. When the detection result within the correction period obtained according to the new threshold identifies the existence of network anomaly, the new threshold is used as the quasi-threshold; When the anomaly detection module performs subsequent regular detection, the corresponding detection results and their anomaly coefficients are obtained according to the default threshold and the quasi-threshold, the threshold corresponding to the high anomaly coefficient is used as the decision threshold, and the corresponding fault handling score is recorded; If the number of quasi-thresholds used as decision thresholds is relatively large within a preset period, and the corresponding fault processing scores are greater than or equal to the score threshold, the default threshold is modified to the quasi-threshold.

9. The network anomaly monitoring and processing system according to claim 8, characterized in that: Acquiring the detection result within the correction period according to the new threshold includes: According to the new threshold, the monitoring data of the back-end network / front-end business in the correction period is re-detected to obtain a new sequence to be detected; the historical sequence to be detected in the correction period is obtained from the business / network analysis result sequence; the abnormal coefficients of the new sequence to be detected and the historical sequence to be detected are calculated to obtain the detection results in the correction period.

10. The network anomaly monitoring and processing system according to claim 6, characterized in that: The network threshold is a network delay threshold, and the service threshold is a plurality of drop thresholds, a plurality of time thresholds and a number threshold in a plurality of abnormal scenario modes.

Images