Real-time monitoring and fault detection method and system for server network

Abstract

The invention provides a real-time monitoring and fault detection method and system for a server network, and the method achieves the real-time monitoring, anomaly detection and fault positioning of a cloud computing power system through collecting the multi-dimensional indexes of a CPU, a GPU, storage I / O, network performance and the like. The method comprises the specific steps of data acquisition, data preprocessing, real-time monitoring, anomaly detection, fault positioning and alarm and response, and through the method steps, the defect that a traditional monitoring scheme in a cloud computing power server network cannot meet the requirements of multiple dimensions, high dynamics and complex dependency relationships is overcome.

Description

Technical Field

[0001] The present invention relates to the technical field of server data monitoring, and in particular to a real-time monitoring and fault detection method and system for a server network. Background Art

[0002] With the rapid development of cloud computing technology, cloud computing has become an important part of modern computing architecture. Cloud computing integrates physical computing resources through virtualization technology to provide efficient, flexible, and on-demand computing power, and is widely used in big data processing, artificial intelligence training, video rendering and other fields. However, the complexity and dynamics of cloud computing systems make them prone to performance bottlenecks, resource waste, and failures during operation. For example, high loads on computing nodes, I / O bottlenecks in storage systems, and network delays may lead to a decline in the quality of cloud computing services, or even task interruptions.

[0003] At present, cloud computing servers usually rely on traditional monitoring solutions, which are often only suitable for static monitoring of a single indicator and cannot meet the performance monitoring and fault detection requirements of multi-dimensional, highly dynamic and distributed architectures in cloud computing scenarios. In addition, the fault location methods of existing methods are relatively simple and lack the ability to analyze the complex dependencies between cloud computing resources, resulting in low troubleshooting efficiency.

[0004] Therefore, a real-time monitoring and fault detection method suitable for cloud computing servers is proposed to solve the above technical problems. Summary of the invention

[0005] The present invention provides a real-time monitoring and fault detection method and system for a server network, which are used to solve the technical problems mentioned in the background technology.

[0006] To solve the above problems, the present invention provides the following technical solution: a real-time monitoring and fault detection method for a server network, comprising the following steps: S100, acquiring operation status data from the server and network devices through a collection tool, wherein the operation status data includes but is not limited to CPU usage, GPU utilization, memory occupancy, disk I / O rate, disk latency, bandwidth utilization, packet loss rate, and network latency; S200, cleaning, standardizing and aggregating the collected operation status data, wherein: Data cleaning is done by removing outliers or missing values; The normalized data z is calculated by the following formula: ; Among them, x is the original data, μ is the mean, and σ is the standard deviation; Data aggregation is done by taking sliding average of data at multiple time points; S300, inputting the processed data into a real-time monitoring system, and displaying the operating status of the server and network equipment through a dashboard, wherein the dashboard includes a trend chart of performance indicators and a highlight mark of abnormal status; S400, performing anomaly detection on the operation status data based on a set static threshold or a dynamic threshold, wherein the dynamic threshold is calculated based on the mean and standard deviation of historical data; S500, when an abnormal state is detected, locating the fault source through log analysis and dependency analysis methods; S600 handles faults by sending alarm notifications and triggering predefined automated response scripts.

[0007] Furthermore, in step S200, the data cleaning step uses the 3-times standard deviation method to remove outliers, and a data point x is considered an outlier when it meets the following conditions: |x−μ|>3σ; Furthermore, in step S200, the sliding average processing is calculated by the sliding average calculation formula: ; Among them, MA t is the sliding average of time t, n is the sliding window size, X t-i is the sampling data of the previous n time points.

[0008] Furthermore, in step S400, the abnormality detection includes two methods: Anomaly detection based on static thresholds, where the static thresholds are fixed values ​​predefined by the user based on the operating environment; Anomaly detection based on dynamic threshold, the dynamic threshold N t Calculated by the following formula: N t = μ ± kσ; Among them, k is the dynamic coefficient, and the value of k is 2-3.

[0009] Furthermore, in step S500, the following steps are included: S501, locate the log records related to the fault through keyword matching and timestamp alignment technology; S502, construct a dependency graph between servers and services, and quickly locate the source of the fault through impact range analysis.

[0010] Furthermore, in step S600, a predefined automation script is triggered according to the fault type, including but not limited to automatically restarting services, dynamically expanding server resources, or switching backup nodes.

[0011] The present invention also provides a real-time monitoring and fault detection system for a server network, which is used to implement the above method, including: A data collection module is used to collect operation status data from servers and network devices; Data processing module, used to clean, standardize and aggregate the collected data; Monitoring module, used to display the operating status of servers and network devices in real time; An anomaly detection module, used to detect abnormal conditions based on static thresholds or dynamic thresholds; Fault detection module, used to locate fault sources through log analysis and dependency analysis; The alarm and response module is used to send alarm notifications and execute automated fault handling scripts.

[0012] Furthermore, the data collection module is implemented based on Prometheus or Telegraf tools, and supports data collection in Pull mode and Push mode.

[0013] Furthermore, the monitoring module builds a dashboard through the Grafana visualization tool to display performance indicators including CPU usage, GPU utilization, memory occupancy, disk I / O rate, disk latency, bandwidth utilization, packet loss rate and network latency.

[0014] Compared with the prior art, the present invention has at least the following beneficial effects: The present invention solves the defect that traditional monitoring solutions in cloud computing server networks cannot meet the requirements of multi-dimensionality, high dynamics and complex dependencies through data cleaning, standardization, aggregation, dynamic threshold detection, log analysis and dependency analysis. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0016] Figure 1 is a flowchart of the steps of a real-time monitoring and fault detection method for a server network in an embodiment of the present invention; Figure 2 It is a structural diagram of a real-time monitoring and fault detection system for a server network in an embodiment of the present invention. DETAILED DESCRIPTION

[0017] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work belong to the protection scope of the present invention. It is understood that the drawings are only provided for reference and illustration, and are not used to limit the present invention. The connection relationship shown in the drawings is only for the convenience of clear description and does not limit the connection method.

[0018] The present invention discloses a real-time monitoring and fault detection method for a server network. By collecting multiple types of performance data such as CPU, GPU, memory, storage I / O, network performance, etc., combined with dynamic threshold detection and dependency analysis, the method realizes real-time monitoring, anomaly detection and fault location of a cloud computing server network, and specifically discloses the following steps: Figure 1 As shown: Step S100, data collection In the cloud computing server network, the performance parameters of each node are collected through Prometheus, and the collection frequency is once every 5 seconds. The collected monitoring indicators include: 1. Compute node performance indicators: CPU usage: The CPU usage percentage of each virtual machine or physical machine.

[0019] GPU utilization: For AI training tasks, collect GPU utilization and video memory occupancy.

[0020] 2. Storage performance indicators: Disk I / O rate: The amount of data read and written per second (unit: MB / s).

[0021] Disk latency: The average latency of I / O operations (unit: ms).

[0022] 3. Network performance indicators: Bandwidth Utilization: The percentage of real-time traffic used by the network interface.

[0023] Packet loss rate: Packet loss on the network link.

[0024] Network latency: The network response time from the compute node to the storage node.

[0025] For example, a cloud computing cluster contains 10 computing nodes and 5 storage nodes. Some of the collected data is shown in Table 1 below: Table 1: Timestamp node CPU usage GPU Utilization Disk I / O (MB / s) Disk latency (ms) Bandwidth Utilization Network latency (ms) 10:00:00 Node1 65% 70% 200 2.5 50% 12 10:00:05 Node2 90% 80% 300 3.2 70% 15 Purpose: Through real-time collection and analysis of multiple performance indicators such as CPU, GPU, storage I / O, and network latency, the solution can fully cover the core resources and performance bottlenecks of the cloud computing system, form a multi-dimensional monitoring perspective, and thus fully understand the operating status of the system.

[0026] Step S200: Data preprocessing The collected raw data may contain noise or outliers, so data cleaning, standardization and aggregation are required.

[0027] 1. Data cleaning The 3-sigma method is used to eliminate outliers. Take the CPU usage of Node1 for the last 10 times as an example: [65%, 70%, 68%, 90%, 67%, 71%, 300%, 69%, 70%, 68%] After calculation, the mean value is: μ=87.8% Standard deviation σ = 69.2% After eliminating outliers, the normal range according to the 3-times standard deviation method is: [μ-3σ, μ+3σ]=[-119.8%, 295.4%] Since 300% is beyond the range, it is removed. The cleaned data is: [65%, 70%, 68%, 90%, 67%, 71%, 69%, 70%, 68%] 2. Data Standardization The cleaned data is standardized to a standard normal distribution (mean is 0, standard deviation is 1). The formula for the standardized value z is:

[0028] Among them, x is the original data, μ is the mean, and σ is the standard deviation; In this embodiment, the mean of the cleaned data is μ=70%, and the standard deviation is σ=7%. The standardized results of the first to third data points 65%, 70%, and 68% are: Z 1 =(65-70) / 7=-0.71; Z 2 =(70-70) / 8=0; Z 3 =(68-70) / 7=-0.29; Data normalization can eliminate the impact of dimension. For example, CPU utilization is a percentage (0-100%), disk I / O rate is in MB / s, and network latency is in ms. It is unreasonable to compare these indicators directly. Through normalization, the data is converted to a dimensionless standard normal distribution (mean is 0, standard deviation is 1), making different indicators comparable.

[0029] 3. Data Aggregation The sliding average method is used to aggregate the data to reduce the impact of short-term fluctuations. The sliding average formula is: ; Among them, MA t is the sliding average of time t, n is the sliding window size, X t-i is the sampling data of the previous n time points.

[0030] Assume the sliding window size n=3 and aggregate the standardized data: CPU data after cleaning: [65%, 70%, 68%, 90%, 67%, 71%, 69%, 70%, 68%] Aggregate calculation (sliding window n=3): First aggregate value: MA1 = (65% + 70% + 68%) / 3 = 67.67% Second aggregate value: MA2 = (70% + 68% + 90%) / 3 = 76% The third aggregate value: MA3 = (68% + 90% + 67%) / 3 = 75% Purpose: Through preprocessing methods such as data cleaning (such as 3-sigma method) and aggregation (sliding average), the solution can effectively remove noise and outliers, making anomaly detection more reliable and stable.

[0031] Step S300, implement monitoring The processed data is displayed in real time through the Grafana dashboard, including: 1. CPU usage trend chart: displays the CPU usage change curve of each server, and the aggregated data is displayed in the form of a broken line.

[0032] 2. Network equipment performance meter: The dashboard displays indicators such as bandwidth utilization, packet loss rate and latency, and abnormal values ​​are marked in red.

[0033] Step S400, abnormality detection 1. Static threshold detection Static thresholds are suitable for scenarios where the business has fixed requirements for certain indicators. For example, the CPU usage limit of a design computing task is 85%, and when the CPU usage of a node exceeds 85%, an abnormal alarm is triggered.

[0034] 2. Dynamic Threshold Detection Dynamic threshold N t Calculated by the following formula: N t = μ ± kσ; Wherein, k is a dynamic coefficient, and the value of k is 2-3. In this embodiment, k is 2.

[0035] The dynamic threshold is calculated by the mean and standard deviation of historical data. Taking the CPU usage of Node1 as an example, the historical data (after cleaning) is: [65%, 70%, 68%, 90%, 67%, 71%, 69%, 70%, 68%]; Mean μ = 70.89%; Standard deviation σ = 6.95%; Dynamic threshold range: [μ-2σ, μ+2σ]=[56.9%, 84.8%] When the CPU usage exceeds 84.8%, it is considered abnormal.

[0036] Purpose: Dynamic thresholds adaptively adjust the detection range based on the statistical characteristics (mean and standard deviation) of historical data. They can adapt to the dynamics and complexity of cloud computing systems and avoid false positives or negatives caused by unreasonable fixed thresholds in traditional static threshold methods.

[0037] Step S500: Fault detection and location 1. Log analysis Combined with the ELK system, analyze the logs during the abnormal period, extract keywords (such as "timeout" and "error"), and locate the possible cause of the failure. For example, a task times out when running on Node1, and the log shows: "I / O timeout atstorage node".

[0038] 2. Dependency Analysis Based on the resource distribution in the cloud computing system, a dependency graph is constructed between nodes. For example, computing node Node1 depends on storage node Storage1 and network switch SwitchA. When Node1 is abnormal, the impact range is analyzed to confirm that the problem is caused by bandwidth congestion of SwitchA.

[0039] Purpose: With the help of log analysis and dependency graphs, the solution can quickly locate the fault point and analyze its impact on other nodes. This capability significantly improves the efficiency of problem troubleshooting and location, and reduces system downtime.

[0040] Step S600: Alarm and response 1. Alarm notification: send alarm information via SMS, email and instant messaging tools (such as Slack). For example: Alarm content: Time: 2024-12-19 10:05:00 Node: Node2 Abnormal indicator: CPU usage = 90% (exceeding the dynamic threshold of 84.8%) 2. Automated response Trigger an automatic recovery script, for example: Restart abnormal services Dynamically migrate tasks to other computing nodes.

[0041] Purpose: The system supports automatic alarm and rapid response mechanisms (such as task migration and service restart), and can take timely measures when problems occur, reduce the impact on user service quality, and improve the system's fault tolerance and operational reliability.

[0042] The present invention solves the defect that traditional monitoring solutions in cloud computing server networks cannot meet the requirements of multi-dimensionality, high dynamics and complex dependencies through data cleaning, standardization, aggregation, dynamic threshold detection, log analysis and dependency analysis.

[0043] After testing, this embodiment can achieve the following effects: 1. Accuracy: The accuracy of anomaly detection is over 95%.

[0044] 2. Response time: The average time from detection to alarm does not exceed 5 seconds.

[0045] 3. Stability: Successfully avoided multiple potential server downtime risks.

[0046] like Figure 2 As shown, the present invention also discloses a real-time monitoring and fault detection system for a server network, which is used to implement the above method, including: The data collection module 11 is used to collect operation status data from the server and network equipment; The data processing module 12 is used to clean, standardize and aggregate the collected data; Monitoring module 13, used to display the operating status of servers and network devices in real time; An anomaly detection module 14, used to detect an abnormal state based on a static threshold or a dynamic threshold; Fault detection module 15, used to locate the fault source through log analysis and dependency analysis; The alarm and response module 16 is used to send alarm notifications and execute automated fault handling scripts.

[0047] The data collection module 11 is implemented based on Prometheus or Telegraf tools, and supports data collection in Pull mode and Push mode.

[0048] The monitoring module 13 builds a dashboard through the Grafana visualization tool to display performance indicators including CPU usage, GPU utilization, memory occupancy, disk I / O rate, disk latency, bandwidth utilization, packet loss rate and network latency.

[0049] In the specification and claims of the present application, the words "include / comprise" and the words "have / include" and their variations are used to specify the existence of stated features, values, steps or components, but do not exclude the existence or addition of one or more other features, values, steps, components or combinations thereof.

Claims

1. A real-time monitoring and fault detection method for a server network, characterized in that: The following steps are involved: S100, acquiring operation status data from the server and network devices through a collection tool, wherein the operation status data includes but is not limited to CPU usage, GPU utilization, memory occupancy, disk I / O rate, disk latency, bandwidth utilization, packet loss rate, and network latency; S200, cleaning, standardizing and aggregating the collected operation status data, wherein: Data cleaning is done by removing outliers or missing values; The normalized data z is calculated by the following formula: ； Among them, x is the original data, μ is the mean, and σ is the standard deviation; Data aggregation is done by taking sliding average of data at multiple time points; S300, inputting the processed data into a real-time monitoring system, and displaying the operating status of the server and network equipment through a dashboard, wherein the dashboard includes a trend chart of performance indicators and a highlight mark of abnormal status; S400, performing anomaly detection on the operation status data based on a set static threshold or a dynamic threshold, wherein the dynamic threshold is calculated according to the mean and standard deviation of the historical data; S500, when an abnormal state is detected, locating the fault source through log analysis and dependency analysis methods; S600 handles faults by sending alarm notifications and triggering predefined automated response scripts.

2. The real-time monitoring and fault detection method for a server network according to claim 1, characterized in that: In step S200, the data cleaning step uses the 3-times standard deviation method to remove outliers. A data point x is considered an outlier if it meets the following conditions: |x−μ|>3σ.

3. The real-time monitoring and fault detection method for a server network according to claim 2, characterized in that: In step S200, the sliding average processing is calculated by the sliding average calculation formula: ； Among them, MA t is the sliding average of time t, n is the sliding window size, X t-i is the sampling data of the previous n time points.

4. The real-time monitoring and fault detection method for a server network according to claim 1, characterized in that: In step S400, the abnormality detection includes two methods: Anomaly detection based on static thresholds, where the static thresholds are fixed values ​​predefined by the user based on the operating environment; Anomaly detection based on dynamic threshold, the dynamic threshold N t Calculated by the following formula: N t =μ±kσ; Among them, k is the dynamic coefficient, and the value of k is 2-3.

5. The real-time monitoring and fault detection method for a server network according to claim 1, characterized in that: In step S500, the following steps are included: S501, locate the log records related to the fault through keyword matching and timestamp alignment technology; S502, construct a dependency graph between servers and services, and quickly locate the source of the fault through impact range analysis.

6. The real-time monitoring and fault detection method for a server network according to claim 1, characterized in that: In step S600, a predefined automation script is triggered according to the fault type, including but not limited to automatically restarting services, dynamically expanding server resources, or switching standby nodes.

7. A real-time monitoring and fault detection system for a server network, used to implement the method of any one of claims 1 to 6, characterized in that: include: A data collection module is used to collect operation status data from servers and network devices; Data processing module, used to clean, standardize and aggregate the collected data; Monitoring module, used to display the operating status of servers and network devices in real time; An anomaly detection module, used to detect abnormal conditions based on static thresholds or dynamic thresholds; Fault detection module, used to locate fault sources through log analysis and dependency analysis; The alarm and response module is used to send alarm notifications and execute automated fault handling scripts.

8. The real-time monitoring and fault detection system for a server network according to claim 7, characterized in that: The data collection module is implemented based on Prometheus or Telegraf tools, and supports data collection in Pull mode and Push mode.

9. The real-time monitoring and fault detection system for a server network according to claim 8, characterized in that: The monitoring module builds a dashboard through the Grafana visualization tool to display performance indicators including CPU usage, GPU utilization, memory occupancy, disk I / O rate, disk latency, bandwidth utilization, packet loss rate, and network latency.

Images