Troubleshooting methods, software products, electronic devices and storage media

Abstract

This application provides a fault handling method, program product, electronic device, and storage medium, applied in the field of computer technology. The method includes: acquiring network state data and service state data of a network file system within the current detection period to obtain current network state data and current service state data; performing fault risk detection on the network file system by fusing the current network state data and current service state data; and, if a fault risk is detected in the network file system, and the risk level of the fault risk is a target risk level among multiple risk levels, performing a processing operation corresponding to the target risk level on the network file system, where one risk level corresponds to one processing operation. This application solves the problems of low fault detection accuracy and insufficient intelligence in fault handling methods in related technologies, achieving the effect of improving fault detection accuracy and increasing the intelligence of fault handling.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a fault handling method, program product, electronic device and storage medium. Background Technology

[0002] With the growth of data volume, the increasing demands for service reliability from users, and the growing complexity of anomaly detection in network file systems, maintaining the reliability of network file systems faces numerous challenges. Current technologies typically employ static thresholds and single-dimensional detection, resulting in significant false positive and false negative rates. Maintenance personnel must spend considerable time processing invalid alarms, and the probability and real-time detection of subtle faults are low. Furthermore, fault handling methods in these technologies have significant limitations after fault detection. Automated handling often involves simply restarting the system, which can significantly impact services within the network file system, necessitating manual intervention in many fault scenarios.

[0003] It is evident that the fault handling methods in related technologies suffer from low fault detection accuracy and insufficient intelligence in fault handling. Summary of the Invention

[0004] This application provides a fault handling method, program product, electronic device, and storage medium to at least solve the problems of low fault detection accuracy and insufficient intelligence in fault handling methods in the related art.

[0005] This application provides a fault handling method, comprising: acquiring network state data and service state data of a network file system within the current detection period to obtain current network state data and current service state data; performing fault risk detection on the network file system by fusing the current network state data and the current service state data; and performing processing operations corresponding to the target risk level on the network file system when a fault risk is detected and the risk level of the fault risk is a target risk level among multiple risk levels, wherein one of the multiple risk levels corresponds to one processing operation.

[0006] This application also provides a fault handling apparatus, comprising: an acquisition unit, configured to acquire network state data and service state data of a network file system within the current detection period, to obtain current network state data and current service state data; a first detection unit, configured to perform fault risk detection on the network file system by fusing the current network state data and the current service state data; and an execution unit, configured to perform a processing operation corresponding to the target risk level on the network file system when a fault risk is detected in the network file system and the risk level of the fault risk in the network file system is a target risk level among multiple risk levels, wherein one of the multiple risk levels corresponds to one processing operation.

[0007] This application also provides an electronic device, including: a memory for storing a computer program; and a processor for implementing the steps of any of the above-described fault handling methods when executing the computer program.

[0008] This application also provides a computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor, implements the steps of any of the above-described fault handling methods.

[0009] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of any of the above-described fault handling methods.

[0010] This application improves the accuracy of fault risk detection by integrating network-state data and service-state data for multimodal fault risk detection. Furthermore, it enables automatic execution of corresponding fault handling operations based on the fault risk level obtained from fault risk detection, thus handling faults automatically and intelligently. Therefore, it solves the technical problems of low fault detection accuracy and insufficient intelligence in fault handling methods in related technologies, achieving the technical effects of improving fault detection accuracy and enhancing the intelligence of fault handling. Attached Figure Description

[0011] To more clearly illustrate the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0012] Figure 1 This is a schematic diagram illustrating an application scenario of a fault handling method according to an embodiment of this application.

[0013] Figure 2 This is a flowchart illustrating an optional fault handling method according to an embodiment of this application.

[0014] Figure 3 This is a schematic diagram of an optional fault handling method according to an embodiment of this application.

[0015] Figure 4 This is an architecture diagram of an optional fault handling system according to an embodiment of this application.

[0016] Figure 5 This is a structural block diagram of an optional fault handling device according to an embodiment of this application. Detailed Implementation

[0017] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the protection scope of this application.

[0018] It should be noted that, in the description of this application, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. The terms "first," "second," etc., in this application are used to distinguish similar objects and are not used to describe a specific order or sequence.

[0019] To enable those skilled in the art to better understand the present application, the present application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0020] According to one aspect of the embodiments of this application, a fault handling method is provided. Optionally, in this embodiment, the above-described fault handling method may be applied, but is not limited to, to applications such as... Figure 1 The hardware environment shown includes terminal device 102 and server 104. Server 104 can be connected to terminal device 102 via a network and can be used to provide services (e.g., application services, etc.) to terminal device 102 or clients installed on terminal device 102. A database can be set up on server 104 or independently of server 104 to provide data storage services for server 104.

[0021] The aforementioned network may include, but is not limited to, at least one of the following: wired network and wireless network. The aforementioned wired network may include, but is not limited to, at least one of the following: wide area network (WAN), metropolitan area network (MAN), and local area network (LAN). The aforementioned wireless network may include, but is not limited to, at least one of the following: Wireless Fidelity (WIFI) and Bluetooth. Terminal device 102 may be, but is not limited to, a personal computer (PC), mobile phone, tablet computer, etc. Server 104 may be, but is not limited to, a cloud server, server cluster, or other server types.

[0022] The fault handling method of this application embodiment can be executed by server 104, terminal device 102, or jointly by server 104 and terminal device 102. Alternatively, the fault handling method of this application embodiment can be executed by a client installed on the terminal device 102.

[0023] Taking the fault handling method in this embodiment executed by server 104 as an example, Figure 2 This is a flowchart illustrating an optional fault handling method according to an embodiment of this application, such as... Figure 2 As shown, the process of this method may include steps S202 to S206.

[0024] Step S202: Obtain network state data and service state data of the network file system within the current detection period to obtain the current network state data and the current service state data.

[0025] Step S204: By fusing current network state data and current service state data, fault risk detection is performed on the network file system.

[0026] Step S206: If a fault risk is detected in the network file system and the risk level of the fault risk in the network file system is the target risk level among multiple risk levels, the processing operation corresponding to the target risk level is performed on the network file system, wherein one of the multiple risk levels corresponds to one processing operation.

[0027] The fault handling method in this embodiment can be applied to the field of computer technology, specifically to scenarios involving fault risk detection and fault handling in network file systems. A network file system is a distributed file system protocol, a network abstraction built upon a file system, whose core function is to achieve cross-network transparency.

[0028] File sharing allows remote clients to access network file systems over a network in a manner similar to that of a local file system.

[0029] With the growth of data volume, the increasing demands on the reliability of network file systems (NFS) services, and the growing complexity of anomaly detection within NFSs, maintaining the reliability of NFSs faces numerous challenges. Current technologies typically employ static thresholding and single-dimensional detection, resulting in significant false positive and false negative rates. Maintenance personnel must spend considerable time processing invalid alarms, and the probability and real-time detection of subtle faults are low. Furthermore, many technologies employ full polling for fault detection. Full polling is a communication mechanism where the client periodically sends requests to the server to retrieve all data. This method consumes significant CPU resources, impacting the normal operation of the NFS.

[0030] Furthermore, the methods for handling faults detected in related technologies have significant limitations. Automatic handling mostly involves simply restarting the system, which can have a significant impact on services in the network file system. Many fault scenarios require human intervention.

[0031] To at least partially solve the above-mentioned technical problems, in this embodiment, a dual-modal perception method of network state and service state is used to detect fault risks, and a dynamic adaptive control intelligent fault handling method is used to obtain network state data and service state data in the network file system. Based on the obtained data, fault risk analysis is performed to obtain the fault risk level, and corresponding fault handling operations are automatically performed according to the risk level, thus realizing an intelligent and complete process from fault risk detection to fault risk handling.

[0032] In this embodiment, the network state data and service state data of the network file system within the current detection period are first obtained to obtain the current network state data and current service state data. Network state data and service state data are two different levels of data processing in network communication, corresponding to the underlying transmission and the upper-layer application, respectively. Network state data refers to the raw data units transmitted at the lower layers of the network protocol stack (such as the physical layer, data link layer, and network layer), such as bit streams, data frames, and receive and send queue information; service state data refers to the data in the application layer that is oriented towards business logic processing, such as user requests and request processing results.

[0033] It should be noted that the network-state data and service-state data in this embodiment refer to the data used for fault risk detection. By selectively acquiring this data, performance overhead can be reduced, and the operating efficiency of the network file system can be improved. Optionally, the network-state data may include connection status data, such as the frequency of connection establishment and termination, connection duration, etc., used to monitor the stability of network connections; transmission performance indicators, such as data transmission volume, transmission latency, packet loss rate, etc., used to identify faults such as network congestion, abnormal latency, or abnormal data transmission; network queue length, such as send queue length and receive queue length, used to reflect the network's information reception and transmission status, and to detect data backlog at the network layer. The service-state data may include operation statistics, such as operations per second, read / write operation response time, operation recognition rate, etc., used to evaluate the service's processing capacity and stability; error logs and event records, recording abnormal information and error events during the service process, used for fault location and root cause analysis.

[0034] After acquiring the current network state data and current service state data, fault risk detection is performed on the network file system by fusing these two sets of data. Here, different aspects of the current network state data and current service state data can be integrated to comprehensively detect the fault risk of the network file system within the current detection period, and to determine the location and risk level of the fault risk. For example, if the connection status data in the network state data indicates unstable connection, and the service state data indicates abnormal event reception, it can be determined that the fault risk originates from the network state data. Furthermore, the risk level can be determined based on the persistence of connection instability events. For instance, if connection instability fault risk is detected in several consecutive detection periods, the risk level can be determined to be high; if connection instability fault risk is detected only in the current detection period, the risk level can be determined to be low.

[0035] If a network file system (NFS) is detected to have a fault risk, and the risk level of this fault risk is the target risk level among multiple risk levels, then the processing operation corresponding to the target risk level is performed on the NFS. Here, one risk level corresponds to one processing operation. Optionally, for low-risk cases, only the current fault risk can be recorded to prevent excessive intervention from affecting the stability of the NFS; for high-risk cases, the module or system corresponding to the fault risk can be restarted; for even higher-risk cases, the entire NFS can be saved and restarted, or relevant operations and maintenance personnel can be notified to handle the fault risk.

[0036] The embodiments provided in this application obtain network state data and service state data of the network file system within the current detection period to obtain current network state data and current service state data. By fusing the current network state data and current service state data, fault risk detection is performed on the network file system. When a fault risk is detected in the network file system, and the risk level of the fault risk is a target risk level among multiple risk levels, the processing operation corresponding to the target risk level is performed on the network file system. Here, one of the multiple risk levels corresponds to one processing operation. This solves the technical problems of low fault detection accuracy and insufficient fault processing intelligence in fault processing methods in related technologies, and improves the accuracy of fault detection and the intelligence of fault processing.

[0037] In one exemplary embodiment, the method further includes: using a network probe to collect network state data of the network file system at a first sampling interval, wherein the network state data of the network file system is used to indicate the queue length of the receiving queue and the queue length of the sending queue of the network file system; and using a service probe to collect service state data of the network file system at a second sampling interval, wherein the service state data of the network file system is used to indicate the number of received requests and the number of processed requests, wherein the number of received requests is the number of processed requests received by the network file system and the number of processed requests is the number of processed requests processed by the network file system.

[0038] To obtain network-state and service-state data for fault risk detection, network probes and service probes are used to collect network-state and service-state data respectively. In this embodiment, network probes are used to collect network-state data of the network file system at a first sampling interval, and service probes are used to collect service-state data of the network file system at a second sampling interval. Here, the first and second sampling intervals can be set empirically; for example, the first sampling interval can be 10 seconds and the second sampling interval can be 5 seconds. This embodiment does not impose any limitations on these settings. It should be noted that both the first and second sampling intervals must be less than one detection cycle, that is, ensuring that network-state and service-state data are sampled at least once within one detection cycle.

[0039] In this embodiment, the sampled network state data of the network file system is used to indicate the queue length of the network file system's receive queue and the queue length of the network file system's send queue. The queue length of the receive queue and the queue length of the network file system's send queue can reflect the accumulation and congestion of received and sent data, thereby detecting whether there is a risk of failure in the network state data.

[0040] The `netstat` command is used to display network-related data such as network connections, routing tables, and interface statistics. Optionally, network probes can use `netstat` to obtain real-time Transmission Control Protocol (TCP) connection status data. Based on the polling mechanism of the `netstat-tunpalx` command, network probes can perform sampling tasks at 10-second intervals and output real-time data from the network file system's receive and send queues. Since the network file system typically uses the Transmission Control Protocol / Internet Protocol (TCP / IP) protocol and communicates via the default port 2049, regular expressions can be used to filter network connection information connected to or originating from port 2049 to specifically obtain network transmission activity output related to the network file system service. Here, the network probe's information collection command can be: `nohup netstat -tunpalx | grep ':2049' > / var / log / recvq.log &`.

[0041] In this embodiment, the sampled network file system service-state data is used to indicate the number of received requests and the number of processed requests. The number of received requests refers to the number of processing requests received by the network file system, and the number of processed requests refers to the number of processing requests processed by the network file system. The number of received requests and the number of processed requests can reflect whether the network file system's task execution for receiving and processing requests is normal, and can detect whether there is a risk of the request processing service getting stuck.

[0042] Optionally, the service probe can monitor the reception and processing status of service requests using the Operations Per Second (OPS) statistics command for the network file system service. In the file system's OPS metric, the `total` line typically summarizes the total number of all types of request operations (e.g., read, write, metadata operations, etc.), including the number of received requests and the number of processed requests. The service probe can parse the `total` line data at 5-second sampling intervals to obtain the required number of received and processed requests. Here, the service probe's information collection command can be: `nfs_stat_tool show_icfs_ops > / var / log / ops_metrics.log`.

[0043] In this embodiment, by using network probes and service probes, data required for fault risk detection can be collected in a targeted manner, which can improve data collection efficiency and reduce the performance loss caused by data collection.

[0044] In an exemplary embodiment, in a network file system, queue faults of the receiving queue are recorded in a first fault list, queue faults of the sending queue are recorded in a second fault list, and service-state faults of the network file system are recorded in a third fault list. A service-state fault of the network file system refers to a situation where the number of received requests and the number of processed requests meet specified fault conditions. The specified fault conditions are: the number of received requests is greater than the number of processed requests, and the number of received requests and the number of processed requests indicated by two consecutive sets of service-state data have not changed. Fault risk detection of the network file system is performed by fusing current network-state data and current service-state data, including: updating fault records in the first fault list based on the length of the receiving queue indicated by the current network-state data; updating fault records in the second fault list based on the length of the sending queue indicated by the current network-state data; updating fault records in the third fault list based on the number of received requests and the number of processed requests indicated by the current service-state data; and performing fault risk detection of the network file system based on the fault records in the first, second, and third fault lists.

[0045] In this embodiment, a fault list is used to record potential fault risk information for each piece of data. Specifically, a first fault list records queue faults in the receive queue, a second fault list records queue faults in the send queue, and a third fault list records service-level faults in the network file system. The corresponding fault list can be updated when a potential fault risk is detected in relevant data or when the fault risk is eliminated. The fault list can record the time and frequency of detected fault risks for the corresponding data, and can be used to determine the severity of the fault risk, i.e., to assess the risk level of the fault risk.

[0046] It should be noted that the specified failure conditions for service-state failures in the network file system are: the number of received requests is greater than the number of processed requests, and the number of received and processed requests indicated by two consecutive service-state data collections remains unchanged. Here, the number of received and processed requests collected by the service probe refers to the total number of received and processed requests from the start of the network file system to the current collection time. If the number of received requests is greater than the number of processed requests, it can be determined that there is a backlog of received requests awaiting processing. If the number of received and processed requests indicated by two consecutive service-state data collections remains unchanged, it can be determined that the service for receiving and processing requests is stuck, and no new requests are being received or processed. In this case, it can be determined that there is a risk of service-state data failure.

[0047] During each detection cycle, the first, second, and third fault lists are updated. Based on the fault records in these lists, fault risk detection is performed on the network file system. Optionally, network-state and service-state data can be merged for cross-layer correlation analysis. That is, information from the first, second, and third fault lists can be combined for fault risk detection. For example, if the third fault list indicates that the number of received requests has not changed, and the first fault list indicates that the receive queue is congested, then the fault risk can be determined to originate from the receive queue, and corresponding fault handling can be performed on the receive queue subsequently.

[0048] Alternatively, a weighted sliding window association analysis can be used, employing a dynamic weight adjustment mechanism. When network file system resources are scarce, the network state weight in the association analysis can be reduced to prioritize the accuracy of service state data analysis.

[0049] In this embodiment, fault information is recorded through a first fault queue, a second fault queue, and a third fault queue. Fault detection is performed based on the recorded information from these three fault queues. Fault detection can be combined with historical fault risk information to achieve continuous monitoring of faults and improve the accuracy of fault risk detection. By fusing network state and service state data for fault analysis and using a cross-layer correlation model between the network layer and the service layer to analyze faults, the limitations of single-dimensional detection can be overcome, thereby reducing the false positive rate and improving the accuracy of fault detection.

[0050] In an exemplary embodiment, in order to perform accurate fault detection within a detection period, the current network state data includes network state data collected multiple times within the current detection period; the current service state data includes service state data collected multiple times within the current detection period. That is, both the first sampling interval and the second sampling interval need to be less than one detection period, and multiple samplings of network state data and service state data will be performed within one detection period.

[0051] Correspondingly, based on the receiving queue length indicated by the current network state data, the fault records in the first fault list are updated, including: if the receiving queue length indicated by the network state data collected in a single acquisition is greater than or equal to the receiving queue length threshold, a fault record is added to the first fault list; if the receiving queue length indicated by the network state data collected in a single acquisition is less than the receiving queue length threshold, the fault records in the first fault list are cleared.

[0052] The length of the receiving queue indicated by the network state data can indicate the congestion status of the receiving queue. If the length of the receiving queue indicated by the network state data collected in a single collection is greater than or equal to the receiving queue length threshold, it can be determined that there is congestion in the receiving queue at the time of the collection, and a fault record can be added to the first fault list.

[0053] If the length of the receiving queue indicated by the network state data collected in a single acquisition is less than the receiving queue length threshold, it can be determined that there is no congestion in the receiving queue at the time of the acquisition or that the previous congestion has been eliminated. At this time, there is no risk of failure in the receiving queue, and the fault records in the first fault list can be cleared to prevent outdated fault records from affecting fault detection.

[0054] Correspondingly, based on the transmission queue length indicated by the current network state data, the fault records in the second fault list are updated, including: if the transmission queue length indicated by the network state data collected in a single acquisition is greater than or equal to the transmission queue length threshold, a fault record is added to the second fault list; if the transmission queue length indicated by the network state data collected in a single acquisition is less than the transmission queue length threshold, the fault records in the second fault list are cleared.

[0055] The transmission queue length indicated by the network state data can indicate the congestion status of the transmission queue. If the transmission queue length indicated by the network state data collected in a single collection is greater than or equal to the transmission queue length threshold, it can be determined that there is congestion in the transmission queue at the time of collection, and a fault record can be added to the second fault list.

[0056] If the length of the transmission queue indicated by the network state data collected in a single acquisition is less than the transmission queue length threshold, it can be determined that there is no congestion in the transmission queue at the time of acquisition or that the previous congestion has been eliminated. At this time, there is no risk of failure in the transmission queue, and the fault records in the second fault list can be cleared to prevent outdated fault records from affecting fault detection.

[0057] Correspondingly, based on the number of received requests and the number of processed requests indicated by the current service state data, the fault records in the third fault list are updated, including: if the number of received requests and the corresponding number of processed requests indicated by the service state data collected in a single acquisition meet the specified fault conditions, a fault record is added to the third fault list; if the number of received requests and the corresponding number of processed requests indicated by the service state data collected in a single acquisition do not meet the specified fault conditions, the fault records in the third fault list are cleared.

[0058] The changes in the number of received and processed requests indicated by the service status data can indicate whether the service is operating normally. If the number of received requests is greater than the number of processed requests, there is a backlog of pending requests, and the request processing service may be at risk of stalling. If the number of received and processed requests does not change in two consecutive data collections, both the request receiving and request processing services may be at risk of stalling. If the service status data collected in a single data collection meets both of the above conditions (i.e., specified fault conditions), a fault record is added to the third fault list.

[0059] If the number of received requests and the corresponding number of processed requests indicated by the service state data collected in a single collection do not meet the above-mentioned specified fault conditions, it can be determined that the service receiving and processing requests at the time of the collection is performing normally. At this time, there is no risk of fault in the service state data, and the fault records in the third fault list can be cleared.

[0060] By updating the fault records in the fault list through multiple sampling data in this embodiment, the accuracy and timeliness of fault detection can be improved, and outdated fault information can be avoided from affecting the results of fault detection.

[0061] In one exemplary embodiment, the method further includes: using a system probe to obtain parameter values ​​of load parameters of the network file system at a third sampling interval, wherein the parameter values ​​of the load parameters are determined based on at least one of the following: resource utilization of processor resources of the network file system; resource utilization of memory resources of the network file system.

[0062] To monitor the load of the network file system (NFS), a system probe can be used to obtain the load parameter values ​​of the NFS at a third sampling interval. Here, the third sampling interval can be set empirically, for example, 1 second, 2 seconds, or other values; this embodiment does not limit this. The load parameter values ​​are determined based on the resource utilization of the NFS's processor resources, memory resources, or a combination of both, and can reflect the load status of the NFS's processor or memory resources.

[0063] Optionally, the system probe can monitor processor or memory metrics of the network file system using the sysstat toolchain, a set of performance monitoring tools that can be used to collect and analyze system resource usage data. Here, taking a third sampling interval of 1 second as an example, the system probe's startup script can be:

[0064] #! / bin / bash

[0065] while true; do

[0066] timestamp=$(date +%s%3N)

[0067] cpu_usage=$(mpstat 1 1 | awk ' / Average / {print 100 - $12}')

[0068] mem_usage=$(free -m | awk ' / Mem / {printf "%.1f", $3 / $2*100}')

[0069] echo "{\"timestamp\":$timestamp,\"cpu\":$cpu_usage,\"mem\":$mem_usage}" >> / var / log / system_metrics.log

[0070] sleep 1

[0071] done

[0072] Optionally, a data acquisition layer can be responsible for collecting relevant data using network probes, service probes, and system probes, such as... Figure 3 As shown, network probes can collect Transmission Control Protocol / User Datagram Protocol (TDP) traffic from client connections, service probes can collect operations per second (OPS) metrics from network file system service processes, and system probes can obtain processor / memory data from system resources.

[0073] Correspondingly, based on the load parameter values, the receive queue length threshold and the send queue length threshold are adjusted, wherein the receive queue length threshold and the send queue length threshold are negatively correlated with the load parameter values.

[0074] Here, the receive queue length threshold and send queue length threshold can be used to determine the congestion status of the receive and send queues. When the queue length of the receive or send queue is greater than or equal to the corresponding length threshold, it can be determined that the queue is congested and there is a risk of failure. In order to intervene in queue congestion in advance when the network file system load is high and reduce the consequences of failure, the receive queue length threshold and send queue length threshold can be flexibly adjusted based on the load parameter value. That is, the higher the load parameter value, the lower the receive queue length threshold and send queue length threshold, and the easier it is to detect and handle the risk of failure in the receive and send queues; the lower the load parameter value, the higher the receive queue length threshold and send queue length threshold, which can tolerate the congestion of the receive and send queues to a certain extent and allow for short-term queue congestion.

[0075] Optionally, the receive queue length threshold and the send queue length threshold can be set to 10000 by default, and the threshold can be reduced to 70% when the load is greater than 80, and relaxed to 130% when the load is less than 30. This dynamic threshold adjustment algorithm can be:

[0076] #Dynamic Threshold Algorithm

[0077] def calculate_dynamic_threshold():

[0078] current_load = get_cpu_usage()

[0079] base_value = 10000 # Default threshold

[0080] if current_load > 80:

[0081] return base_value * 0.7 # Reduce the threshold under high load

[0082] elif current_load < 30:

[0083] return base_value * 1.3 # Relax the threshold under low load

[0084] else:

[0085] return base_value

[0086] This embodiment demonstrates that by obtaining the load parameters of the network file system, the load status of the network file system can be determined, and the dynamic threshold algorithm can be improved by adaptively and dynamically adjusting the threshold for fault risk detection based on the load status, thereby increasing the flexibility of fault risk detection.

[0087] In an exemplary embodiment, after using a system probe to obtain the parameter values ​​of the load parameters of the network file system according to a third sampling interval, the method further includes: adjusting a second sampling interval based on the parameter values ​​of the load parameters, wherein the second sampling interval is positively correlated with the parameter values ​​of the load parameters.

[0088] In order to dynamically adjust the fault detection standard of the service-state data side based on the dynamic load of the network file system, the second sampling interval can be adjusted based on the parameter value of the load parameter, and the second sampling interval is positively correlated with the parameter value of the load parameter.

[0089] Here, the longer the second sampling interval, that is, the fewer the number of samples in the current detection period, the higher the standard for judging fault risk, thereby reducing the risk of misjudgment under high load, allowing for a short period of service request backlog, and prioritizing service availability; under low load, the second sampling interval is shorter, the judgment of fault risk is more lenient, and related faults can be detected and handled earlier when resources are sufficient.

[0090] Optionally, the second sampling interval can also be adjusted based on historical data and current load conditions. The code for dynamically adjusting the second sampling interval can be: def dynamic_threshold(historical_data, current_load):

[0091] baseline = np.percentile(historical_data, 90)

[0092] return baseline * (1 + current_load / 100) # Higher load results in a longer second sampling interval

[0093] This embodiment uses the dynamic adjustment of the second sampling interval based on the load of the network file system to reduce the risk of misjudgment and waste of resources under high load, and to strictly judge faults under low load, so as to detect and handle faults early and make effective use of resources.

[0094] In an exemplary embodiment, after using the system probe to obtain the parameter values ​​of the network file system load parameters according to the third sampling interval, the method further includes: when the current parameter value of the load parameter is greater than or equal to the first load parameter threshold, and when the number of connected clients recorded in the client monitoring list is greater than a specified number, clearing the records of other connected clients in the client monitoring list except for the previously specified number of connected clients, wherein the client monitoring list is used to record the connected clients of the network file system in descending order of connection heat, and the connection heat is determined by the number of pending requests existing for the corresponding connected client.

[0095] To reduce load pressure under high load and avoid unnecessary waste of detection resources, an LRU (Least Recently Used) caching structure can be used to intelligently manage connections. The core idea of ​​the LRU caching structure is to evict the least recently accessed data, that is, the connection with the lowest connection frequency.

[0096] In this embodiment, the client monitoring list sorts connections in descending order of connection popularity. If the current load parameter value is greater than or equal to a first load parameter threshold, the connections with the lowest popularity can be removed, retaining only a specified number of connections with higher popularity. This specified number can be set empirically; it can be a fixed value, such as 500, 550, or other values, or it can be adjusted according to the specific load conditions. For example, if the current load is high, the specified number can be correspondingly lower to reduce the number of retained connections and decrease load pressure; if the current load is low, the specified number can be correspondingly higher to increase the number of retained connections.

[0097] Alternatively, the connection heat can be calculated as shown in formula (1).

[0098] Heat = a * amount of data accumulated in the receive queue / dynamic threshold + b * duration of data accumulation in the receive queue; (1)

[0099] Where a and b are weights, the values ​​of a and b can be set empirically, for example, a can be 0.6, 0.7 or 0.8, and b can be 0.2, 0.3 or 0.4. This embodiment does not limit this. The above formula (1) combines the amount of accumulated data in the receiving queue with the duration of the accumulation, which can accurately identify long-term abnormal connections.

[0100] In this embodiment, a client monitoring list with heat weights is used, and its capacity can be automatically adjusted according to the load. By dynamically adjusting the monitoring connections based on the load, the load pressure can be reduced under high load conditions.

[0101] In one exemplary embodiment, the method further includes: periodically detecting the number of pending requests from connected clients of the network file system; if, in the detected client monitoring list, there is a connected client with zero pending requests, clearing the record of the connected client with zero pending requests from the client monitoring list; if, in the detected case, there is a connected client not recorded in the client monitoring list with zero pending requests, adding a corresponding record to the client monitoring list for the connected client not recorded in the client monitoring list with zero pending requests.

[0102] To reduce resource consumption by low-activity connected clients, the number of pending requests from connected clients in the network file system can be periodically checked, and connected clients can be intelligently added or removed based on the detection data, while connected clients with zero pending requests can be cleared, thereby improving monitoring efficiency.

[0103] In this embodiment, if there are connected clients with zero pending requests recorded in the monitored client list, these clients can be directly cleared to reduce resource consumption. Furthermore, some connected clients may not be recorded in the client monitoring list. If a connected client not recorded in the client monitoring list is detected and has zero pending requests, this client can be added to the client monitoring list, and a corresponding record can be added for it. This facilitates the subsequent clearing of clients with zero pending requests during subsequent monitoring.

[0104] This embodiment demonstrates how adding or deleting monitoring items based on detection data can eliminate low-activity clients with zero pending requests, thereby improving the efficiency and accuracy of monitoring.

[0105] In an exemplary embodiment, after using the system probe to obtain the parameter value of the load parameter of the network file system according to the third sampling interval, the above method further includes: when the parameter value of the load parameter is greater than or equal to the second load parameter threshold, and when the target risk level is not the highest risk level among multiple risk levels, updating the target risk level to a risk level one level higher than the target risk level among multiple risk levels.

[0106] In order to thoroughly handle failure risks and quickly restore services under high load conditions, in this embodiment, the failure risk handling strategy can be upgraded under high load conditions. That is, when the load parameter value is greater than or equal to the second load parameter threshold, if the target risk level is not the highest risk level among multiple risk levels, the target risk level is updated to a risk level one level higher than the target risk level among multiple risk levels.

[0107] Optionally, appropriate fault handling strategies can be adopted based on the processor load or memory load. For example, when the processor utilization is too high, the target risk level of the fault risk can be increased; when the memory utilization is too high, the pstack command can be disabled to prioritize memory release.

[0108] This embodiment demonstrates that by increasing the target risk level when the load parameter value is high, the probability of completely resolving the fault risk in one go can be increased, the overall resource consumption caused by handling the fault risk can be reduced, and the recovery of services affected by the fault risk can be accelerated.

[0109] In one exemplary embodiment, the multiple risk levels include a first risk level and a second risk level with increasing severity. Here, the failure risk corresponding to the first risk level is a failure risk with low risk or minor impact, and the failure risk corresponding to the second risk level is a failure risk with high risk or significant impact.

[0110] Correspondingly, based on the fault records in the first fault list, the second fault list, and the third fault list, fault risk detection is performed on the network file system, including: determining that the network file system has a fault risk and the risk level of the fault risk of the network file system is the first risk level if at least one of the following conditions is met: the number of fault records in the first fault list is greater than or equal to the first threshold and less than the second threshold; the number of fault records in the second fault list is greater than or equal to the third threshold and less than the fourth threshold; the number of fault records in the third fault list is greater than or equal to the fifth threshold and less than the sixth threshold.

[0111] Here, the number of fault records in the first, second, or third fault list refers to the number of consecutive data anomalies. If the number of consecutive anomalies is low, i.e., the number of fault records in the first fault list is less than the first threshold, the number of fault records in the second fault list is less than the third threshold, and the number of fault records in the third fault list is less than the fifth threshold, the probability of a fault risk is low and it is not considered a fault risk. If the number of consecutive anomalies is high but does not meet the conditions for the second risk level, it is considered that the network file system has a fault risk of the first risk level.

[0112] Correspondingly, a network file system is determined to have a failure risk if at least one of the following conditions is met, and the risk level of the failure risk of the network file system is the second risk level: the number of failure records in the first failure list is greater than or equal to the second threshold; the number of failure records in the second failure list is greater than or equal to the fourth threshold; or the number of failure records in the third failure list is greater than or equal to the sixth threshold.

[0113] Here, if there are many fault records in the first fault list, the second fault list, or the third fault list, that is, if multiple consecutive anomalies are detected, it can be determined that the severity of the fault risk is high, and its risk level is also the second risk level, which is higher than the first risk level.

[0114] Optionally, the aforementioned first, second, third, fourth, fifth, and sixth count thresholds can be adjusted according to load parameters. For example, when the load parameters are high, the count thresholds can be increased accordingly to reduce the number of fault risk detections and the probability of false fault risk reports, thereby reducing resource consumption caused by fault handling. When the load parameters are low, the count thresholds can be decreased accordingly to increase the number of fault risk detections, enabling earlier detection and handling of fault risks, and making full use of surplus system resources.

[0115] In this embodiment, the risk level of a fault is determined by detecting the number of fault records in the fault list. The risk level of a fault can be determined based on the number of consecutive faults detected, thereby enabling different treatments to be performed on fault risks of different severity and improving the intelligence of automatic fault risk handling.

[0116] In one exemplary embodiment, the multiple risk levels include a first risk level and a second risk level with increasing severity; performing processing operations corresponding to the target risk level on the network file system includes: when the target risk level is the first risk level, recording the fault risk information of the network file system to a log file; when the target risk level is the second risk level, capturing stack information after detecting the fault risk of the network file system at least twice, and performing differential analysis on the captured at least two first stack information.

[0117] For the first risk level, which has a low severity, only the risk of failure needs to be recorded. The risk information of the network file system failure can be logged to a log file, and no direct action should be taken to prevent unnecessary impact on the network file system. For the second risk level, which has a higher severity, the failure may have a significant impact on the network file system. Therefore, it is necessary to capture and analyze the relevant data for this risk in order to carry out further processing.

[0118] At the second risk level, stack traces need to be captured at least twice after the failure occurred. The first capture could be at the moment the failure risk was detected, and the second capture could be some time after the failure risk was detected, to ensure that the impact of the failure risk on the stack environment is captured. After capturing the stack traces at least twice, differential analysis can be performed on the captured stack traces to identify changes in the network file system during the failure and the actual impact of the failure risk. This helps to further determine the appropriate measures to be taken to address the failure risk and accelerates the location of the root cause of the failure.

[0119] This embodiment demonstrates how different handling measures can be taken for different fault risk levels. By recording or analyzing fault risks, the efficiency of fault handling can be improved, service interruption events during fault handling can be reduced, and the impact of faults on business can be minimized.

[0120] In one exemplary embodiment, the multiple risk levels further include a third risk level and a fourth risk level, which are of progressively increasing severity and are all higher than the second risk level; performing the processing operation corresponding to the target risk level on the network file system further includes: if it is determined, based on the differential analysis results of at least two first stack traces, that some connected clients of the network file system are abnormal, performing the following processing operation corresponding to the third risk level on the network file system: first terminating the network connection between the network file system and the abnormal client, and then triggering the restoration of the network connection between the network file system and the abnormal client, wherein the abnormal client is a connected client among the connected clients of the network file system that is abnormal; if it is determined, based on the differential analysis results of at least two first stack traces, that all connected clients of the network file system are abnormal, performing the following processing operation corresponding to the fourth risk level on the network file system: capturing stack traces at least twice more, saving at least two captured second stack traces, and restarting the network file system service.

[0121] After conducting a differential analysis of the second risk level, the risk level corresponding to the fault risk can be further determined, and corresponding operations can be performed. If the target risk level of the fault risk is determined to be the third risk level, a reconnection can be established with the abnormal client to handle the fault risk without affecting other business services. If the target risk level of the fault risk is determined to be the fourth risk level, the entire network file system service needs to be restarted, and at least two stack traces should be captured before the restart for evidence recording.

[0122] In this embodiment, at the third risk level, the impact of this fault risk is limited to a subset of clients. The network connection with the abnormal client can be terminated first, then a recovery mechanism can be triggered to re-establish the connection. This process only affects the connection of the client identified as abnormal and will not interfere with other normally functioning clients. The network file system can maintain continuity for the majority of normal clients.

[0123] At the fourth risk level, all or most connected clients are deemed abnormal, requiring a more aggressive recovery strategy. This includes at least two re-captures of the second stack trace to ensure the stack details at the moment of the anomaly are stored, which can be used for subsequent root cause analysis. Next, to resolve the global issue, the network file system service needs to be restarted to restore the service to an initial, relatively stable state. During the restart, all client connections will be safely disconnected and re-established to minimize service recovery time and impact on business operations.

[0124] Optionally, the code for handling fault risk under the fourth risk level can be:

[0125] # Recovery logic for the fourth risk level (service restart + stack trace evidence collection)

[0126] if [ $RISK_LEVEL -eq 4 ]; then

[0127] pid=$(pidof nfs.nfsd)

[0128] pstack $pid > stack_$(date +%s).log

[0129] sleep 5

[0130] pstack $pid > stack_$(date +%s).log

[0131] systemctl restart nfs

[0132] fi

[0133] This embodiment introduces third and fourth risk level handling strategies, which can control the impact of fault risks within the abnormal client when the impact range is small, and save the stack details at the time of the anomaly and restart the service when the impact range is large. By using differentiated handling methods, fault risks can be handled flexibly, improving the fault handling capability and recovery speed of the network file system.

[0134] The fault handling method in this application embodiment will be explained below with reference to optional examples. In this optional example, the fault handling method is executed by a fault handling system.

[0135] Figure 4 This is an architecture diagram of the fault handling system in this optional example, such as Figure 4 As shown, the fault handling system includes a data acquisition layer, an analysis and decision-making layer, and an execution control layer.

[0136] Client connection: User clients can mount distributed file storage through the network file system protocol and run business under the mount point, generating input / output (I / O) business interactions with the storage.

[0137] Data Acquisition Layer: Introduces a multi-source heterogeneous data fusion engine, and uses network probes, service probes, and system probes to collect multi-dimensional information, including real-time network state data of receiving queues and sending queues, service state data of operations per second, and monitoring and collection of system resource utilization data of processors / memory.

[0138] Analysis and decision-making layer: Introducing a multimodal perception engine (network state, service state, system resources) to establish a multidimensional detection model; a dynamic baseline library stores 7 days of historical data and supports sliding window threshold calculation; the decision tree divides risk levels from level 1 to level 4 according to different risk levels, and selects different handling methods to perform graded risk judgment according to risk level.

[0139] Optionally, the aforementioned dynamic baseline library can store historical data and generate adaptive thresholds. These adaptive thresholds can be used to assess fault risks and achieve adaptive business fluctuation adjustment thresholds. It can store 7 days of historical data and update it hourly, using a tiered storage approach, divided into hot data, warm data, and cold data. Hot data is data from 0 to 1 hour, warm data is data from 1 to 24 hours, and cold data is data from 24 hours to 7 days.

[0140] The decision tree module described above can trigger tiered handling actions based on multi-dimensional rules. These multi-dimensional rules can include time, network, and application dimensions. The time dimension is related to the current adaptive threshold, the network dimension is related to the current queue length growth rate of the receiving queue, and the application dimension is related to the current business request processing efficiency.

[0141] Optionally, the fault handling system can simultaneously analyze the growth rate of the network layer's accepted request queue length and the application layer's request processing efficiency, including calculating the growth rate of the TCP connection state matrix at the network dimension, and calculating request processing efficiency in real time at the application dimension and predicting request processing efficiency trends.

[0142] Execution Control Layer: Implements a tiered handling strategy, taking corresponding actions based on the handling level, including logging, connection termination, stack trace analysis, and service restart. Specifically: Level 1 Warning: Logs are logged; Level 2 Evidence Analysis: Stack traces are captured using the pstack command and analyzed differentially; Level 3 Handling: Abnormal client connections are terminated, triggering recovery, affecting only the connections of clients with abnormalities, minimizing the impact and ensuring service continuity of nodes; Level 4 Recovery: The network file system service is restarted. When the entire service is abnormal, Level 3 cannot guarantee its serviceability, requiring a service restart for recovery.

[0143] This optional example utilizes a fault risk detection method combining network and service states, significantly improving the fault detection rate while drastically reducing the false alarm rate. This greatly reduces the workload of maintenance personnel and lowers labor costs. Furthermore, the fault handling system optimizes processor utilization for service monitoring, conserving processor resources and improving the stability and performance of the network file system. In terms of fault recovery, a four-level closed-loop control system from early warning to restart is adopted, reducing manual intervention and improving recovery speed. Through the data acquisition layer, analysis and decision-making layer, and execution control layer, closed-loop self-healing is achieved, constructing a three-level response strategy of "early warning → evidence collection → handling." The three-layer decoupled architecture of data acquisition → analysis and decision-making → execution control allows for modular expansion, improving both the stability of the network file system and reducing maintenance costs.

[0144] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to this application.

[0145] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method.

[0146] According to another aspect of the embodiments of this application, a fault handling apparatus is also provided, which can be used to implement the fault handling method provided in the above embodiments, and will not be repeated hereafter. As used below, the term "module" can be a combination of software and / or hardware that implements a predetermined function. Although the apparatus described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0147] Figure 5 This is a structural block diagram of an optional fault handling device according to an embodiment of this application, such as... Figure 5 As shown, the fault handling device includes an acquisition unit 502, a first detection unit 504, and an execution unit 506.

[0148] The acquisition unit 502 is used to acquire network state data and service state data of the network file system in the current detection period, and obtain the current network state data and the current service state data;

[0149] The first detection unit 504 is used to perform fault risk detection on the network file system by fusing current network state data and current service state data;

[0150] The execution unit 506 is used to perform a processing operation corresponding to the target risk level on the network file system when a fault risk is detected in the network file system and the risk level of the fault risk in the network file system is a target risk level among multiple risk levels. Here, one of the multiple risk levels corresponds to one processing operation.

[0151] It should be noted that the acquisition unit 502 in this embodiment can be used to execute the above step S202, the first detection unit 504 in this embodiment can be used to execute the above step S204, and the execution unit 506 in this embodiment can be used to execute the above step S206.

[0152] The embodiments provided in this application obtain network state data and service state data of the network file system within the current detection period to obtain current network state data and current service state data. By fusing the current network state data and current service state data, fault risk detection is performed on the network file system. When a fault risk is detected in the network file system, and the risk level of the fault risk is a target risk level among multiple risk levels, the processing operation corresponding to the target risk level is performed on the network file system. Here, one of the multiple risk levels corresponds to one processing operation. This solves the technical problems of low fault detection accuracy and insufficient fault processing intelligence in fault processing methods in related technologies, and improves the accuracy of fault detection and the intelligence of fault processing.

[0153] In one exemplary embodiment, the apparatus further includes: a first acquisition unit, configured to acquire network state data of the network file system at a first sampling interval using a network probe, wherein the network state data of the network file system is used to indicate the queue length of the receiving queue and the queue length of the sending queue of the network file system; and a second acquisition unit, configured to acquire service state data of the network file system at a second sampling interval using a service probe, wherein the service state data of the network file system is used to indicate the number of received requests and the number of processed requests, wherein the number of received requests is the number of processed requests received by the network file system and the number of processed requests is the number of processed requests processed by the network file system.

[0154] In an exemplary embodiment, in a network file system, queue faults of the receiving queue are recorded in a first fault list, queue faults of the sending queue are recorded in a second fault list, and service-state faults of the network file system are recorded in a third fault list. A service-state fault of the network file system refers to a situation where the number of received requests and the number of processed requests meet specified fault conditions. The specified fault conditions are: the number of received requests is greater than the number of processed requests, and the number of received requests and the number of processed requests indicated by two consecutive sets of service-state data have not changed. The first detection unit includes: a first update module, used to update the fault records in the first fault list based on the length of the receiving queue indicated by the current network-state data; a second update module, used to update the fault records in the second fault list based on the length of the sending queue indicated by the current network-state data; a third update module, used to update the fault records in the third fault list based on the number of received requests and the number of processed requests indicated by the current service-state data; and a detection module, used to perform fault risk detection on the network file system based on the fault records in the first fault list, the second fault list, and the third fault list.

[0155] In an exemplary embodiment, the current network state data includes network state data collected through multiple acquisitions within the current detection period; the current service state data includes service state data collected through multiple acquisitions within the current detection period; the first update module includes: a first add submodule, configured to add a fault record to a first fault list when the length of the receive queue indicated by the network state data collected in a single acquisition is greater than or equal to a receive queue length threshold; a first clear submodule, configured to clear the fault record in the first fault list when the length of the receive queue indicated by the network state data collected in a single acquisition is less than a receive queue length threshold; the second update module includes: a second add submodule, configured to add a fault record to a ... If the column length is greater than or equal to the sending queue length threshold, add a fault record to the second fault list; the second clearing submodule is used to clear the fault record in the second fault list if the sending queue length indicated by the network state data collected in a single acquisition is less than the sending queue length threshold in the current network state data; the third updating module includes: a third adding submodule, used to add a fault record to the third fault list if the number of received requests and the corresponding number of processed requests indicated by the service state data collected in a single acquisition in the current service state data meet the specified fault conditions; and a third clearing submodule, used to clear the fault record in the third fault list if the number of received requests and the corresponding number of processed requests indicated by the service state data collected in a single acquisition in the current service state data do not meet the specified fault conditions.

[0156] In one exemplary embodiment, the apparatus further includes: a third acquisition unit, configured to acquire, using a system probe, parameter values ​​of load parameters of the network file system at a third sampling interval, wherein the parameter values ​​of the load parameters are determined based on at least one of: resource utilization of processor resources of the network file system; resource utilization of memory resources of the network file system; and a first adjustment unit, configured to adjust a receive queue length threshold and a send queue length threshold based on the parameter values ​​of the load parameters, wherein the receive queue length threshold and the send queue length threshold are negatively correlated with the parameter values ​​of the load parameters.

[0157] In one exemplary embodiment, the above apparatus further includes: a second adjustment unit, configured to adjust a second sampling interval based on the load parameter value after obtaining the load parameter value of the network file system according to a third sampling interval using a system probe, wherein the second sampling interval is positively correlated with the load parameter value.

[0158] In an exemplary embodiment, the above apparatus further includes: a first clearing unit, configured to, after obtaining the parameter value of the load parameter of the network file system according to a third sampling interval using a system probe, and when the current parameter value of the load parameter is greater than or equal to a first load parameter threshold, clear the records of other connected clients in the client monitoring list except for the previously specified number of connected clients, when the number of connected clients recorded in the client monitoring list is greater than a specified number, wherein the client monitoring list is used to record the connected clients of the network file system in descending order of connection heat, and the connection heat is determined by the number of pending requests existing for the corresponding connected client.

[0159] In one exemplary embodiment, the apparatus further includes: a second detection unit, configured to periodically detect the number of pending requests from connected clients of the network file system; a second clearing unit, configured to clear the record of the connected client with zero pending requests from the client monitoring list if there is a connected client with zero pending requests recorded in the detected client monitoring list; and an adding unit, configured to add a corresponding record to the client monitoring list if there is a connected client with zero pending requests not recorded in the client monitoring list.

[0160] In one exemplary embodiment, the above apparatus further includes: an update unit, configured to, after obtaining the parameter value of the load parameter of the network file system according to the third sampling interval using the system probe, update the target risk level to a risk level one level higher than the target risk level when the target risk level is not the highest risk level among multiple risk levels if the parameter value of the load parameter is greater than or equal to the second load parameter threshold.

[0161] In one exemplary embodiment, multiple risk levels include a first risk level and a second risk level with progressively increasing severity; the detection module includes: a first determining submodule, configured to determine that the network file system has a fault risk, and the risk level of the fault risk of the network file system is a first risk level, provided that at least one of the following conditions is met: the number of fault records in a first fault list is greater than or equal to a first threshold and less than a second threshold; the number of fault records in a second fault list is greater than or equal to a third threshold and less than a fourth threshold; the number of fault records in a third fault list is greater than or equal to a fifth threshold and less than a sixth threshold; a second determining submodule, configured to determine that the network file system has a fault risk, and the risk level of the fault risk of the network file system is a second risk level, provided that at least one of the following conditions is met: the number of fault records in the first fault list is greater than or equal to a second threshold; the number of fault records in the second fault list is greater than or equal to a fourth threshold; the number of fault records in the third fault list is greater than or equal to a sixth threshold.

[0162] In one exemplary embodiment, the multiple risk levels include a first risk level and a second risk level with increasing severity; the execution unit includes: a recording module, configured to record fault risk information of the network file system to a log file when the target risk level is the first risk level; and a first execution module, configured to capture stack information at least twice after detecting the fault risk of the network file system, and perform differential analysis on the captured at least two stack information when the target risk level is the second risk level.

[0163] In one exemplary embodiment, the multiple risk levels further include a third risk level and a fourth risk level, which are of progressively increasing severity and are all higher than the second risk level; the execution unit further includes: a second execution module, configured to perform the following processing operation corresponding to the third risk level on the network file system when it is determined, based on the differential analysis results of at least two first stack information, that some connected clients of the network file system are abnormal: first terminating the network connection between the network file system and the abnormal client, and then triggering the restoration of the network connection between the network file system and the abnormal client, wherein the abnormal client is a connected client among the connected clients of the network file system that is abnormal; and a third execution module, configured to perform the following processing operation corresponding to the fourth risk level on the network file system when it is determined, based on the differential analysis results of at least two first stack information, that all connected clients of the network file system are abnormal: capture stack information at least twice more, save at least two captured second stack information, and restart the network file system service.

[0164] Embodiments of this application also provide an electronic device, including a memory and a processor, wherein the memory stores a computer program and the processor is configured to run the computer program to perform the steps in any of the above-described fault handling method embodiments.

[0165] Embodiments of this application also provide a computer-readable storage medium storing a computer program, wherein the computer program is configured to execute the steps in any of the above-described fault handling method embodiments when it is run.

[0166] In one exemplary embodiment, the aforementioned computer-readable storage medium may include, but is not limited to, various media capable of storing computer programs, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard disk, magnetic disk, or optical disk.

[0167] Embodiments of this application also provide a computer program product, which includes a computer program that, when executed by a processor, implements the steps in any of the above-described fault handling method embodiments.

[0168] Embodiments of this application also provide another computer program product, including a non-volatile computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps in any of the above-described fault handling method embodiments.

[0169] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0170] The foregoing has provided a detailed description of a fault handling method, apparatus, electronic device, and storage medium provided in this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the embodiments above are merely for the purpose of helping to understand the method and its core ideas. It should be noted that those skilled in the art can make various improvements and modifications to this application without departing from its principles, and these improvements and modifications also fall within the protection scope of the claims of this application.

Claims

1. A fault handling method, characterized in that, include: Obtain network state data and service state data of the network file system within the current detection period to obtain current network state data and current service state data. The network state data of the network file system is used to indicate the queue length of the receiving queue and the queue length of the sending queue of the network file system. The service state data of the network file system is used to indicate the number of received requests and the number of processed requests. The number of received requests is the number of processed requests received by the network file system, and the number of processed requests is the number of processed requests processed by the network file system. By fusing the current network state data and the current service state data, fault risk detection is performed on the network file system; If a fault risk is detected in the network file system and the risk level of the fault risk in the network file system is a target risk level among multiple risk levels, the processing operation corresponding to the target risk level is performed on the network file system, wherein one of the multiple risk levels corresponds to one processing operation. In the network file system, queue faults of the receiving queue are recorded in a first fault list, queue faults of the sending queue are recorded in a second fault list, and service-state faults of the network file system are recorded in a third fault list. A service-state fault of the network file system refers to a situation where the number of received requests and the number of processed requests meet a specified fault condition. The specified fault condition is that the number of received requests is greater than the number of processed requests, and the number of received requests and the number of processed requests indicated by the service-state data collected in two consecutive periods have not changed. The step of performing fault risk detection on the network file system by fusing the current network state data and the current service state data includes: Update the fault records in the first fault list based on the receive queue length indicated by the current network state data; Update the fault records in the second fault list based on the sending queue length indicated by the current network state data; Based on the number of received requests and the number of processed requests indicated by the current service state data, update the fault records in the third fault list; Based on the fault records in the first fault list, the second fault list, and the third fault list, fault risk detection is performed on the network file system.

2. The method according to claim 1, characterized in that, The method further includes: Using a network probe, collect network-state data of the network file system at a first sampling interval; Using a service probe, service-state data of the network file system is collected at the second sampling interval.

3. The method according to claim 2, characterized in that, The current network state data includes network state data collected multiple times within the current detection period; the current service state data includes service state data collected multiple times within the current detection period. The step of updating the fault record in the first fault list based on the receive queue length indicated by the current network state data includes: adding a fault record to the first fault list if the receive queue length indicated by the network state data collected in a single acquisition is greater than or equal to a receive queue length threshold; and clearing the fault record in the first fault list if the receive queue length indicated by the network state data collected in a single acquisition is less than the receive queue length threshold. The step of updating the fault record in the second fault list based on the sending queue length indicated by the current network state data includes: adding a fault record to the second fault list if the sending queue length indicated by the network state data collected in a single acquisition is greater than or equal to a sending queue length threshold; and clearing the fault record in the second fault list if the sending queue length indicated by the network state data collected in a single acquisition is less than the sending queue length threshold. The step of updating the fault records in the third fault list based on the number of received requests and the number of processed requests indicated by the current service state data includes: adding a fault record to the third fault list if the number of received requests and the corresponding number of processed requests indicated by the service state data collected in a single acquisition meet the specified fault condition; and clearing the fault records in the third fault list if the number of received requests and the corresponding number of processed requests indicated by the service state data collected in a single acquisition do not meet the specified fault condition.

4. The method according to claim 3, characterized in that, The method further includes: Using a system probe, the parameter values ​​of the load parameters of the network file system are obtained at a third sampling interval, wherein the parameter values ​​of the load parameters are determined based on at least one of the following: the resource utilization rate of the processor resources of the network file system; the resource utilization rate of the memory resources of the network file system; Based on the parameter values ​​of the load parameters, the receive queue length threshold and the send queue length threshold are adjusted, wherein the receive queue length threshold and the send queue length threshold are negatively correlated with the parameter values ​​of the load parameters.

5. The method according to claim 4, characterized in that, After obtaining the parameter values ​​of the network file system load parameters according to the third sampling interval using the system probe, the method further includes: Based on the parameter value of the load parameter, the second sampling interval is adjusted, wherein the second sampling interval is positively correlated with the parameter value of the load parameter.

6. The method according to claim 4, characterized in that, After obtaining the parameter values ​​of the network file system load parameters according to the third sampling interval using the system probe, the method further includes: If the current value of the load parameter is greater than or equal to the first load parameter threshold, and the number of connected clients recorded in the client monitoring list is greater than a specified number, then the records of other connected clients in the client monitoring list, except for the specified number of connected clients mentioned above, are cleared. The client monitoring list is used to record the connected clients of the network file system in descending order of connection popularity, and the connection popularity is determined by the number of pending requests of the corresponding connected client.

7. The method according to claim 6, characterized in that, The method further includes: The number of pending requests from connected clients of the network file system is periodically detected. If, in the client monitoring list, there is a connected client with zero pending requests, then the record of the connected client with zero pending requests in the client monitoring list is cleared. If a connected client with zero pending requests is detected but not recorded in the client monitoring list, a corresponding record is added to the client monitoring list for that connected client with zero pending requests.

8. The method according to claim 4, characterized in that, After obtaining the parameter values ​​of the network file system load parameters according to the third sampling interval using the system probe, the method further includes: If the value of the load parameter is greater than or equal to the second load parameter threshold, and the target risk level is not the highest risk level among the plurality of risk levels, the target risk level is updated to a risk level one level higher than the target risk level among the plurality of risk levels.

9. The method according to claim 1, characterized in that, The multiple risk levels include a first risk level and a second risk level, with the severity increasing sequentially. The fault risk detection of the network file system based on fault records in the first fault list, the second fault list, and the third fault list includes: The network file system is determined to have a failure risk if at least one of the following conditions is met, and the risk level of the failure risk of the network file system is the first risk level: the number of failure records in the first failure list is greater than or equal to the first threshold and less than the second threshold; the number of failure records in the second failure list is greater than or equal to the third threshold and less than the fourth threshold; the number of failure records in the third failure list is greater than or equal to the fifth threshold and less than the sixth threshold. The network file system is determined to have a failure risk if at least one of the following conditions is met, and the risk level of the failure risk of the network file system is the second risk level: the number of failure records in the first failure list is greater than or equal to the second threshold; the number of failure records in the second failure list is greater than or equal to the fourth threshold; the number of failure records in the third failure list is greater than or equal to the sixth threshold.

10. The method according to any one of claims 1 to 8, characterized in that, The multiple risk levels include a first risk level and a second risk level, with the severity increasing sequentially. The step of performing the processing operation corresponding to the target risk level on the network file system includes: When the target risk level is the first risk level, the fault risk information of the network file system is recorded in the log file. When the target risk level is the second risk level, at least two stack information records are captured after the network file system is detected to have a fault risk, and differential analysis is performed on the captured at least two stack information records.

11. The method according to claim 10, characterized in that, The multiple risk levels also include a third risk level and a fourth risk level, which are of progressively increasing severity and are all higher than the second risk level; The step of performing the processing operation corresponding to the target risk level on the network file system further includes: If, based on the differential analysis results of the at least two first stack information, it is determined that some of the connected clients of the network file system are abnormal, the following processing operation corresponding to the third risk level is performed on the network file system: first, the network connection between the network file system and the abnormal client is terminated, and then the network connection between the network file system and the abnormal client is restored, wherein the abnormal client is a connected client among the connected clients of the network file system that is abnormal. If, based on the differential analysis results of the at least two first stack traces, it is determined that all connected clients of the network file system are abnormal, the following processing operation corresponding to the fourth risk level is performed on the network file system: capture the stack traces at least twice more, save the captured at least two second stack traces, and restart the network file system service.

12. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the steps of the fault handling method as described in any one of claims 1 to 11.

13. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor, configured to implement the steps of the fault handling method as described in any one of claims 1 to 11 when executing the computer program.

14. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, wherein the computer program, when executed by a processor, implements the steps of the fault handling method as described in any one of claims 1 to 11.

Images