Fault processing method, program product, electronic device and storage medium

Abstract

The embodiment of the invention provides a fault processing method, a program product, electronic equipment and a storage medium, which are applied to the technical field of computers, and the method comprises the following steps: obtaining network state data and service state data of a network file system in a current detection period, and obtaining current network state data and current service state data; performing fault risk detection on the network file system by fusing the current network state data and the current service state data; when it is detected that the network file system has the fault risk and the risk level of the fault risk of the network file system is the target risk level in the multiple risk levels, processing operation corresponding to the target risk level is executed on the network file system, and one risk level corresponds to one processing operation. Through the fault processing method and device, the problems of low fault detection accuracy and insufficient intelligent fault processing of a fault processing method in the related technology are solved, and the effects of improving the fault detection accuracy and improving the intelligent degree of fault processing are achieved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a fault handling method, a program product, an electronic device, and a storage medium. Background Art

[0002] With the growth of data scale, the increasing requirements of users for the service reliability of network file systems, and the increasing complexity of anomaly detection in network file systems, the reliability maintenance of network file systems faces many challenges. In related technologies, static thresholds and single-dimensional detection are usually used, with a large proportion of false alarm rates and missed detection rates. Relevant operation and maintenance personnel need to spend a lot of time on handling invalid alarms, and the probability of discovering obscure faults and the real-time detection are low. In addition, after a fault is detected, the fault handling methods in related technologies also have significant limitations. Most automatic processing is a simple restart of the system, which will have a significant impact on the business in the network file system. Many fault scenarios require human intervention.

[0003] It can be seen that the fault handling methods in related technologies have problems such as low fault detection accuracy and insufficient intelligence in fault handling. Summary of the Invention

[0004] The present application provides a fault handling method, a program product, an electronic device, and a storage medium to at least solve the problems of low fault detection accuracy and insufficient intelligence of fault handling in the fault handling methods in the related art.

[0005] The present application provides a fault handling method, comprising: acquiring network state data and service state data of a network file system within a current detection cycle to obtain current network state data and current service state data; performing fault risk detection on the network file system by fusing the current network state data and the current service state data; and, when it is detected that a fault risk exists in the network file system and the risk level of the fault risk exists in the network file system is a target risk level among multiple risk levels, performing a processing operation corresponding to the target risk level on the network file system, wherein each risk level among the multiple risk levels corresponds to one processing operation.

[0006] The present application also provides a fault handling device, comprising: an acquisition unit, configured to acquire network state data and service state data of a network file system within a current detection cycle to obtain current network state data and current service state data; a first detection unit, configured to perform fault risk detection on the network file system by fusing the current network state data and the current service state data; and an execution unit, configured to, upon detecting that a fault risk exists in the network file system and that the risk level of the fault risk exists in the network file system is a target risk level among multiple risk levels, execute a processing operation corresponding to the target risk level on the network file system, wherein one risk level among the multiple risk levels corresponds to one processing operation.

[0007] The present application also provides an electronic device, comprising: a memory for storing a computer program; and a processor for implementing the steps of any of the above-mentioned fault handling methods when executing the computer program.

[0008] The present application also provides a computer-readable storage medium, in which a computer program is stored. When the computer program is executed by a processor, the steps of any of the above-mentioned fault handling methods are implemented.

[0009] The present application also provides a computer program product, including a computer program, which implements the steps of any of the above-mentioned fault handling methods when executed by a processor.

[0010] Through this application, the accuracy of fault risk detection can be improved by integrating network state data and service state data for multi-modal fault risk detection; in addition, the corresponding fault handling operations are automatically executed based on the fault risk level obtained from the fault risk detection, and faults can be handled automatically and intelligently. Therefore, the technical problems of low fault detection accuracy and insufficient intelligence of fault handling in the fault handling methods in related technologies can be solved, and the technical effect of improving the accuracy of fault detection and improving the intelligence of fault handling can be achieved. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] In order to more clearly illustrate the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0012] Figure 1 This is a schematic diagram of an application scenario of a fault handling method according to an embodiment of the present application.

[0013] Figure 2 It is a flowchart of an optional fault handling method according to an embodiment of the present application.

[0014] Figure 3 This is a schematic diagram of an optional fault handling method according to an embodiment of the present application.

[0015] Figure 4 This is an architectural diagram of an optional fault handling system according to an embodiment of the present application.

[0016] Figure 5 This is a structural block diagram of an optional fault handling device according to an embodiment of the present application. DETAILED DESCRIPTION

[0017] The following will be combined with the accompanying drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0018] It should be noted that, in the description of this application, the terms "comprises," "includes," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. The terms "first," "second," etc., in this application are used to distinguish similar objects, and are not used to describe a particular order or sequence.

[0019] In order to enable those skilled in the art to better understand the present application, the present application is further described in detail below with reference to the accompanying drawings and specific implementation methods.

[0020] According to one aspect of the embodiment of the present application, a fault handling method is provided. Optionally, in this embodiment, the fault handling method can be applied to, but is not limited to, Figure 1 The hardware environment shown includes a terminal device 102 and a server 104. The server 104 can be connected to the terminal device 102 via a network and can be used to provide services (e.g., application services, etc.) for the terminal device 102 or a client installed on the terminal device 102. A database can be set on the server 104 or independently of the server 104 to provide data storage services for the server 104.

[0021] The aforementioned network may include, but is not limited to, at least one of the following: a wired network and a wireless network. The aforementioned wired network may include, but is not limited to, at least one of the following: a wide area network, a metropolitan area network, or a local area network. The aforementioned wireless network may include, but is not limited to, at least one of the following: wireless fidelity (Wi-Fi) and Bluetooth. The terminal device 102 may be, but is not limited to, a personal computer (PC), a mobile phone, a tablet computer, etc. The server 104 may be, but is not limited to, a cloud server, a server cluster, or other server types.

[0022] The fault handling method of the embodiment of the present application can be executed by the server 104, or by the terminal device 102, or jointly by the server 104 and the terminal device 102. The fault handling method of the embodiment of the present application can also be executed by the client installed on the terminal device 102.

[0023] Taking the server 104 as an example to execute the fault handling method in this embodiment, Figure 2 FIG. 1 is a flow chart of an optional fault handling method according to an embodiment of the present application, such as Figure 2 As shown, the process of the method may include steps S202 to S206.

[0024] Step S202: Acquire the network state data and service state data of the network file system in the current detection period to obtain the current network state data and the current service state data.

[0025] Step S204 : performing fault risk detection on the network file system by integrating the current network state data and the current service state data.

[0026] Step S206: When it is detected that the network file system has a failure risk, and the risk level of the failure risk of the network file system is a target risk level among multiple risk levels, a processing operation corresponding to the target risk level is performed on the network file system, wherein one risk level among the multiple risk levels corresponds to one processing operation.

[0027] The fault handling method of this embodiment can be applied to the field of computer technology, and can be applied to scenarios of fault risk detection and fault handling for network file systems. A network file system is a distributed file system protocol that is a network abstraction above the file system. Its core function is to achieve cross-network transparency.

[0028] File sharing allows remote clients to access network file systems over the network in a manner similar to local file systems.

[0029] With the growth of data scale, the increasing requirements of users for the service reliability of network file systems, and the increasing complexity of anomaly detection in network file systems, the reliability maintenance of network file systems faces many challenges. In related technologies, static thresholds and single-dimensional detection are usually used, with a large proportion of false alarm rates and missed detection rates. Relevant operation and maintenance personnel need to spend a lot of time on handling invalid alarms, and the probability of discovering obscure faults and the real-time nature of discovery are low. In addition, in related technologies, full polling is usually used for fault detection. Full polling is a communication mechanism in which a client periodically sends a request to the server to obtain all data. This method consumes a lot of central processing unit (CPU) resources and will affect the normal operation of the network file system.

[0030] In addition, in related technologies, the processing methods after detecting a fault also have great limitations. Automatic processing is mostly a simple restart of the system, which will have a great impact on the business in the network file system. Many fault scenarios require human intervention.

[0031] In order to at least partially solve the above technical problems, in this embodiment, through the dual-modal perception detection of fault risks of network state and service state, and the intelligent fault handling method of dynamic adaptive control, it is possible to obtain network state data and service state data in the network file system in a targeted manner, perform fault risk analysis based on the acquired data, obtain the risk level of the fault, and automatically perform corresponding fault handling operations according to the risk level, thereby realizing an intelligent and complete process from fault risk detection to fault risk handling.

[0032] In this embodiment, the network state data and service state data of the network file system during the current detection cycle are first obtained to obtain the current network state data and current service state data. Network state data and service state data are two different data processing forms in network communications, corresponding to the underlying transmission layer and the upper-layer application layer, respectively. Network state data refers to the raw data units transmitted at the bottom layer of the network protocol stack (such as the physical layer, data link layer, and network layer), such as bit streams, data frames, receive and transmit queue information, etc.; service state data refers to data processed for business logic in the application layer, such as user requests and request processing results.

[0033] It should be noted that the network state data and service state data in this embodiment refer to partial data used for fault risk detection. By obtaining the above partial data in a targeted manner, the performance occupation can be reduced and the operating efficiency of the network file system can be improved. Optionally, the above network state data may include connection state data, such as the frequency of connection establishment and disconnection, connection duration, etc., for monitoring the stability of network connections; transmission performance indicators, such as the number of data transmitted, transmission delay, packet loss rate, etc., for identifying faults such as network congestion, abnormal delay, or abnormal data transmission; network queue lengths, such as the send queue length and the receive queue length, for reflecting the network's reception and transmission of information and detecting data backlogs at the network level. The above service state data may include operation statistics, such as the number of operations per second, the response time of read and write operations, the operation recognition rate, etc., for evaluating the processing capacity and stability of the service; error logs and event records, which record abnormal information and error events during the service process for fault location and root cause analysis.

[0034] After obtaining the current network state data and the current service state data, the network file system is subjected to fault risk detection by integrating the current network state data and the current service state data. Here, the data of different aspects in the current network state data and the current service state data can be integrated to comprehensively detect the fault risk of the network file system in the current detection cycle, and determine the location where the fault risk occurs and the risk level of the fault risk. For example, when the connection state data of the network state data indicates that the connection is unstable, and the service state data indicates that the reception of the event is abnormal, it can be determined that the fault risk comes from the network state data, and the risk level of the fault risk can be determined based on the continuous event of unstable connection. For example, if the fault risk of unstable connection is detected in several consecutive detection cycles, it can be determined that the risk level of the fault risk is high; if the fault risk of unstable connection is detected only in the current detection cycle, it can be determined that the risk level of the fault risk is low.

[0035] If a failure risk is detected in a network file system, and the risk level of the failure risk is a target risk level among multiple risk levels, a processing operation corresponding to the target risk level is performed on the network file system, where each risk level among the multiple risk levels corresponds to a processing operation. Optionally, for lower risk levels, only the failure risk may be recorded to prevent excessive interference with the failure risk from affecting the stability of the network file system. For higher risk levels, the module or system corresponding to the failure risk may be restarted. For even higher risk levels, the entire network file system may be saved and restarted, or relevant operations and maintenance personnel may be notified to address the failure risk.

[0036] Through the embodiments provided by the present application, network state data and service state data of a network file system in a current detection cycle are obtained to obtain current network state data and current service state data; fault risk detection is performed on the network file system by fusing the current network state data and the current service state data; and when it is detected that a fault risk exists in the network file system and the risk level of the fault risk exists in the network file system is a target risk level among multiple risk levels, a processing operation corresponding to the target risk level is performed on the network file system, wherein each risk level in the multiple risk levels corresponds to a processing operation. This solves the technical problems of low fault detection accuracy and insufficient intelligence of fault processing in fault processing methods in related technologies, thereby improving the accuracy of fault detection and the intelligence of fault processing.

[0037] In an exemplary embodiment, the above method also includes: using a network probe to collect network status data of the network file system according to a first sampling interval, wherein the network status data of the network file system is used to indicate the queue length of the receiving queue of the network file system and the queue length of the sending queue of the network file system; using a service probe to collect service status data of the network file system according to a second sampling interval, wherein the service status data of the network file system is used to indicate the number of received requests and the number of processed requests, the number of received requests is the number of processed requests received by the network file system, and the number of processed requests is the number of processed requests processed by the network file system.

[0038] In order to obtain network state data and service state data for fault risk detection, it is necessary to use a network probe and a service probe to collect network state data and service state data respectively. In this embodiment, the network state data of the network file system is collected using a network probe according to a first sampling interval, and the service state data of the network file system is collected using a service probe according to a second sampling interval. Here, the first sampling interval and the second sampling interval can be set based on experience. For example, the first sampling interval can be 10 seconds and the second sampling interval can be 5 seconds. This is not limited in this embodiment. It should be noted that both the first sampling interval and the second sampling interval need to be less than one detection cycle, that is, ensure that the network state data and service state data are sampled at least once within one detection cycle.

[0039] In this embodiment, the sampled network status data of the network file system is used to indicate the queue length of the receiving queue of the network file system and the queue length of the sending queue of the network file system. The queue length of the receiving queue and the queue length of the sending queue of the network file system can reflect the accumulation and congestion of the received and sent data, thereby detecting whether there is a failure risk in the network status data.

[0040] The netstat command displays network-related data, including network connections, routing tables, and interface statistics. Optionally, a network probe can use the netstat command to obtain real-time Transmission Control Protocol (TCP) connection status data. Using the "netstat -tunpalx" command's polling mechanism, the network probe can perform sampling at 10-second intervals and output real-time data from the network file system's receive and send queues. Since network file systems typically communicate using the Transmission Control Protocol / Internet Protocol (TCP / IP) and over the default port 2049, to specifically obtain output related to network transmission activity related to the network file system service, regular expressions can be used to filter out network connections to or originating from port 2049. The network probe's information collection command might be: "nohup netstat -tunpalx | grep ':2049' > / var / log / recvq.log &."

[0041] In this embodiment, the sampled network file system service status data is used to indicate the number of received requests and the number of processed requests. The number of received requests is the number of processing requests received by the network file system, and the number of processed requests is the number of processing requests processed by the network file system. The number of received requests and the number of processed requests can reflect whether the network file system is performing the tasks of receiving and processing requests normally, and can detect whether the request processing service is at risk of being stuck.

[0042] Optionally, service probes can monitor the status of service request reception and processing using the Network File System (NFS) Operations Per Second (OPS) statistics command. In the file system's OPS metrics, the "total" row typically summarizes the total number of request operations of all types (e.g., reads, writes, metadata operations, etc.), including the number of received and processed requests. Service probes can parse this row at a 5-second sampling interval to obtain the required number of received and processed requests. The service probe's information collection command can be: "nfs_stat_tool show_icfs_ops > / var / log / ops_metrics.log."

[0043] Through this embodiment, by using network probes and service probes, data required for fault risk detection can be collected in a targeted manner, which can improve data collection efficiency and reduce performance loss caused by data collection.

[0044] In an exemplary embodiment, in a network file system, queue failures of a receiving queue are recorded through a first fault list, queue failures of a sending queue are recorded through a second fault list, and service state failures of the network file system are recorded through a third fault list. The service state failure of the network file system refers to the number of received requests and the number of received requests satisfying specified fault conditions. The specified fault conditions are: the number of received requests is greater than the number of processed requests, and the number of received requests and the number of processed requests indicated by the service state data collected twice adjacently have not changed; by fusing the current network state data and the current service state data, fault risk detection is performed on the network file system, including: updating the fault records in the first fault list based on the receiving queue length indicated by the current network state data; updating the fault records in the second fault list based on the sending queue length indicated by the current network state data; updating the fault records in the third fault list based on the number of received requests indicated by the current service state data and the number of processed requests indicated by the current service state data; and performing fault risk detection on the network file system based on the fault records in the first fault list, the fault records in the second fault list, and the fault records in the third fault list.

[0045] In this embodiment, fault lists are used to record potential fault risk information for each data item. A first fault list records queue faults for the receiving queue, a second fault list records queue faults for the sending queue, and a third fault list records service-level faults for the network file system. When a potential fault risk is detected for relevant data or the fault risk is eliminated, the corresponding fault list can be updated. The fault list can record the time and number of times the fault risk was detected for the corresponding data, which can be used to determine the severity of the fault risk, that is, to assess the risk level of the fault risk.

[0046] It should be noted that the specified failure conditions for a network file system's service state failure are: the number of received requests is greater than the number of processed requests, and the number of received requests and the number of processed requests indicated by the service state data collected twice consecutively have not changed. Here, the number of received requests and the number of processed requests collected by the service probe are the total number of received requests and the number of processed requests from the time the network file system was started to the current collection time. If the number of received requests is greater than the number of processed requests, it can be determined that a backlog of pending requests has occurred. If the number of received requests and the number of processed requests indicated by the service state data collected twice consecutively have not changed, it can be determined that the service receiving and processing requests is stuck and no new requests are being received or processed. In this case, it can be determined that there is a risk of failure in the service state data.

[0047] During each detection cycle, the first, second, and third fault lists are updated, and fault risk detection is performed on the network file system based on the fault records in the first, second, and third fault lists. Optionally, network state and service state data can be integrated to perform cross-layer correlation analysis, that is, information from the first, second, and third fault lists can be integrated to perform fault risk detection. For example, if the third fault list indicates that the number of received requests has not changed, and the first fault list indicates that the receiving queue is congested, it can be determined that the fault risk originates from the receiving queue, and corresponding fault handling can be performed subsequently for the receiving queue.

[0048] Optionally, weighted sliding window correlation analysis can be used, and a dynamic weight adjustment mechanism can be used. When the resources of the network file system are tight, the network state weight in the correlation analysis can be reduced to prioritize the accuracy of service state data analysis.

[0049] Through this embodiment, fault information is recorded through the first fault queue, the second fault queue, and the third fault queue, and fault detection is performed based on the recorded information of the above three fault queues. Fault detection can be performed in combination with historical fault risk information to achieve continuous monitoring of faults and improve the accuracy of fault risk detection. Fault analysis is performed by integrating network status and service status data, and a cross-layer correlation model of the network layer and the service layer is used to analyze faults. This can break through the limitations of single-dimensional detection, thereby reducing the misjudgment rate and improving the accuracy of fault detection.

[0050] In an exemplary embodiment, to accurately detect faults within a detection cycle, the current network state data includes network state data collected multiple times within the current detection cycle; the current service state data includes service state data collected multiple times within the current detection cycle. That is, both the first sampling interval and the second sampling interval need to be less than one detection cycle, and multiple network state data and service state data samples are collected within one detection cycle.

[0051] Correspondingly, based on the receiving queue length indicated by the current network state data, the fault records in the first fault list are updated, including: in the current network state data, when the receiving queue length indicated by the network state data collected in one collection is greater than or equal to the receiving queue length threshold, a fault record is added to the first fault list; in the current network state data, when the receiving queue length indicated by the network state data collected in one collection is less than the receiving queue length threshold, the fault records in the first fault list are cleared.

[0052] The receiving queue length indicated by the network status data can indicate the congestion of the receiving queue. When the receiving queue length indicated by the network status data collected in one collection is greater than or equal to the receiving queue length threshold, it can be determined that there is congestion in the receiving queue at the time of collection, and a fault record can be added to the first fault list.

[0053] When the receiving queue length indicated by the network status data collected in one collection is less than the receiving queue length threshold, it can be determined that there is no congestion in the receiving queue at the time of collection or the previous congestion has been eliminated. At this time, there is no failure risk in the receiving queue. The fault records in the first fault list can be cleared to prevent outdated fault records from affecting fault detection.

[0054] Correspondingly, based on the sending queue length indicated by the current network state data, the fault records in the second fault list are updated, including: in the current network state data, when the sending queue length indicated by the network state data collected in one collection is greater than or equal to the sending queue length threshold, a fault record is added to the second fault list; in the current network state data, when the sending queue length indicated by the network state data collected in one collection is less than the sending queue length threshold, the fault records in the second fault list are cleared.

[0055] The sending queue length indicated by the network status data can indicate the congestion of the sending queue. When the sending queue length indicated by the network status data collected in one collection is greater than or equal to the sending queue length threshold, it can be determined that there is congestion in the sending queue at the time of collection, and a fault record can be added to the second fault list.

[0056] When the sending queue length indicated by the network status data collected in one collection is less than the sending queue length threshold, it can be determined that there is no congestion in the sending queue at the time of collection or the previous congestion has been eliminated. At this time, there is no failure risk in the sending queue. The fault records in the second fault list can be cleared to prevent outdated fault records from affecting fault detection.

[0057] Correspondingly, based on the number of received requests indicated by the current service state data and the number of processed requests indicated by the current service state data, the fault records in the third fault list are updated, including: in the current service state data, when the number of received requests and the corresponding number of processed requests indicated by the service state data collected in one collection meet the specified fault condition, a fault record is added to the third fault list; in the current service state data, when the number of received requests and the corresponding number of processed requests indicated by the service state data collected in one collection do not meet the specified fault condition, the fault records in the third fault list are cleared.

[0058] Changes in the number of received and processed requests indicated by service status data can indicate whether the service is operating normally. If the number of received requests is greater than the number of processed requests, there is a backlog of pending requests, and the request processing service may be at risk of being stuck. If the number of received and processed requests remains unchanged between two consecutive collections, both the request reception and request processing services may be at risk of being stuck. If the service status data collected in a single collection meets both of these conditions (i.e., the specified fault conditions), a fault record is added to the third fault list.

[0059] When the number of received requests and the corresponding number of processed requests indicated by the service state data collected in one collection do not meet the above-mentioned specified fault conditions, it can be determined that the service receiving and processing requests at the time of collection is executing normally. At this time, there is no fault risk in the service state data, and the fault records in the third fault list can be cleared.

[0060] According to this embodiment, by updating the fault records in the fault list through multiple sampling data, the accuracy and timeliness of fault detection can be improved, and outdated fault information can be prevented from affecting the results of fault detection.

[0061] In an exemplary embodiment, the above method also includes: using a system probe to obtain a parameter value of a load parameter of the network file system according to a third sampling interval, wherein the parameter value of the load parameter is determined based on at least one of the following: resource utilization of processor resources of the network file system; resource utilization of memory resources of the network file system.

[0062] To monitor the load of the network file system, a system probe can be used to obtain the value of a load parameter of the network file system at a third sampling interval. The third sampling interval can be set empirically, for example, to 1 second, 2 seconds, or another value, and is not limited in this embodiment. The value of the load parameter is determined based on the resource utilization of the network file system's processor resources, the resource utilization of the network file system's memory resources, or a combination of the resource utilization of the network file system's processor resources and memory resources, and can reflect the load of the network file system's processor or memory resources.

[0063] Optionally, the system probe can monitor the processor or memory metrics of the network file system using the sysstat toolchain, which is a set of performance monitoring tools that can be used to collect and analyze system resource usage data. Here, taking the third sampling interval as 1 second as an example, the system probe startup script can be:

[0064] #! / bin / bash

[0065] while true; do

[0066] timestamp=$(date +%s%3N)

[0067] cpu_usage=$(mpstat 1 1 | awk ' / Average / {print 100 - $12}')

[0068] mem_usage=$(free -m | awk ' / Mem / {printf "%.1f", $3 / $2*100}')

[0069] echo "{\"timestamp\":$timestamp,\"cpu\":$cpu_usage,\"mem\":$mem_usage}" >> / var / log / system_metrics.log

[0070] sleep 1

[0071] done

[0072] Optionally, there can be a data collection layer responsible for collecting relevant data using network probes, service probes, and system probes, such as Figure 3 As shown, network probes can collect TCP / UDP traffic from client connections, service probes can collect operations per second metrics from network file system service processes, and system probes can obtain processor / memory data from system resources.

[0073] Correspondingly, based on the parameter value of the load parameter, the receiving queue length threshold and the sending queue length threshold are adjusted, wherein the receiving queue length threshold and the sending queue length threshold are negatively correlated with the parameter value of the load parameter.

[0074] Here, the receive queue length threshold and the send queue length threshold can be used to determine the congestion status of the receive queue and the send queue. When the queue length of the receive queue or the send queue is greater than or equal to the corresponding length threshold, it can be determined that the queue is congested and there is a risk of failure. In order to intervene in queue congestion in advance when the network file system is highly loaded and reduce the consequences of failure, the receive queue length threshold and the send queue length threshold can be flexibly adjusted based on the parameter value of the load parameter. That is, the higher the parameter value of the load parameter, the lower the receive queue length threshold and the send queue length threshold, making it easier to detect and handle the risk of failure in the receive queue and the send queue; and the lower the parameter value of the load parameter, the higher the receive queue length threshold and the send queue length threshold, which can tolerate congestion in the receive queue and the send queue to a certain extent, allowing for short-term queue congestion.

[0075] Optionally, the receive queue length threshold and the send queue length threshold can be set to 10,000 by default, and the thresholds can be lowered to 70% when the load is greater than 80, and raised to 130% when the load is less than 30. The algorithm for dynamically adjusting the thresholds can be:

[0076] #Dynamic threshold algorithm

[0077] def calculate_dynamic_threshold():

[0078] current_load = get_cpu_usage()

[0079] base_value = 10000 # default threshold

[0080] if current_load > 80:

[0081] return base_value * 0.7 # Lower the threshold when the load is high

[0082] elif current_load < 30:

[0083] return base_value * 1.3 # Relax the threshold when the load is low

[0084] else:

[0085] return base_value

[0086] Through this embodiment, by obtaining the load parameters of the network file system, the load condition of the network file system can be determined, and the dynamic threshold algorithm can be improved. The threshold of fault risk detection can be adaptively and dynamically adjusted based on the load condition, which can improve the flexibility of fault risk detection.

[0087] In an exemplary embodiment, after using a system probe to obtain the parameter value of the load parameter of the network file system according to the third sampling interval, the above method also includes: adjusting the second sampling interval based on the parameter value of the load parameter, wherein the second sampling interval is positively correlated with the parameter value of the load parameter.

[0088] In order to dynamically adjust the fault detection standard of the service state data side based on the dynamic load situation of the network file system, the second sampling interval can be adjusted based on the parameter value of the load parameter, and the second sampling interval is positively correlated with the parameter value of the load parameter.

[0089] Here, the longer the second sampling interval, that is, the fewer the number of samples in the current detection cycle, the higher the standard for fault risk assessment can be, thereby reducing the risk of misjudgment under high load conditions, allowing short-term service request accumulation, and prioritizing service availability; under low load conditions, the second sampling interval is shorter, the assessment of fault risk is more relaxed, and related faults can be discovered and handled earlier when resources are sufficient.

[0090] Optionally, the second sampling interval can also be adjusted based on historical data and current load conditions. The code for dynamically adjusting the second sampling interval can be: def dynamic_threshold(historical_data, current_load):

[0091] baseline = np.percentile(historical_data, 90)

[0092] return baseline * (1 + current_load / 100) #The higher the load, the longer the second sampling interval

[0093] Through this embodiment, by dynamically adjusting the second sampling interval based on the load conditions of the network file system, the risk of misjudgment can be reduced under high load conditions, the waste of resources can be reduced, and faults can be strictly judged under low load conditions, so that faults can be discovered and handled early and resources can be used effectively.

[0094] In an exemplary embodiment, after using a system probe to obtain the parameter value of the load parameter of the network file system according to a third sampling interval, the above method also includes: when the current parameter value of the load parameter is greater than or equal to the first load parameter threshold, when the number of connected clients recorded in the client monitoring list is greater than a specified number, clearing the records of other connected clients in the client monitoring list except for the first specified number of connected clients, wherein the client monitoring list is used to record the connected clients of the network file system in order of connection heat from high to low, and the connection heat is determined by the number of pending requests existing in the corresponding connected clients.

[0095] To reduce load pressure and avoid unnecessary waste of detection resources under high load conditions, we can use an LRU (Least Recently Used) cache structure to intelligently manage connections. The core idea of the LRU cache structure is to eliminate data that has not been accessed for the longest time, that is, the connection with the lowest connection popularity.

[0096] In this embodiment, the client monitoring list sorts the connections in descending order of connection heat. When the current parameter value of the load parameter is greater than or equal to the first load parameter threshold, the connections with the lowest connection heat can be cleared, and only a specified number of connections with higher connection heat can be retained. Here, the specified number can be set based on experience. It can be a fixed value, such as 500, 550, or other values, or it can be adjusted according to the specific load situation. For example, when the current load is high, the specified number can be correspondingly lower, reducing the number of retained connections to reduce load pressure. When the current load is low, the specified number can be correspondingly higher, increasing the number of retained connections.

[0097] Optionally, the connection heat can be calculated as shown in formula (1).

[0098] Heat = a*receive queue accumulation data volume / dynamic threshold + b*receive queue accumulation duration; (1)

[0099] Where a and b are weight systems. The values of a and b can be set based on experience. For example, a can be 0.6, 0.7, or 0.8, and b can be 0.2, 0.3, or 0.4. This embodiment does not impose any restrictions on this. The above formula (1) combines the amount of accumulated data in the receiving queue and the accumulation duration, which can accurately identify long-term abnormal connections.

[0100] Through this embodiment, a client monitoring list with heat weight is adopted, and its capacity can be automatically adjusted according to the load. By dynamically adjusting the monitored connections based on the load situation, the load pressure can be reduced under high load conditions.

[0101] In an exemplary embodiment, the above method also includes: periodically detecting the number of pending requests of connected clients of the network file system; when there are connected clients with zero number of pending requests among the connected clients recorded in the detected client monitoring list, clearing the records of the connected clients with zero number of pending requests in the client monitoring list; when there are connected clients with zero number of pending requests that are not recorded in the detected client monitoring list, adding corresponding records in the client monitoring list for the connected clients with zero number of pending requests that are not recorded in the client monitoring list.

[0102] In order to reduce the resource usage of low-activity connected clients, the number of pending requests of connected clients of the network file system can be periodically detected, and connected clients to be monitored can be intelligently added or deleted based on the detection data, and connected clients with zero pending requests can be cleared, thereby improving monitoring efficiency.

[0103] In this embodiment, if a connected client with zero pending requests is found in the detected client monitoring list, the connected client can be directly cleared to reduce resource usage. In addition, some connected clients may not be recorded in the client monitoring list. If a connected client with zero pending requests that is not recorded in the client monitoring list is detected, the client can be added to the client monitoring list and a corresponding record can be added to the client monitoring list, making it easier to clear the recorded clients with zero pending requests in subsequent detections.

[0104] Through this embodiment, by adding or deleting monitoring items according to detection data, low-heat clients with zero pending requests can be cleared, thereby improving monitoring efficiency and accuracy.

[0105] In an exemplary embodiment, after using a system probe to obtain the parameter value of the load parameter of the network file system according to a third sampling interval, the above method also includes: when the parameter value of the load parameter is greater than or equal to a second load parameter threshold, when the target risk level is not the highest risk level among multiple risk levels, updating the target risk level to a risk level one level higher than the target risk level among multiple risk levels.

[0106] In order to thoroughly handle failure risks and quickly restore business under high load conditions, in this embodiment, the failure risk handling strategy can be upgraded under high load conditions, that is, when the parameter value of the load parameter is greater than or equal to the second load parameter threshold, when the target risk level is not the highest risk level among multiple risk levels, the target risk level is updated to a risk level one level higher than the target risk level among multiple risk levels.

[0107] Optionally, you can also adopt corresponding fault handling strategies based on the processor load or memory load. For example, if the processor utilization is too high, you can increase the target risk level of the fault risk; if the memory utilization is too high, you can disable the pstack command to prioritize memory freeing.

[0108] Through this embodiment, by increasing the target risk level when the parameter value of the load parameter is high, the probability of completely handling the fault risk in one go can be increased, the overall resource loss caused by handling the fault risk can be reduced, and the business affected by the fault risk can be restored faster.

[0109] In an exemplary embodiment, the multiple risk levels include a first risk level and a second risk level with increasing severity. Here, the first risk level corresponds to a low risk or a small impact failure risk, and the second risk level corresponds to a high risk or a large impact failure risk.

[0110] Correspondingly, based on the fault records in the first fault list, the fault records in the second fault list, and the fault records in the third fault list, fault risk detection is performed on the network file system, including: determining that there is a fault risk in the network file system, and the risk level of the fault risk in the network file system is the first risk level when at least one of the following is met: the number of fault records in the first fault list is greater than or equal to the first number threshold and less than the second number threshold; the number of fault records in the second fault list is greater than or equal to the third number threshold and less than the fourth number threshold; the number of fault records in the third fault list is greater than or equal to the fifth number threshold and less than the sixth number threshold.

[0111] Here, the number of fault records in the first fault list, the second fault list or the third fault list is the number of consecutive abnormalities of the corresponding data. When the number of consecutive abnormalities is small, that is, the number of fault records in the first fault list is less than the first number threshold, the number of fault records in the second fault list is less than the third number threshold, and the number of fault records in the third fault list is less than the fifth number threshold, the probability of a fault risk is low and it is not considered to be a fault risk. When the number of consecutive abnormalities is large but does not meet the conditions of the second risk level, the network file system is considered to have a first risk level fault risk.

[0112] Correspondingly, when at least one of the following is met, it is determined that there is a failure risk in the network file system, and the risk level of the failure risk in the network file system is the second risk level: the number of failure records in the first failure list is greater than or equal to the second number threshold; the number of failure records in the second failure list is greater than or equal to the fourth number threshold; the number of failure records in the third failure list is greater than or equal to the sixth number threshold.

[0113] Here, there are many fault records in the first fault list, the second fault list or the third fault list, that is, when multiple consecutive abnormalities are detected, it can be determined that the severity of the fault risk is high, and its risk level is also the second risk level higher than the first risk level.

[0114] Optionally, the above-mentioned first number threshold, second number threshold, third number threshold, fourth number threshold, fifth number threshold and sixth number threshold can be adjusted according to the load parameters. For example, when the load parameters are high, the number threshold can be increased accordingly to reduce the number of detections of fault risks and the probability of false alarms of fault risks, thereby reducing the resource loss caused by handling faults; when the load parameters are low, the number threshold can be lowered accordingly to increase the number of detections of fault risks, thereby achieving early detection, early handling of fault risks, and full use of surplus system resources.

[0115] Through this embodiment, the risk level of the fault risk is judged by detecting the number of fault records in the fault list, and the risk level of the fault risk can be determined according to the number of consecutive fault detections, thereby achieving different processing for fault risks of different severity and improving the intelligence of automatic processing of fault risks.

[0116] In an exemplary embodiment, multiple risk levels include a first risk level and a second risk level with increasing severity; performing processing operations corresponding to the target risk level on the network file system, including: when the target risk level is the first risk level, recording failure risk information of the failure risk existing in the network file system into a log file; when the target risk level is the second risk level, capturing stack information after detecting the failure risk of the network file system at least twice, and performing differential analysis on the at least two captured first stack information.

[0117] For the lower-severity first risk level, simply record the failure risk and log the failure risk information of the network file system to a log file. No direct action is taken to prevent unnecessary impact on the network file system. For the higher-severity second risk level, which may have a significant impact on the network file system, the corresponding data needs to be captured and analyzed for further processing.

[0118] At the second risk level, stack traces must be captured at least twice after the fault occurs. The first capture can be at the moment the fault risk is detected, and the second capture can be performed some time later to ensure that the impact of the fault risk on the stack environment is captured. After capturing at least two stack traces, differential analysis can be performed on the captured stack traces to identify changes to the network file system during the fault period and the actual impact of the fault risk. This helps further determine the appropriate remediation measures for the fault risk and accelerates root cause identification.

[0119] Through this embodiment, by taking different processing measures for different risk levels of fault risks and recording or analyzing fault risks, the efficiency of fault handling can be improved, service interruption events during fault handling can be reduced, and the impact of faults on the business can be reduced.

[0120] In an exemplary embodiment, the multiple risk levels also include a third risk level and a fourth risk level with increasing severity and both higher than the second risk level; executing the processing operation corresponding to the target risk level on the network file system also includes: when it is determined based on the differential analysis results of at least two first stack information that some connected clients of the network file system have abnormalities, executing the following processing operation corresponding to the third risk level on the network file system: first terminating the network connection between the network file system and the abnormal client, and then triggering the restoration of the network connection between the network file system and the abnormal client, wherein the abnormal client is an abnormal connected client among the connected clients of the network file system; when it is determined based on the differential analysis results of at least two first stack information that all connected clients of the network file system have abnormalities, executing the following processing operation corresponding to the fourth risk level on the network file system: capturing stack information at least twice again, saving the captured at least two second stack information, and restarting the service of the network file system.

[0121] After performing a differential analysis of the second risk level, the corresponding risk level can be further determined and the corresponding actions can be taken. If the target risk level of the fault risk is determined to be the third risk level, the abnormal client can be reconnected to address the fault risk without affecting other business services. If the target risk level of the fault risk is determined to be the fourth risk level, the entire network file system service must be restarted, and at least two stack traces must be captured before the restart for recordkeeping and evidence collection.

[0122] In this embodiment, at the third risk level, the impact of the failure risk is limited to a subset of clients. The network connection with the abnormal client is terminated, and then the recovery mechanism is triggered to reestablish the connection with the abnormal client. This process only affects the connection of the client identified as abnormal and does not interfere with other normally operating clients. The network file system can maintain continuity for the majority of normal clients.

[0123] At the fourth risk level, all or most connected clients are identified as anomalies, requiring a more aggressive recovery strategy. This includes re-capturing the secondary stack at least twice to ensure the stack details at the time of the anomaly are stored for subsequent root cause analysis. Next, to address the global issue, the network file system service must be restarted to restore it to its initial, relatively stable state. During the restart, all client connections are securely disconnected and re-established to minimize service recovery time and business impact.

[0124] Optionally, the code for handling failure risk at the fourth risk level may be:

[0125] # Fourth risk level recovery logic (service restart + stack forensics)

[0126] if [ $RISK_LEVEL -eq 4 ]; then

[0127] pid=$(pidof nfs.nfsd)

[0128] pstack $pid > stack_$(date +%s).log

[0129] sleep 5

[0130] pstack $pid > stack_$(date +%s).log

[0131] systemctl restart nfs

[0132] fi

[0133] Through this embodiment, by introducing the handling strategies of the third risk level and the fourth risk level, the impact scope of the fault risk can be controlled within the abnormal client when the impact scope of the fault risk is small. When the impact scope of the fault risk is large, the stack details at the abnormal moment can be saved and the service can be restarted. Through differentiated processing methods, the fault risk can be handled flexibly, and the fault handling capability and recovery speed of the network file system can be improved.

[0134] The following describes the fault handling method in the embodiment of the present application in conjunction with an optional example. In this optional example, the fault handling method is executed by a fault handling system.

[0135] Figure 4 This is the architecture diagram of the fault handling system in this optional example. Figure 4 As shown, the fault handling system includes a data acquisition layer, an analysis and decision-making layer, and an execution control layer.

[0136] Client connection: The user client can mount distributed file storage through the network file system protocol, run services under the mount point, and generate input / output (I / O) business interactions with the storage.

[0137] Data collection layer: Introduces a multi-source heterogeneous data fusion engine and uses network probes, service probes, and system probes to collect multi-dimensional information, including real-time network state data of receiving queues and sending queues, service state data of operations per second, and monitoring and collection of processor / memory system resource utilization data.

[0138] Analysis and decision-making layer: Introduce a multimodal perception engine (network status, service status, system resources) to establish a multidimensional detection model; the dynamic baseline library stores 7 days of historical data and supports sliding window threshold calculation; the decision tree divides risk levels from level 1 to level 4 according to different risk levels, and selects different disposal methods according to the risk level to perform graded risk judgment.

[0139] Optionally, the dynamic baseline library can store historical data and generate adaptive thresholds. These adaptive thresholds can be used to assess failure risk and adjust thresholds to adapt to business fluctuations. Seven days of historical data can be stored and updated hourly. Tiered storage is used, divided into hot, warm, and cold data. Hot data is data from 0 to 1 hour, warm data is data from 1 to 24 hours, and cold data is data from 24 hours to 7 days.

[0140] The above-mentioned decision tree module can trigger hierarchical disposal actions based on multi-dimensional rules. Here, the multi-dimensional rules may include time dimension, network dimension and application dimension, among which the time dimension is related to the current adaptive threshold, the network dimension is related to the queue length growth rate of the current receiving queue, and the application dimension is related to the current business request processing efficiency.

[0141] Optionally, the fault handling system can simultaneously analyze the growth rate of the length of the receiving request queue at the network layer and the request processing efficiency at the application layer, including calculating the growth rate of the TCP connection state matrix in the network dimension, calculating the request processing efficiency in real time in the application dimension, and predicting the request processing efficiency trend.

[0142] The Execution Control Layer implements a hierarchical handling strategy, taking appropriate actions based on the handling level, including logging, connection termination, stack forensics, and service restart. Level 1 Warning: Log recording; Level 2 Forensics: Stack capture and differential analysis using the pstack command; Level 3 Handling: Terminate the abnormal client connection, triggering recovery. This only affects the abnormal client connection, minimizing the impact and ensuring service continuity for the node; Level 4 Recovery: Restart the network file system service. When the entire service fails, Level 3 cannot guarantee serviceability, requiring a service restart.

[0143] Through this optional example, a fault risk detection method combining network status and service status is used, which greatly improves the fault detection rate and significantly reduces the false alarm rate, greatly reducing the workload of maintenance personnel and reducing labor costs; and the fault handling system can optimize the processor utilization of service monitoring, save processor resources, and improve the stability and performance of the network file system; in terms of fault recovery, a four-level closed-loop control from early warning to restart is adopted to reduce manual intervention and improve the recovery speed; through the data collection layer, analysis and decision-making layer, and execution control layer, closed-loop self-healing is achieved, and a three-level response strategy of "early warning → evidence collection → disposal" is constructed, and the three-layer decoupled architecture of data collection → analysis and decision-making → execution control can achieve modular expansion, which improves the stability of the network file system and reduces maintenance costs.

[0144] It should be noted that for the aforementioned method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that this application is not limited by the order of the actions described, because according to this application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by this application.

[0145] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus the necessary general hardware platform, and of course it can also be implemented by hardware, but in many cases the former is a better implementation method.

[0146] According to another aspect of the embodiments of the present application, a fault handling device is also provided, which can be used to implement the fault handling method provided in the above embodiments, and will not be repeated hereafter. As used below, the term "module" can be a combination of software and / or hardware that implements a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, implementation in hardware, or a combination of software and hardware, is also possible and contemplated.

[0147] Figure 5 is a structural block diagram of an optional fault handling device according to an embodiment of the present application, such as Figure 5 As shown in , the fault handling device includes an acquisition unit 502 , a first detection unit 504 and an execution unit 506 .

[0148] An acquiring unit 502 is configured to acquire network state data and service state data of the network file system in a current detection cycle, and obtain current network state data and current service state data;

[0149] A first detection unit 504 is configured to perform fault risk detection on the network file system by integrating current network state data and current service state data;

[0150] The execution unit 506 is used to perform a processing operation corresponding to the target risk level on the network file system when it is detected that the network file system has a failure risk and the risk level of the failure risk of the network file system is a target risk level among multiple risk levels, wherein one risk level among the multiple risk levels corresponds to one processing operation.

[0151] It should be noted that the acquisition unit 502 in this embodiment can be used to execute the above step S202, the first detection unit 504 in this embodiment can be used to execute the above step S204, and the execution unit 506 in this embodiment can be used to execute the above step S206.

[0152] Through the embodiments provided by the present application, network state data and service state data of a network file system in a current detection cycle are obtained to obtain current network state data and current service state data; fault risk detection is performed on the network file system by fusing the current network state data and the current service state data; and when it is detected that a fault risk exists in the network file system and the risk level of the fault risk exists in the network file system is a target risk level among multiple risk levels, a processing operation corresponding to the target risk level is performed on the network file system, wherein each risk level in the multiple risk levels corresponds to a processing operation. This solves the technical problems of low fault detection accuracy and insufficient intelligence of fault processing in fault processing methods in related technologies, thereby improving the accuracy of fault detection and the intelligence of fault processing.

[0153] In an exemplary embodiment, the above-mentioned device also includes: a first collection unit, used to use a network probe to collect network status data of the network file system according to a first sampling interval, wherein the network status data of the network file system is used to indicate the queue length of the receiving queue of the network file system and the queue length of the sending queue of the network file system; a second collection unit, used to use a service probe to collect service status data of the network file system according to a second sampling interval, wherein the service status data of the network file system is used to indicate the number of received requests and the number of processed requests, the number of received requests is the number of processed requests received by the network file system, and the number of processed requests is the number of processed requests processed by the network file system.

[0154] In an exemplary embodiment, in a network file system, queue faults of a receiving queue are recorded through a first fault list, queue faults of a sending queue are recorded through a second fault list, and service state faults of the network file system are recorded through a third fault list. A service state fault of the network file system refers to a situation where the number of received requests and the number of received requests satisfy a specified fault condition. The specified fault condition is: the number of received requests is greater than the number of processed requests, and the number of received requests and the number of processed requests indicated by two adjacent collected service state data do not change. A first detection unit includes: a first update module for updating fault records in the first fault list based on the receiving queue length indicated by current network state data; a second update module for updating fault records in the second fault list based on the sending queue length indicated by current network state data; a third update module for updating fault records in the third fault list based on the number of received requests and the number of processed requests indicated by current service state data; and a detection module for performing fault risk detection on the network file system based on the fault records in the first fault list, the fault records in the second fault list, and the fault records in the third fault list.

[0155] In an exemplary embodiment, the current network state data includes the network state data collected by multiple collections in the current detection cycle; the current service state data includes the service state data collected by multiple collections in the current detection cycle; the first update module includes: a first adding submodule for adding a fault record in the first fault list when the receiving queue length indicated by the network state data collected by one collection is greater than or equal to the receiving queue length threshold; a first clearing submodule for clearing the fault record in the first fault list when the receiving queue length indicated by the network state data collected by one collection is less than the receiving queue length threshold in the current network state data; the second updating module includes: a second adding submodule for clearing the sending queue length indicated by the network state data collected by one collection in the current network state data. When the column length is greater than or equal to the sending queue length threshold, a fault record is added to the second fault list once; a second clearing submodule is used to clear the fault record in the second fault list when, in the current network state data, the sending queue length indicated by the network state data collected by one collection is less than the sending queue length threshold; the third update module includes: a third adding submodule, used to add a fault record to the third fault list when, in the current service state data, the number of received requests and the corresponding number of processed requests indicated by the service state data collected by one collection meet the specified fault condition; a third clearing submodule is used to clear the fault record in the third fault list when, in the current service state data, the number of received requests and the corresponding number of processed requests indicated by the service state data collected by one collection do not meet the specified fault condition.

[0156] In an exemplary embodiment, the above-mentioned device also includes: a third acquisition unit, used to use a system probe to obtain the parameter value of the load parameter of the network file system according to a third sampling interval, wherein the parameter value of the load parameter is determined based on at least one of the following: resource utilization of the processor resources of the network file system; resource utilization of the memory resources of the network file system; a first adjustment unit, used to adjust the receiving queue length threshold and the sending queue length threshold based on the parameter value of the load parameter, wherein the receiving queue length threshold and the sending queue length threshold are negatively correlated with the parameter value of the load parameter.

[0157] In an exemplary embodiment, the above-mentioned device also includes: a second adjustment unit, which is used to adjust the second sampling interval based on the parameter value of the load parameter after using the system probe to obtain the parameter value of the load parameter of the network file system according to the third sampling interval, wherein the second sampling interval is positively correlated with the parameter value of the load parameter.

[0158] In an exemplary embodiment, the above-mentioned device also includes: a first clearing unit, which is used to use a system probe to obtain the parameter value of the load parameter of the network file system according to the third sampling interval, and when the current parameter value of the load parameter is greater than or equal to the first load parameter threshold, when the number of connected clients recorded in the client monitoring list is greater than a specified number, clear the records of other connected clients in the client monitoring list except for the first specified number of connected clients, wherein the client monitoring list is used to record the connected clients of the network file system in order of connection heat from high to low, and the connection heat is determined by the number of pending requests existing in the corresponding connected clients.

[0159] In an exemplary embodiment, the above-mentioned device also includes: a second detection unit, which is used to periodically detect the number of pending requests of connected clients of the network file system; a second clearing unit, which is used to, when there is a connected client with zero number of pending requests among the connected clients recorded in the detected client monitoring list, clear the record of the connected client with zero number of pending requests in the client monitoring list; and an adding unit, which is used to, when there is a connected client with zero number of pending requests that is not recorded in the client monitoring list, add a corresponding record in the client monitoring list for the connected client with zero number of pending requests that is not recorded in the client monitoring list.

[0160] In an exemplary embodiment, the above-mentioned device also includes: an updating unit, which is used to, after using a system probe to obtain the parameter value of the load parameter of the network file system according to a third sampling interval, update the target risk level to a risk level one level higher than the target risk level among the multiple risk levels when the parameter value of the load parameter is greater than or equal to a second load parameter threshold and when the target risk level is not the highest risk level among the multiple risk levels.

[0161] In an exemplary embodiment, multiple risk levels include a first risk level and a second risk level with increasing severity; the detection module includes: a first determination submodule, used to determine that there is a failure risk in the network file system, and the risk level of the failure risk in the network file system is the first risk level when at least one of the following is met: the number of failure records in the first failure list is greater than or equal to the first number threshold and less than the second number threshold; the number of failure records in the second failure list is greater than or equal to the third number threshold and less than the fourth number threshold; the number of failure records in the third failure list is greater than or equal to the fifth number threshold and less than the sixth number threshold; a second determination submodule, used to determine that there is a failure risk in the network file system, and the risk level of the failure risk in the network file system is the second risk level when at least one of the following is met: the number of failure records in the first failure list is greater than or equal to the second number threshold; the number of failure records in the second failure list is greater than or equal to the fourth number threshold; the number of failure records in the third failure list is greater than or equal to the sixth number threshold.

[0162] In an exemplary embodiment, multiple risk levels include a first risk level and a second risk level with increasing severity; the execution unit includes: a recording module, which is used to record the failure risk information of the failure risk of the network file system into a log file when the target risk level is the first risk level; a first execution module, which is used to capture stack information after detecting the failure risk of the network file system at least twice when the target risk level is the second risk level, and perform differential analysis on the at least two captured first stack information.

[0163] In an exemplary embodiment, the multiple risk levels also include a third risk level and a fourth risk level with increasing severity and both higher than the second risk level; the execution unit also includes: a second execution module, which is used to perform the following processing operations corresponding to the third risk level on the network file system when it is determined based on the differential analysis results of at least two first stack information that some connected clients of the network file system have abnormalities: first terminate the network connection between the network file system and the abnormal client, and then trigger the restoration of the network connection between the network file system and the abnormal client, wherein the abnormal client is an abnormal connected client among the connected clients of the network file system; a third execution module, which is used to perform the following processing operations corresponding to the fourth risk level on the network file system when it is determined based on the differential analysis results of at least two first stack information that all connected clients of the network file system have abnormalities: capture the stack information at least twice again, save the captured at least two second stack information, and restart the service of the network file system.

[0164] An embodiment of the present application further provides an electronic device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to execute the steps in any one of the above-mentioned fault handling method embodiments.

[0165] An embodiment of the present application further provides a computer-readable storage medium, in which a computer program is stored. The computer program is configured to execute the steps of any of the above-mentioned fault handling method embodiments when running.

[0166] In an exemplary embodiment, the computer-readable storage medium may include, but is not limited to, various media that can store computer programs, such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk, or an optical disk.

[0167] An embodiment of the present application further provides a computer program product, which includes a computer program. When the computer program is executed by a processor, the steps in any of the above-mentioned fault handling method embodiments are implemented.

[0168] An embodiment of the present application further provides another computer program product, including a non-volatile computer-readable storage medium, wherein the non-volatile computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in any of the above-mentioned fault handling method embodiments are implemented.

[0169] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the above description has generally described the components and steps of each example according to their functions. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0170] The above is a detailed introduction to a fault handling method and device, electronic device and storage medium provided by the present application. Specific examples are used herein to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea. It should be pointed out that for ordinary technicians in this technical field, without departing from the principles of the present application, several improvements and modifications can be made to the present application, and these improvements and modifications also fall within the scope of protection of the claims of the present application.

Claims

1. A fault handling method, characterized in that: include: Obtain the network state data and service state data of the network file system in the current detection cycle, and obtain the current network state data and current service state data; Performing fault risk detection on the network file system by fusing the current network state data and the current service state data; When it is detected that the network file system has a failure risk, and the risk level of the failure risk of the network file system is a target risk level among multiple risk levels, a processing operation corresponding to the target risk level is performed on the network file system, wherein one risk level among the multiple risk levels corresponds to one processing operation.

2. The method according to claim 1, characterized in that The method further comprises: Using a network probe, collecting network status data of the network file system at a first sampling interval, wherein the network status data of the network file system is used to indicate a queue length of a receiving queue of the network file system and a queue length of a sending queue of the network file system; Use a service probe to collect service status data of the network file system according to a second sampling interval, wherein the service status data of the network file system is used to indicate the number of received requests and the number of processed requests, the number of received requests is the number of processing requests received by the network file system, and the number of processed requests is the number of processing requests processed by the network file system.

3. The method according to claim 2, characterized in that In the network file system, queue faults of the receiving queue are recorded in a first fault list, queue faults of the sending queue are recorded in a second fault list, and service state faults of the network file system are recorded in a third fault list. The service state fault of the network file system refers to the number of received requests and the number of received requests satisfying a specified fault condition. The specified fault condition is: the number of received requests is greater than the number of processed requests, and the number of received requests and the number of processed requests indicated by two consecutive collected service state data do not change. The performing fault risk detection on the network file system by fusing the current network state data and the current service state data includes: Based on the receiving queue length indicated by the current network status data, updating the fault record in the first fault list; Based on the sending queue length indicated by the current network state data, updating the fault record in the second fault list; updating the fault records in the third fault list based on the number of received requests indicated by the current service state data and the number of processed requests indicated by the current service state data; Fault risk detection is performed on the network file system based on the fault records in the first fault list, the fault records in the second fault list, and the fault records in the third fault list.

4. The method according to claim 3, characterized in that The current network state data includes the network state data collected by multiple collections during the current detection period; the current service state data includes the service state data collected by multiple collections during the current detection period; The updating of the fault record in the first fault list based on the receive queue length indicated by the current network state data includes: adding a fault record to the first fault list when the receive queue length indicated by the network state data collected in one collection in the current network state data is greater than or equal to the receive queue length threshold; and clearing the fault record in the first fault list when the receive queue length indicated by the network state data collected in one collection in the current network state data is less than the receive queue length threshold; The updating of the fault record in the second fault list based on the sending queue length indicated by the current network state data includes: adding a fault record to the second fault list when the sending queue length indicated by the network state data collected in one collection in the current network state data is greater than or equal to the sending queue length threshold; and clearing the fault record in the second fault list when the sending queue length indicated by the network state data collected in one collection in the current network state data is less than the sending queue length threshold; The updating of the fault records in the third fault list based on the number of received requests and the number of processed requests indicated by the current service state data includes: in the current service state data, if the number of received requests and the corresponding number of processed requests indicated by the service state data collected in one collection meet the specified fault condition, adding a fault record to the third fault list; in the current service state data, if the number of received requests and the corresponding number of processed requests indicated by the service state data collected in one collection do not meet the specified fault condition, clearing the fault records in the third fault list.

5. The method according to claim 4, characterized in that The method further comprises: Using a system probe, obtaining a parameter value of a load parameter of the network file system at a third sampling interval, wherein the parameter value of the load parameter is determined based on at least one of the following: a resource utilization rate of a processor resource of the network file system; a resource utilization rate of a memory resource of the network file system; Based on the parameter value of the load parameter, the receiving queue length threshold and the sending queue length threshold are adjusted, wherein the receiving queue length threshold and the sending queue length threshold are negatively correlated with the parameter value of the load parameter.

6. The method according to claim 5, characterized in that After using the system probe to obtain the parameter value of the load parameter of the network file system according to the third sampling interval, the method further includes: The second sampling interval is adjusted based on a parameter value of the load parameter, wherein the second sampling interval is positively correlated with the parameter value of the load parameter.

7. The method according to claim 5, characterized in that After using the system probe to obtain the parameter value of the load parameter of the network file system according to the third sampling interval, the method further includes: In the case where the current parameter value of the load parameter is greater than or equal to the first load parameter threshold, when the number of connected clients recorded in the client monitoring list is greater than a specified number, the records of other connected clients except the specified number of connected clients in the client monitoring list are cleared, wherein the client monitoring list is used to record the connected clients of the network file system in descending order of connection heat, and the connection heat is determined by the number of pending requests existing in the corresponding connected clients.

8. The method according to claim 7, characterized in that The method further comprises: Periodically detecting the number of pending requests of connected clients of the network file system; If there is a connected client with zero number of pending requests among the detected connected clients recorded in the client monitoring list, clear the record of the connected client with zero number of pending requests from the client monitoring list; When it is detected that there is a connected client that is not recorded in the client monitoring list and has zero pending requests, a corresponding record is added to the client monitoring list for the connected client that is not recorded in the client monitoring list and has zero pending requests.

9. The method according to claim 5, characterized in that After using the system probe to obtain the parameter value of the load parameter of the network file system according to the third sampling interval, the method further includes: When the parameter value of the load parameter is greater than or equal to a second load parameter threshold, and when the target risk level is not the highest risk level among the multiple risk levels, the target risk level is updated to a risk level one level higher than the target risk level among the multiple risk levels.

10. The method according to claim 3, characterized in that The multiple risk levels include a first risk level and a second risk level with increasing severity; The performing fault risk detection on the network file system based on the fault records in the first fault list, the fault records in the second fault list, and the fault records in the third fault list includes: The network file system is determined to have a failure risk, and the risk level of the failure risk of the network file system is the first risk level if at least one of the following is satisfied: the number of failure records in the first failure list is greater than or equal to a first number threshold and less than a second number threshold; the number of failure records in the second failure list is greater than or equal to a third number threshold and less than a fourth number threshold; and the number of failure records in the third failure list is greater than or equal to a fifth number threshold and less than a sixth number threshold. If at least one of the following is met, it is determined that the network file system has a failure risk, and the risk level of the failure risk of the network file system is the second risk level: the number of failure records in the first failure list is greater than or equal to the second number threshold; the number of failure records in the second failure list is greater than or equal to the fourth number threshold; the number of failure records in the third failure list is greater than or equal to the sixth number threshold.

11. The method according to any one of claims 1 to 9, characterized in that The multiple risk levels include a first risk level and a second risk level with increasing severity; The performing of the processing operation corresponding to the target risk level on the network file system includes: When the target risk level is the first risk level, recording the failure risk information of the failure risk existing in the network file system into a log file; When the target risk level is the second risk level, stack information after the failure risk of the network file system is detected is captured at least twice, and differential analysis is performed on the captured at least two pieces of first stack information.

12. The method according to claim 11, characterized in that The multiple risk levels further include a third risk level and a fourth risk level, which have increasing severity levels and are both higher than the second risk level; The performing of the processing operation corresponding to the target risk level on the network file system further includes: When it is determined based on the differential analysis results of the at least two first stack information that some of the clients connected to the network file system are abnormal, performing the following processing operations corresponding to the third risk level on the network file system: first terminating the network connection between the network file system and the abnormal client, and then triggering restoration of the network connection between the network file system and the abnormal client, wherein the abnormal client is an abnormal client among the connected clients of the network file system; When it is determined based on the differential analysis results of the at least two first stack information that all connected clients of the network file system are abnormal, the following processing operation corresponding to the fourth risk level is performed on the network file system: the stack information is captured at least twice again, the captured at least two second stack information are saved, and the service of the network file system is restarted.

13. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the fault handling method according to any one of claims 1 to 12 are implemented.

14. An electronic device, characterized in that: include: memory for storing computer programs; A processor, configured to implement the steps of the fault handling method according to any one of claims 1 to 12 when executing the computer program.

15. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, wherein the computer program, when executed by a processor, implements the steps of the fault handling method according to any one of claims 1 to 12.

Images