A method, device, computer-readable storage medium and electronic device for intelligently processing distributed system faults

Abstract

The present application relates to a method and device for intelligently processing faults in a distributed system. The method includes: starting an application service and generating a heartbeat packet; sending a heartbeat packet to a heartbeat queue at a set time interval; a monitoring service receiving and saving the data in the heartbeat queue; continuously collecting the data in the heartbeat queue from each node and preprocessing it, extracting features related to fault detection, and using an isolation forest model for model training; using an adaptive threshold adjustment method based on an isolation forest algorithm to analyze the data in the heartbeat packet; comparing the heartbeat according to the threshold, and when an abnormal heartbeat is found in a node, determining the fault location according to the feature code; generating a compensation message, and the application service executing the compensation task according to the compensation message. The method detects faults by combining a heartbeat mechanism with an AI algorithm, and recovers unfinished tasks by a compensation mechanism. The method is adaptable to complex distributed environments, has high real-time performance and sensitivity, and significantly improves the stability and reliability of the system.

Description

Technical Field

[0001] The present application belongs to the technical field of system fault processing, and in particular relates to a method, device, computer-readable storage medium, and electronic device for intelligently processing distributed system faults. Background Art

[0002] In recent years, with the needs of civil aviation business development, the freight rate publishing business has gradually transitioned from the original mainframe to a distributed cloud-native environment. With the cloudification of the freight rate publishing business, distributed systems have become a core component of the modern Internet architecture. However, due to the inherent complexity of the distributed environment, multi-node distributed task processing leads to frequent task interruptions, which poses a huge challenge to the overall stability of the system and service quality.

[0003] While distributed systems bring high availability, scalability, and flexibility, they also inevitably generate complexity and variability in business processing, which is mainly reflected in the increased risk of task interruption in multi-node and multi-task chains. There are many factors that lead to task interruption, including hardware failure, software failure, resource exhaustion, load imbalance, resource competition, improper scheduling strategies, scheduling delays, configuration errors, etc. Summary of the Invention

[0004] To address the aforementioned issues posed by distributed systems, this application provides an intelligent fault handling method based on a heartbeat mechanism and artificial intelligence algorithms. This method aims to accurately identify faulty nodes by dynamically adjusting heartbeat detection criteria and recover unfinished tasks on faulty nodes through a compensation mechanism. This method combines heartbeat packet design, a heartbeat anomaly detection algorithm, and a fault compensation mechanism to achieve rapid detection of faulty nodes and task compensation in distributed systems.

[0005] In summary, this application was designed based on a thorough analysis of the multi-node, multi-task distributed processing architecture of the freight rate publishing system. As the number of nodes increases, the probability of failure of a single node also increases, posing a severe challenge to the overall stability and reliability of the system. Therefore, it is crucial to design a mechanism that can promptly detect and compensate for failed node processing tasks. The intelligent fault handling method proposed in this application uses a heartbeat mechanism combined with an AI algorithm to detect faults and a compensation mechanism to recover unfinished tasks, significantly improving the stability and reliability of the system.

[0006] In order to achieve the above objectives, this application provides the following technical solutions:

[0007] A first aspect of the present application provides a method for intelligently handling faults in a distributed system, the method comprising:

[0008] Start the application service and generate a heartbeat packet containing a globally unique feature code;

[0009] Start the heartbeat service and send heartbeat packets to the heartbeat queue at the set time interval;

[0010] The monitoring service receives and saves the data of the heartbeat queue;

[0011] Continuously collect heartbeat queue data from each node, preprocess the collected heartbeat data, extract features related to fault detection from the preprocessed data, and then use the isolation forest model for model training;

[0012] Adopting the adaptive threshold adjustment method based on the isolation forest algorithm to analyze the data in the heartbeat packet;

[0013] Compare heartbeats against thresholds. If a node's heartbeat is abnormal, determine the fault location based on a globally unique signature.

[0014] Generate a compensation message containing the faulty node's signature code and send it to the compensation queue. After the application service receives the compensation message from the compensation queue, it queries the progress of the task currently being processed by the faulty node.

[0015] According to the task progress information of the faulty node, the breakpoint location is selected to continue executing the unfinished task.

[0016] Optionally, in the method of the present application, the heartbeat packet includes the following information:

[0017] Machine IP: used to identify the location of the node in the network;

[0018] Service name: used to describe the type of service provided by the node;

[0019] Startup time: records the timestamp of node startup, used to track the life cycle of the node;

[0020] Distributed ID: A globally unique identifier assigned to a node;

[0021] Current time: the timestamp when the heartbeat packet is sent, used to determine whether the heartbeat has timed out;

[0022] Verification code: Calculated by hash function based on the signature code and current time, used to prevent the heartbeat packet from being forged;

[0023] Message body: used to transmit characteristic information related to fault detection.

[0024] Optionally, in the method of the present application, the machine IP, service name, startup time and distributed ID constitute a globally unique feature code;

[0025] The message body contains the following information: CPU utilization, memory usage, network latency, and the operating indicators of the service itself.

[0026] Optionally, in the method of the present application, the preprocessing of the collected heartbeat data includes: removing noise, filling missing values, and normalizing values.

[0027] Optionally, in the method of the present application, the process of constructing an isolated tree in the isolation forest algorithm is as follows:

[0028] (1) Randomly select a feature: randomly select a feature from all features;

[0029] (2) Randomly select a split point: randomly select a split point on the selected feature, and the split point is any value between the minimum and maximum values ​​of the feature;

[0030] (3) Segmenting the data set: Divide the feature data into two parts according to the segmentation point, and recursively repeat the above segmentation process for the two parts of data until any of the following stopping conditions is met:

[0031] Stop condition a: When there is only one sample left in the data subset;

[0032] Stop condition b: When the pre-set maximum tree depth is reached.

[0033] Optionally, in the method of the present application, the adaptive threshold adjustment method includes: learning normal heartbeat patterns through the isolation forest algorithm and adjusting the threshold accordingly; when the heartbeat signal is detected to deviate from the normal range, dynamically adjusting the detection threshold of the heartbeat signal according to the degree of deviation.

[0034] Optionally, the method of the present application further includes a step of dynamic threshold adjustment calculation, including:

[0035] During the training phase, a baseline threshold is set according to the distribution of normal heartbeat signals;

[0036] When the heartbeat signal is detected to deviate from the normal range, the threshold is adjusted according to the degree of deviation;

[0037] Continuously optimize the threshold adjustment strategy through online learning or incremental learning.

[0038] A second aspect of the present application provides a distributed system fault intelligent processing device, the device comprising:

[0039] Task startup module: used to start the application service, generate a heartbeat packet, and the heartbeat packet contains a globally unique feature code; start the heartbeat service and send the heartbeat packet to the heartbeat queue at a set time interval;

[0040] Data processing module: used to receive and save heartbeat queue data; continuously collect heartbeat queue data from each node, preprocess the collected heartbeat data, extract features related to fault detection from the preprocessed data, and then use the isolation forest model for model training;

[0041] Data analysis module: This module uses an adaptive threshold adjustment method based on the isolation forest algorithm to analyze the data in the heartbeat packet. It compares the heartbeat according to the threshold. When an abnormal heartbeat is found on a node, the fault location is determined based on the globally unique signature code.

[0042] Task compensation module: used to generate a compensation message containing the fault node feature code and send it to the compensation queue. After the application service obtains the compensation message from the compensation queue, it queries the progress of the task currently being processed by the faulty node; based on the task progress information of the faulty node, it selects the breakpoint location to continue executing the unfinished task.

[0043] The device implements the steps of the aforementioned distributed system fault intelligent processing method when running.

[0044] A third aspect of the present application provides an electronic device, comprising: a memory and a processor;

[0045] Memory: used to store computer programs;

[0046] Processor: used to execute the computer program to implement the steps of the aforementioned distributed system fault intelligent processing method.

[0047] A fourth aspect of the present application provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the steps of the aforementioned method for intelligently handling distributed system faults are implemented.

[0048] In summary, this application proposes a new method for intelligently handling distributed system faults. As a necessary auxiliary system for the business system, this method mainly completes intelligent handling of distributed system faults based on the heartbeat mechanism and AI algorithm. The advantages of this method are as follows:

[0049] (1) Adaptability to complex distributed environments: Traditional heartbeat mechanisms often use fixed thresholds to determine whether a heartbeat has timed out. This approach is difficult to adapt to dynamically changing workloads. This method uses adaptive threshold adjustment technology combined with the isolation forest algorithm to improve the accuracy of fault detection.

[0050] (2) Real-time performance and high sensitivity: The heartbeat mechanism can regularly check the status of each node. Once it is found that the node does not send the heartbeat signal on time, the fault detection process is immediately triggered. Combined with the AI ​​algorithm, it can more quickly and accurately determine whether it is a real fault or a network delay problem, so that timely measures can be taken to reduce the impact time of the fault on the system.

[0051] (3) Strong automatic recovery capability: When the AI ​​algorithm detects a fault, it can automatically perform corresponding compensation operations according to preset rules or models, such as reallocating tasks, starting backup nodes, etc., and complete the fault recovery process without human intervention, greatly improving the system's self-healing capability and efficiency.

[0052] (4) Strong scalability: As the scale of the system expands, traditional fault detection and recovery mechanisms are difficult to cope with the increasingly complex environment; however, the AI-based heartbeat mechanism and compensation strategy of this application are more flexible and easy to adjust to adapt to changes in demand in different scenarios, supporting the continuous growth and development of the system.

[0053] (5) This method improves the overall stability and reliability of the system, and significantly enhances the intelligence level of the system.

[0054] Other features and advantages of the present application will be described in the following description, and in part will become apparent from the description, or will be understood by practicing the present application. The purpose and other advantages of the present application can be achieved and obtained by the techniques indicated in the description, claims and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] In order to more clearly illustrate the technical solutions of the embodiments of the present application, a brief introduction will be given below to the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without any creative work.

[0056] Figure 1 This is a diagram of the overall design of the distributed system fault intelligent processing method provided in the embodiment of the present application.

[0057] Figure 2 This is a diagram of the heartbeat packet structure design in an embodiment of the present application.

[0058] Figure 3 This is a flowchart of the overall implementation of the distributed system intelligent fault handling method provided in the embodiment of the present application.

[0059] Figure 4 This is a flowchart of the intelligent processing of distributed system faults provided in the embodiment of the present application.

[0060] Figure 5 This is a structural diagram of the distributed system fault intelligent processing device provided in an embodiment of the present application.

[0061] Figure 6 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0062] To make the purpose, technical solutions, and advantages of the embodiments of the present application more clear, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0063] As used herein, the term “including” and its variations are open inclusions, ie, “including but not limited to”; the term “based on” means “based at least in part on”; and the term “one embodiment” means “at least one embodiment”.

[0064] It should be noted that the modifications of "one" and "multiple" mentioned in this application are illustrative rather than restrictive. Those skilled in the art should understand that unless the context clearly indicates otherwise, it should be understood as "one or more".

[0065] Explanation of terms:

[0066] Heartbeat mechanism: a common fault detection method in distributed systems.

[0067] Adaptive threshold adjustment: When the heartbeat signal is detected to deviate from the normal range, the threshold is dynamically adjusted according to the degree of deviation.

[0068] Isolation Forest: An unsupervised learning algorithm for anomaly detection. Its basic idea is to quickly isolate anomalous data points by building multiple isolation trees. Anomalous data points are easier to segment due to their uniqueness, resulting in shorter path lengths.

[0069] Signature code: is a term widely used in many fields such as computer science, software engineering, and information security. It refers to a piece of code, string or digital sequence used to identify a specific function, attribute or pattern.

[0070] Figure 1 The figure shows the overall design scheme of the method of this embodiment. This method receives heartbeat packets and checks whether the heartbeat packets are legal. It performs heartbeat anomaly detection on legal heartbeat packets to determine whether there are heartbeat anomalies. It identifies abnormal tasks (specific faults) based on the information carried by the abnormal heartbeat, and then performs compensation operations on the abnormal tasks.

[0071] The specific implementation process is as follows:

[0072] (1) Heartbeat packet design. In a distributed system, a node failure may cause the performance of the entire system to degrade or even crash. To prevent this from happening, there must be a mechanism that can quickly respond when a failure occurs. The heartbeat mechanism is one of the commonly used mechanisms. In order to more conveniently apply the heartbeat mechanism, the embodiment of the present application designs a unique heartbeat packet structure, specifically as follows: Figure 2 shown.

[0073] (1) The structure of the heartbeat packet of this application includes the following parts:

[0074] Machine IP: identifies the location of the node in the network.

[0075] Service name: describes the type of service provided by the node, that is, the type of tasks that the node can handle.

[0076] Startup time: records the timestamp of node startup to track the life cycle of the node.

[0077] Distributed ID: A globally unique identifier assigned to a node.

[0078] Current time: The timestamp when the heartbeat packet is sent, used to determine whether the heartbeat has timed out.

[0079] Verification code: Calculated based on the first four feature codes and the current time to prevent the heartbeat packet from being forged.

[0080] Message body: Conveys necessary characteristic information that helps with fault detection, including but not limited to CPU utilization, memory usage, network latency, and the service's operating indicators.

[0081] (2) Rules for generating feature codes and check codes:

[0082] The signature code consists of the node machine's IP, service name, startup time, and distributed ID. The checksum is calculated by hashing the signature code and the current time to ensure the authenticity and integrity of the heartbeat packet.

[0083] (2) Heartbeat Anomaly Detection Algorithm. To accurately detect heartbeat anomalies, this application uses an adaptive threshold adjustment strategy based on the isolation forest algorithm. Furthermore, this application can also utilize pattern recognition technology instead of the isolation forest algorithm to achieve adaptive threshold adjustment.

[0084] (1) Principle of Isolation Forest Algorithm: Isolation Forest is an unsupervised learning algorithm used for anomaly detection. Its basic idea is to quickly isolate abnormal data points by constructing multiple isolation trees. Abnormal data points are easier to segment due to their uniqueness, so the path length is shorter.

[0085] (2) Adaptive Threshold Adjustment: Dynamically adjusts the heartbeat signal detection criteria to suit different operating conditions. The Isolation Forest algorithm learns normal heartbeat patterns and adjusts the threshold accordingly. When the detected heartbeat signal deviates from the normal range, the threshold is dynamically adjusted based on the degree of deviation.

[0086] (3) Fault detection and compensation mechanism.

[0087] (1) Fault detection: When the heartbeat packet analysis finds that the heartbeat of a node is abnormal, the global unique signature is used to determine which IP and service has failed.

[0088] (2) Compensation mechanism: When a fault is detected, a compensation message containing the fault node signature is generated and sent to the compensation queue. After receiving the compensation message from the compensation queue, the application service queries the progress of the task currently being processed by the faulty service. Based on the task progress information, the application service selects an appropriate location (breakpoint) to continue executing the unfinished task, ensuring the integrity of task processing.

[0089] Figure 3 The figure shows the overall implementation process of the distributed system fault intelligent processing method of this embodiment, which includes the following steps:

[0090] Start the application service and generate a heartbeat packet containing a globally unique feature code;

[0091] Start the heartbeat service and send heartbeat packets to the heartbeat queue at the set time interval;

[0092] The monitoring service receives and saves the data of the heartbeat queue;

[0093] Continuously collect heartbeat queue data from each node, preprocess the collected heartbeat data, extract features related to fault detection from the preprocessed data, and then use the isolation forest model for model training;

[0094] Adopting the adaptive threshold adjustment method based on the isolation forest algorithm to analyze the data in the heartbeat packet;

[0095] Compare heartbeats against thresholds. If a node's heartbeat is abnormal, determine the fault location based on a globally unique signature.

[0096] Generate a compensation message containing the faulty node's signature code and send it to the compensation queue. After the application service receives the compensation message from the compensation queue, it queries the progress of the task currently being processed by the faulty node.

[0097] According to the task progress information of the faulty node, the breakpoint location is selected to continue executing the unfinished task.

[0098] In order to better understand the technical solution of this application, the distributed system fault intelligent processing process is described more clearly, and the following scenario is further used as an example to illustrate the specific process. Figure 4 shown.

[0099] The simulation sets up a distributed system environment, including application service (N working nodes), queue service (a virtual machine), and monitoring service (N monitoring nodes).

[0100] Step 1: After the application service is started, a globally unique signature is generated as part of the heartbeat packet.

[0101] Step 2: Start the heartbeat service and send heartbeat packets to the heartbeat queue at a certain time interval.

[0102] In step 3, the monitoring service subscribes to the heartbeat queue's messages and saves them locally after receiving them, which are then provided to the heartbeat detection model for analysis.

[0103] Step 4: Preprocessing heartbeat data. This involves continuously collecting heartbeat signals and related information from each node in the distributed system. Information such as CPU utilization, memory usage, network latency, and service performance metrics are captured from the heartbeat packet body. This data is used for subsequent feature extraction and model training. The preprocessing stage also includes operations such as noise removal, missing value filling, and numerical standardization to ensure input data quality.

[0104] Step 5, feature extraction, extracts features that are helpful for fault detection from the collected data and uses the labeled dataset to guide the model to learn the ability to distinguish normal from abnormal heartbeat signals.

[0105] In step 6, in order to accurately detect heartbeat anomalies, an adaptive threshold adjustment technique based on the isolation forest algorithm is used. The analyzed data comes from the message body of the heartbeat packet.

[0106] The construction process of the isolation tree includes the following steps:

[0107] (1) Randomly select a feature: Randomly select a feature from all features.

[0108] (2) Randomly select a split point: Randomly select a split point on the selected feature, usually any value between the minimum and maximum values ​​of the feature.

[0109] (3) Split the data set: Divide the data into two parts according to this split point, and recursively repeat this process for the two parts of data until the stopping condition is met.

[0110] The orphan tree construction process stops when one of the following conditions is met:

[0111] (1) When there is only one sample left in the data subset.

[0112] (2) When the pre-set maximum tree depth is reached.

[0113] Adaptive threshold adjustment dynamically adjusts the heartbeat signal detection criteria to suit different operating conditions. Using the Isolation Forest algorithm, the system learns normal heartbeat patterns and adjusts the threshold accordingly. When a detected heartbeat signal deviates from the normal range, the threshold is dynamically adjusted based on the degree of deviation.

[0114] Step 7: Dynamic threshold adjustment calculation. Assuming that there is already a model that can recognize normal heartbeat patterns, you can set a baseline threshold and then adjust it according to the actual situation:

[0115] Baseline threshold: During the training phase, an initial threshold is determined based on the distribution of normal heartbeat signals.

[0116] Threshold update rule: When a heartbeat signal is detected to deviate from the normal range, the threshold is adjusted based on the degree of deviation. For example, if the heartbeat interval becomes unstable, the threshold can be appropriately relaxed; conversely, if the system load is low but the heartbeat interval is still large, the threshold may need to be tightened.

[0117] Adaptive learning: As time goes by, the model will continue to receive new data points. Through online learning or incremental learning, the model can gradually optimize its threshold adjustment strategy to make it more consistent with the current system status.

[0118] Step 8: Compare the heartbeats based on the threshold. When the heartbeat packet analysis finds that the heartbeat of a certain node is abnormal, determine which IP and service has failed based on the globally unique signature.

[0119] Step 9: Send compensation message. When a fault is detected, a compensation message containing the fault node feature code is generated and sent to the compensation queue.

[0120] In step 10, the application service subscribes to the compensation message. After the application node obtains the compensation message from the compensation queue, it queries the progress of the task currently being processed by the fault service.

[0121] In step 11, the compensation operation is performed. According to the task progress information, a suitable location is selected to continue the unfinished task to ensure the integrity of task processing.

[0122] Figure 5 The figure shows the structure of the distributed system fault intelligent processing device of this embodiment, which includes:

[0123] Task startup module: used to start the application service, generate a heartbeat packet, and the heartbeat packet contains a globally unique feature code; start the heartbeat service and send the heartbeat packet to the heartbeat queue at a set time interval;

[0124] Data processing module: used to receive and save heartbeat queue data; continuously collect heartbeat queue data from each node, preprocess the collected heartbeat data, extract features related to fault detection from the preprocessed data, and then use the isolation forest model for model training;

[0125] Data analysis module: This module uses an adaptive threshold adjustment method based on the isolation forest algorithm to analyze the data in the heartbeat packet. It compares the heartbeat according to the threshold. When an abnormal heartbeat is found on a node, the fault location is determined based on the globally unique signature code.

[0126] Task compensation module: used to generate a compensation message containing the fault node feature code and send it to the compensation queue. After the application service obtains the compensation message from the compensation queue, it queries the progress of the task currently being processed by the faulty node; based on the task progress information of the faulty node, it selects the breakpoint location to continue executing the unfinished task.

[0127] When the above device is running, the steps of the distributed system fault intelligent processing method disclosed in this application are implemented.

[0128] The flow charts and block diagrams in the accompanying drawings illustrate the possible architecture, function and operation of the device, method and computer program product according to various embodiments of the present application. In this regard, each box in the flow chart or block diagram can represent a module, program segment or a part of code, and the part of this module, program segment or code comprises one or more executable instructions for realizing the logical function of regulation. It should also be noted that each box in the block diagram and / or flow chart and the combination of the boxes in the block diagram and / or flow chart can be realized by a special hardware-based system that performs the function or operation of regulation, or can be realized by a combination of special hardware and computer instructions.

[0129] like Figure 6 As shown, an embodiment of the present application further discloses an electronic device, comprising: a processor 310, a communication interface 320, a memory 330 for storing a computer program executable by the processor, and a communication bus 340. The processor 310, the communication interface 320, and the memory 330 communicate with each other via the communication bus 340. The processor 310 executes the executable computer program to implement the steps of the above-described method for intelligently handling distributed system faults.

[0130] It is understood that, in addition to the memory and processor, the electronic device may also include input devices such as a keyboard, output devices such as a display, and other communication modules. The input devices, output devices, and other communication modules communicate with the processor via an I / O interface (i.e., an input / output interface).

[0131] Computer program code for performing the operations of the present application may be written in one or more programming languages, or a combination thereof, including, but not limited to, object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0132] Furthermore, the present application also discloses a computer-readable storage medium. When the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device can execute each step of the distributed system fault intelligent processing method disclosed in the present application.

[0133] In the context of this application, a computer-readable storage medium may be a tangible medium, more specific examples of which would include a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0134] In particular, according to embodiments of the present application, the process described in the flowchart can be implemented as a computer software program. For example, embodiments of the present application include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for executing the method for intelligently handling distributed system faults disclosed in the present application. When the computer program is executed by a processing device, the above-described functions defined in the method of the embodiments of the present application are performed.

[0135] Although the above discussion includes several specific implementation details, these should not be interpreted as limiting the scope of this application. The above description is only a preferred embodiment of the present application and an explanation of the technical principles used. Those skilled in the art should understand that the scope of disclosure involved in this application is not limited to the technical solutions formed by the specific combination of the above technical features, but also should cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above disclosed concepts.

[0136] Those skilled in the art should also understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein; and these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A distributed system fault intelligent processing method, characterized in that: The method comprises: Start the application service and generate a heartbeat packet containing a globally unique feature code; Start the heartbeat service and send heartbeat packets to the heartbeat queue at the set time interval; The monitoring service receives and saves the data of the heartbeat queue; Continuously collect heartbeat queue data from each node, preprocess the collected heartbeat data, extract features related to fault detection from the preprocessed data, and then use the isolation forest model for model training; Adopting the adaptive threshold adjustment method based on the isolation forest algorithm to analyze the data in the heartbeat packet; Compare heartbeats against thresholds. If a node's heartbeat is abnormal, determine the fault location based on a globally unique signature. Generate a compensation message containing the faulty node's signature code and send it to the compensation queue. After the application service receives the compensation message from the compensation queue, it queries the progress of the task currently being processed by the faulty node. According to the task progress information of the faulty node, the breakpoint location is selected to continue executing the unfinished task.

2. The method according to claim 1, characterized in that The heartbeat packet contains the following information: Machine IP: used to identify the location of the node in the network; Service name: used to describe the type of service provided by the node; Startup time: records the timestamp of node startup, used to track the life cycle of the node; Distributed ID: A globally unique identifier assigned to a node; Current time: the timestamp when the heartbeat packet is sent, used to determine whether the heartbeat has timed out; Verification code: Calculated by hash function based on the signature code and current time, used to prevent the heartbeat packet from being forged; Message body: used to transmit characteristic information related to fault detection.

3. The method according to claim 2, characterized in that The machine IP, service name, startup time and distributed ID constitute a globally unique feature code; The message body contains the following information: CPU utilization, memory usage, network latency, and the operating indicators of the service itself.

4. The method according to claim 1, wherein The preprocessing of the collected heartbeat data includes: removing noise, filling missing values ​​and normalizing values.

5. The method according to claim 1, wherein The construction process of the isolation tree in the isolation forest algorithm is as follows: (1) Randomly select a feature: randomly select a feature from all features; (2) Randomly select a split point: randomly select a split point on the selected feature, and the split point is any value between the minimum and maximum values ​​of the feature; (3) Segmenting the data set: Divide the feature data into two parts according to the segmentation point, and recursively repeat the above segmentation process for the two parts of data until any of the following stopping conditions is met: Stop condition a: When there is only one sample left in the data subset; Stop condition b: When the pre-set maximum tree depth is reached.

6. The method according to claim 1, characterized in that The adaptive threshold adjustment method includes: learning normal heartbeat patterns through an isolation forest algorithm and adjusting the threshold accordingly; when a heartbeat signal is detected to deviate from a normal range, dynamically adjusting the detection threshold of the heartbeat signal according to the degree of deviation.

7. The method according to claim 1, characterized in that The method further includes a step of dynamic threshold adjustment calculation, including: During the training phase, a baseline threshold is set according to the distribution of normal heartbeat signals; When the heartbeat signal is detected to deviate from the normal range, the threshold is adjusted according to the degree of deviation; Continuously optimize the threshold adjustment strategy through online learning or incremental learning.

8. An intelligent device for handling distributed system faults, characterized in that: The device comprises: Task startup module: used to start the application service, generate a heartbeat packet, and the heartbeat packet contains a globally unique feature code; start the heartbeat service and send the heartbeat packet to the heartbeat queue at a set time interval; Data processing module: used to receive and save heartbeat queue data; continuously collect heartbeat queue data from each node, preprocess the collected heartbeat data, extract features related to fault detection from the preprocessed data, and then use the isolation forest model for model training; Data analysis module: This module uses an adaptive threshold adjustment method based on the isolation forest algorithm to analyze the data in the heartbeat packet. It compares the heartbeat according to the threshold. When an abnormal heartbeat is found on a node, the fault location is determined based on the globally unique signature code. Task compensation module: used to generate a compensation message containing the fault node feature code and send it to the compensation queue. After the application service obtains the compensation message from the compensation queue, it queries the progress of the task currently being processed by the faulty node; based on the task progress information of the faulty node, it selects the breakpoint location to continue executing the unfinished task.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the distributed system fault intelligent processing method according to any one of claims 1 to 7 are implemented.

10. An electronic device, characterized in that: include: memory and processor; Memory: used to store computer programs; Processor: used to execute the computer program to implement the steps of the distributed system fault intelligent processing method according to any one of claims 1 to 7.

Images