Distributed system fault intelligent processing method and device, computer readable storage medium and electronic equipment

Abstract

The invention relates to a distributed system fault intelligent processing method and device. The method comprises the following steps: starting an application service, and generating a heartbeat packet; sending the heartbeat packets to a heartbeat queue according to a set time interval; the monitoring service receives and stores data of the heartbeat queue; continuously collecting data of a heartbeat queue from each node and preprocessing the data, extracting features related to fault detection, and performing model training by using an isolated forest model; analyzing data in the heartbeat packet by adopting a self-adaptive threshold adjustment method based on an isolated forest algorithm; comparing the heartbeat according to a threshold value, and determining a fault position according to the feature code when the heartbeat of a certain node is abnormal; and generating a compensation message, and executing the compensation task by the application service according to the compensation message. According to the method, a heartbeat mechanism is combined with an AI algorithm to detect faults, uncompleted tasks are recovered through a compensation mechanism, the method adapts to a complex distributed environment, the real-time performance and the sensitivity are high, and the stability and the reliability of a system are remarkably improved.

Description

Technical Field

[0001] The present application belongs to the technical field of system fault processing, and in particular, relates to a distributed system fault intelligent processing method, device, computer-readable storage medium and electronic device. Background Art

[0002] In recent years, with the needs of the development of civil aviation business, the freight rate publishing business has gradually transitioned from the original mainframe to a distributed cloud-native environment. With the cloudification process of the freight rate publishing business, distributed systems have become a core component of the modern Internet architecture. However, due to the inherent complexity of the distributed environment, multi-node distributed task processing leads to frequent task interruptions, which poses a huge challenge to the overall stability of the system and service quality.

[0003] While distributed systems bring high availability, scalability and flexibility, they also inevitably generate complexity and variability in business processing, which is mainly reflected in the increased risk of task interruption in multi-node and multi-task chains. There are many factors that lead to task interruption, including hardware failure, software failure, resource exhaustion, load imbalance, resource competition, improper scheduling strategy, scheduling delay, configuration error, etc. Summary of the invention

[0004] In response to the above problems caused by distributed systems, this application provides an intelligent fault handling method based on a heartbeat mechanism and an artificial intelligence algorithm, which aims to accurately identify faulty nodes by dynamically adjusting the heartbeat detection standard and restore unfinished tasks of faulty nodes through a compensation mechanism. This method combines the heartbeat packet design, the heartbeat anomaly detection algorithm and the fault compensation mechanism to achieve rapid detection of faulty nodes and task compensation in distributed systems.

[0005] In summary, this application is designed based on a full analysis of the architecture of the multi-node multi-task distributed processing of the freight rate publishing system. As the number of nodes increases, the probability of failure of a single node also increases, which poses a severe challenge to the overall stability and reliability of the system. Therefore, it is crucial to design a mechanism that can promptly detect and compensate for the processing tasks of faulty nodes. The intelligent fault handling method proposed in this application detects faults through a heartbeat mechanism combined with an AI algorithm, and recovers unfinished tasks through a compensation mechanism, which significantly improves the stability and reliability of the system.

[0006] In order to achieve the above objectives, this application provides the following technical solutions:

[0007] The first aspect of the present application provides a distributed system fault intelligent processing method, the method comprising:

[0008] Start the application service and generate a heartbeat packet, wherein the heartbeat packet contains a globally unique feature code;

[0009] Start the heartbeat service and send heartbeat packets to the heartbeat queue at the set time interval;

[0010] The monitoring service receives and saves the data of the heartbeat queue;

[0011] Continuously collect heartbeat queue data from each node, preprocess the collected heartbeat data, extract features related to fault detection from the preprocessed data, and then use the isolation forest model for model training;

[0012] Adopting the adaptive threshold adjustment method based on isolation forest algorithm to analyze the data in the heartbeat packet;

[0013] Compare the heartbeats according to the threshold. When a node's heartbeat is abnormal, determine the fault location based on the globally unique signature code.

[0014] Generate a compensation message containing the faulty node feature code and send it to the compensation queue. After the application service obtains the compensation message from the compensation queue, it queries the progress of the task currently being processed by the faulty node.

[0015] According to the task progress information of the faulty node, select the breakpoint location to continue executing the unfinished task.

[0016] Optionally, in the method of the present application, the heartbeat packet includes the following information:

[0017] Machine IP: used to identify the location of the node in the network;

[0018] Service name: used to describe the type of service provided by the node;

[0019] Startup time: records the timestamp of node startup, used to track the life cycle of the node;

[0020] Distributed ID: A globally unique identifier assigned to a node;

[0021] Current time: the timestamp when the heartbeat packet is sent, used to determine whether the heartbeat has timed out;

[0022] Verification code: calculated by hash function based on the feature code and current time, used to prevent the heartbeat packet from being forged;

[0023] Message body: used to transmit characteristic information related to fault detection.

[0024] Optionally, in the method of the present application, the machine IP, service name, startup time and distributed ID constitute a globally unique feature code;

[0025] The message body includes the following information: CPU utilization, memory usage, network latency, and the operating indicators of the service itself.

[0026] Optionally, in the method of the present application, the preprocessing of the collected heartbeat data includes: removing noise, filling missing values, and standardizing values.

[0027] Optionally, in the method of the present application, the construction process of the isolated tree in the isolation forest algorithm is as follows:

[0028] (1) Randomly select a feature: randomly select a feature from all features;

[0029] (2) Randomly select a split point: randomly select a split point on the selected feature, and the split point is any value between the minimum and maximum values ​​of the feature;

[0030] (3) Segmenting the data set: The feature data is divided into two parts according to the segmentation point, and the above segmentation process is recursively repeated for the two parts of data until any of the following stopping conditions is met:

[0031] Stop condition a: When there is only one sample left in the data subset;

[0032] Stop condition b: When the preset maximum tree depth is reached.

[0033] Optionally, in the method of the present application, the adaptive threshold adjustment method includes: learning the normal heartbeat pattern through the isolation forest algorithm and adjusting the threshold accordingly; when the heartbeat signal is detected to deviate from the normal range, dynamically adjusting the detection threshold of the heartbeat signal according to the degree of deviation.

[0034] Optionally, the method of the present application further includes a step of dynamic threshold adjustment calculation, including:

[0035] During the training phase, a baseline threshold is set according to the distribution of normal heartbeat signals;

[0036] When the heartbeat signal is detected to deviate from the normal range, the threshold is adjusted according to the degree of deviation;

[0037] Continuously optimize the threshold adjustment strategy through online learning or incremental learning.

[0038] A second aspect of the present application provides a distributed system fault intelligent processing device, the device comprising:

[0039] Task startup module: used to start the application service, generate a heartbeat packet, and the heartbeat packet contains a globally unique feature code; start the heartbeat service, and send the heartbeat packet to the heartbeat queue at a set time interval;

[0040] Data processing module: used to receive and save the data of the heartbeat queue; continuously collect the data of the heartbeat queue from each node, pre-process the collected heartbeat data, extract the features related to fault detection from the pre-processed data, and then use the isolation forest model for model training;

[0041] Data analysis module: used to analyze the data in the heartbeat packet using an adaptive threshold adjustment method based on the isolation forest algorithm; compare the heartbeat according to the threshold, and when an abnormal heartbeat of a node is found, determine the fault location based on the globally unique feature code;

[0042] Task compensation module: used to generate a compensation message containing the characteristic code of the faulty node and send it to the compensation queue. After the application service obtains the compensation message from the compensation queue, it queries the progress of the task currently being processed by the faulty node; based on the task progress information of the faulty node, it selects the breakpoint location to continue executing the unfinished task.

[0043] The device implements the steps of the aforementioned distributed system fault intelligent processing method when running.

[0044] A third aspect of the present application provides an electronic device, comprising: a memory and a processor;

[0045] Memory: used to store computer programs;

[0046] Processor: used to execute the computer program to implement the steps of the aforementioned distributed system fault intelligent processing method.

[0047] A fourth aspect of the present application provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps of the aforementioned distributed system fault intelligent processing method are implemented.

[0048] In summary, this application proposes a new intelligent processing method for distributed system faults. As a necessary auxiliary system for the business system, this method mainly completes the intelligent processing of distributed system faults based on the heartbeat mechanism and AI algorithm. The advantages of this method are as follows:

[0049] (1) Adapting to complex distributed environments: Traditional heartbeat mechanisms often use fixed thresholds to determine whether the heartbeat has timed out. This approach is difficult to adapt to dynamically changing workloads. This method uses adaptive threshold adjustment technology combined with the isolation forest algorithm to improve the accuracy of fault detection.

[0050] (2) Real-time performance and high sensitivity: The heartbeat mechanism can regularly check the status of each node. Once it is found that the node does not send the heartbeat signal on time, the fault detection process is immediately triggered. Combined with the AI ​​algorithm, it can more quickly and accurately determine whether it is a real fault or a network delay, so as to take timely measures to reduce the impact time of the fault on the system.

[0051] (3) Strong automatic recovery capability: When the AI ​​algorithm detects a fault, it can automatically perform corresponding compensation operations according to preset rules or models, such as reallocating tasks, starting backup nodes, etc., and complete the fault recovery process without human intervention, greatly improving the system's self-healing capability and efficiency.

[0052] (4) Strong scalability: As the scale of the system expands, traditional fault detection and recovery mechanisms are unable to cope with increasingly complex environments; however, the AI-based heartbeat mechanism and compensation strategy of this application are more flexible and easy to adjust to adapt to changing needs in different scenarios, supporting the continuous growth and development of the system.

[0053] (5) This method improves the overall stability and reliability of the system, and significantly enhances the intelligence level of the system.

[0054] Other features and advantages of the present application will be described in the following description, and partly become apparent from the description, or understood by practicing the present application. The purpose and other advantages of the present application can be realized and obtained by the techniques indicated in the description, claims and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] In order to more clearly illustrate the technical solutions of the embodiments of the present application, a brief introduction is given below to the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0056] Figure 1 This is a schematic diagram of the overall design of the distributed system fault intelligent processing method provided in the embodiment of the present application.

[0057] Figure 2 This is a diagram of the heartbeat packet structure design in an embodiment of the present application.

[0058] Figure 3 This is a flowchart of the overall implementation of the distributed system fault intelligent processing method provided in the embodiment of the present application.

[0059] Figure 4 A flow chart of intelligent processing of distributed system faults provided in an embodiment of the present application.

[0060] Figure 5 This is a structural diagram of the composition of the distributed system fault intelligent processing device provided in an embodiment of the present application.

[0061] Figure 6 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0062] In order to make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the technical scheme in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0063] As used herein, the term “including” and its variations are open inclusions, ie, “including but not limited to”; the term “based on” means “based at least in part on”; and the term “one embodiment” means “at least one embodiment”.

[0064] It should be noted that the modifications of "one" and "plurality" mentioned in the present application are illustrative rather than restrictive, and those skilled in the art should understand that unless the context clearly indicates otherwise, it should be understood as "one or more".

[0065] Terminology explanation:

[0066] Heartbeat mechanism: a common fault detection method in distributed systems.

[0067] Adaptive threshold adjustment: When the heartbeat signal is detected to deviate from the normal range, the threshold is dynamically adjusted according to the degree of deviation.

[0068] Isolation Forest: An unsupervised learning algorithm for anomaly detection. The basic idea is to quickly isolate abnormal data points by building multiple isolation trees. Abnormal data points are easier to segment due to their uniqueness, so the path length is shorter.

[0069] Signature code: is a term widely used in many fields such as computer science, software engineering, and information security. It refers to a piece of code, string, or digital sequence used to identify a specific function, attribute, or pattern.

[0070] Figure 1 The figure shows the overall design of the method of this embodiment. The method receives a heartbeat packet and checks whether the heartbeat packet is legal, performs heartbeat anomaly detection on a legal heartbeat packet, determines whether the heartbeat is abnormal, identifies the abnormal task (specific fault) based on the information carried by the abnormal heartbeat, and then performs compensation operations on the abnormal task.

[0071] The specific implementation process is as follows:

[0072] (I) Heartbeat packet design. In a distributed system, a node failure may cause the performance of the entire system to degrade or even crash. To prevent this from happening, there must be a mechanism that can respond quickly when a failure occurs. The heartbeat mechanism is one of the commonly used mechanisms. In order to more conveniently apply the heartbeat mechanism, the embodiment of the present application designs a unique heartbeat packet structure, as shown in the following figure. Figure 2 shown.

[0073] (1) The structure of the heartbeat packet of this application includes the following parts:

[0074] Machine IP: identifies the location of the node in the network.

[0075] Service name: describes the type of service provided by the node, that is, the type of tasks that the node can handle.

[0076] Startup time: records the timestamp when the node is started, used to track the life cycle of the node.

[0077] Distributed ID: A globally unique identifier assigned to a node.

[0078] Current time: The timestamp when the heartbeat packet is sent, used to determine whether the heartbeat has timed out.

[0079] Verification code: Calculated based on the first four feature codes and the current time to prevent the heartbeat packet from being forged.

[0080] Message body: conveys necessary characteristic information that helps with fault detection, including but not limited to CPU utilization, memory usage, network latency, and the operating indicators of the service itself.

[0081] (2) Rules for generating feature codes and check codes:

[0082] The signature code consists of the node machine's IP, service name, startup time, and distributed ID. The checksum is calculated by using a hash function to calculate the signature code and current time to ensure the authenticity and integrity of the heartbeat packet.

[0083] (II) Heartbeat anomaly detection algorithm. In order to accurately detect heartbeat anomalies, the present application adopts an adaptive threshold adjustment strategy based on the isolation forest algorithm. In addition, the present application can also use pattern recognition technology to replace the isolation forest algorithm to achieve adaptive threshold adjustment.

[0084] (1) Principle of Isolation Forest Algorithm: Isolation Forest is an unsupervised learning algorithm used for anomaly detection. Its basic idea is to quickly isolate abnormal data points by building multiple isolation trees. Abnormal data points are easier to segment due to their uniqueness, so the path length is shorter.

[0085] (2) Adaptive threshold adjustment: Dynamically adjust the detection criteria of the heartbeat signal to adapt to different working conditions. The normal heartbeat pattern is learned through the isolation forest algorithm, and the threshold is adjusted accordingly. When the heartbeat signal is detected to deviate from the normal range, the threshold is dynamically adjusted according to the degree of deviation.

[0086] (III) Fault detection and compensation mechanism.

[0087] (1) Fault detection: When the heartbeat packet analysis finds that the heartbeat of a certain node is abnormal, the globally unique signature code is used to determine which IP and service has failed.

[0088] (2) Compensation mechanism: When a fault is detected, a compensation message containing the fault node feature code is generated and sent to the compensation queue. After the application service obtains the compensation message from the compensation queue, it queries the progress of the task currently being processed by the faulty service, and then selects the appropriate position (breakpoint position) to continue executing the unfinished task based on the task progress information to ensure the integrity of task processing.

[0089] Figure 3 The figure shows the overall implementation process of the distributed system fault intelligent processing method of this embodiment, which includes the following steps:

[0090] Start the application service and generate a heartbeat packet, wherein the heartbeat packet contains a globally unique feature code;

[0091] Start the heartbeat service and send heartbeat packets to the heartbeat queue at the set time interval;

[0092] The monitoring service receives and saves the data of the heartbeat queue;

[0093] Continuously collect heartbeat queue data from each node, preprocess the collected heartbeat data, extract features related to fault detection from the preprocessed data, and then use the isolation forest model for model training;

[0094] Adopting the adaptive threshold adjustment method based on isolation forest algorithm to analyze the data in the heartbeat packet;

[0095] Compare the heartbeats according to the threshold. When a node's heartbeat is abnormal, determine the fault location based on the globally unique signature code.

[0096] Generate a compensation message containing the faulty node feature code and send it to the compensation queue. After the application service obtains the compensation message from the compensation queue, it queries the progress of the task currently being processed by the faulty node.

[0097] According to the task progress information of the faulty node, select the breakpoint location to continue executing the unfinished task.

[0098] In order to better understand the technical solution of the present application, the intelligent processing process of distributed system faults is described more clearly, and the following scenario is further used as an example for explanation. The specific process is as follows: Figure 4 shown.

[0099] The simulation sets up a distributed system environment, including application service (N working nodes), queue service (a virtual machine), and monitoring service (N monitoring nodes).

[0100] Step 1: After the application service is started, a globally unique signature is generated as part of the heartbeat packet.

[0101] Step 2: Start the heartbeat service and send heartbeat packets to the heartbeat queue at a certain time interval.

[0102] In step 3, the monitoring service subscribes to the messages in the heartbeat queue and saves them locally after receiving them, which are then provided to the heartbeat detection model for analysis.

[0103] Step 4: Preprocess the heartbeat data. Continuously collect heartbeat signals and related information from each node in the distributed system, and obtain CPU utilization, memory usage, network latency, and the operating indicators of the service itself from the message body of the heartbeat packet. These data will be used for subsequent feature extraction and model training. The preprocessing stage also includes operations such as removing noise, filling missing values, and standardizing values ​​to ensure the quality of input data.

[0104] Step 5, feature extraction, extracts features that are helpful for fault detection from the collected data and uses the labeled dataset to guide the model to learn the ability to distinguish between normal and abnormal heartbeat signals.

[0105] Step 6: In order to accurately detect heartbeat anomalies, an adaptive threshold adjustment technique based on the isolation forest algorithm is used. The analyzed data comes from the message body of the heartbeat packet.

[0106] The construction process of the isolation tree includes the following steps:

[0107] (1) Randomly select a feature: Randomly select a feature from all features.

[0108] (2) Randomly select a split point: Randomly select a split point on the selected feature, usually any value between the minimum and maximum values ​​of the feature.

[0109] (3) Split the data set: Divide the data into two parts according to this split point, and recursively repeat this process for the two parts of data until the stopping condition is met.

[0110] The construction of the orphan tree stops when one of the following conditions is met:

[0111] (1) When there is only one sample left in the data subset.

[0112] (2) When the pre-set maximum tree depth is reached.

[0113] Adaptive threshold adjustment dynamically adjusts the detection criteria of heartbeat signals to adapt to different working conditions. The normal heartbeat pattern is learned through the isolation forest algorithm and the threshold is adjusted accordingly. When the heartbeat signal is detected to deviate from the normal range, the threshold is dynamically adjusted according to the degree of deviation.

[0114] Step 7: Dynamic threshold adjustment calculation. Assuming that there is already a model that can recognize normal heartbeat patterns, you can set a baseline threshold and then adjust it according to the actual situation:

[0115] Baseline threshold: During the training phase, an initial threshold is determined based on the distribution of normal heartbeat signals.

[0116] Threshold update rule: When the heartbeat signal is detected to deviate from the normal range, the threshold is adjusted according to the degree of deviation. For example, if the heartbeat interval becomes unstable, the threshold can be appropriately relaxed; on the contrary, if the system load is low and the heartbeat interval is still large, the threshold may need to be tightened.

[0117] Adaptive learning: As time goes by, the model will continue to receive new data points. Through online learning or incremental learning, the model can gradually optimize its threshold adjustment strategy to make it more consistent with the current system status.

[0118] Step 8: Compare the heartbeat according to the threshold. When the heartbeat packet analysis finds that the heartbeat of a certain node is abnormal, determine which IP and service has failed based on the globally unique feature code.

[0119] Step 9: Send a compensation message. When a fault is detected, a compensation message containing the faulty node feature code is generated and sent to the compensation queue.

[0120] Step 10: The application service subscribes to the compensation message. After the application node obtains the compensation message from the compensation queue, it queries the progress of the task currently being processed by the faulty service.

[0121] Step 11: Perform compensation operations. According to the task progress information, select a suitable location to continue the unfinished task to ensure the integrity of task processing.

[0122] Figure 5 The figure shows the structure of the distributed system fault intelligent processing device of this embodiment, which includes:

[0123] Task startup module: used to start the application service, generate a heartbeat packet, and the heartbeat packet contains a globally unique feature code; start the heartbeat service, and send the heartbeat packet to the heartbeat queue at a set time interval;

[0124] Data processing module: used to receive and save the data of the heartbeat queue; continuously collect the data of the heartbeat queue from each node, pre-process the collected heartbeat data, extract the features related to fault detection from the pre-processed data, and then use the isolation forest model for model training;

[0125] Data analysis module: used to analyze the data in the heartbeat packet using an adaptive threshold adjustment method based on the isolation forest algorithm; compare the heartbeat according to the threshold, and when an abnormal heartbeat of a node is found, determine the fault location based on the globally unique feature code;

[0126] Task compensation module: used to generate a compensation message containing the characteristic code of the faulty node and send it to the compensation queue. After the application service obtains the compensation message from the compensation queue, it queries the progress of the task currently being processed by the faulty node; based on the task progress information of the faulty node, it selects the breakpoint location to continue executing the unfinished task.

[0127] When the above device is running, the steps of the distributed system fault intelligent processing method disclosed in this application are implemented.

[0128] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the device, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, program segment or a part of code, and the module, program segment or a part of code contains one or more executable instructions for realizing the specified logical function. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be realized by a special hardware-based system that performs the specified function or operation, or can be realized by a combination of special hardware and computer instructions.

[0129] like Figure 6 As shown, the embodiment of the present application also discloses an electronic device, including: a processor 310, a communication interface 320, a memory 330 for storing a computer program executable by the processor, and a communication bus 340. The processor 310, the communication interface 320, and the memory 330 communicate with each other through the communication bus 340. The processor 310 runs the executable computer program to implement the steps of the above-mentioned distributed system fault intelligent processing method.

[0130] It is understandable that, in addition to the memory and the processor, the electronic device may also include an input device such as a keyboard, an output device such as a display, and other communication modules. The input device, the output device, and other communication modules communicate with the processor via an I / O interface (i.e., an input / output interface).

[0131] Computer program code for performing the operations of the present application may be written in one or more programming languages ​​or a combination thereof, including, but not limited to, object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0132] Furthermore, the present application also discloses a computer-readable storage medium. When the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device can execute each step of the distributed system fault intelligent processing method disclosed in the present application.

[0133] In the context of the present application, a computer-readable storage medium may be a tangible medium, more specific examples of which would include a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0134] In particular, according to an embodiment of the present application, the process described in the flowchart can be implemented as a computer software program. For example, an embodiment of the present application includes a computer program product, which includes a computer program carried on a non-transitory computer-readable medium, and the computer program contains program code for executing the distributed system fault intelligent processing method disclosed in the present application. When the computer program is executed by the processing device, the above functions defined in the method of the embodiment of the present application are executed.

[0135] Although several specific implementation details are included in the above discussion, these should not be interpreted as limiting the scope of the present application. The above description is only a preferred embodiment of the present application and an explanation of the technical principles used. Those skilled in the art should understand that the scope of disclosure involved in the present application is not limited to the technical solutions formed by a specific combination of the above technical features, but also should cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above public concept.

[0136] Those skilled in the art should also understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein; and these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A distributed system fault intelligent processing method, characterized in that: The method comprises: Start the application service and generate a heartbeat packet, wherein the heartbeat packet contains a globally unique feature code; Start the heartbeat service and send heartbeat packets to the heartbeat queue at the set time interval; The monitoring service receives and saves the data of the heartbeat queue; Continuously collect heartbeat queue data from each node, preprocess the collected heartbeat data, extract features related to fault detection from the preprocessed data, and then use the isolation forest model for model training; Adopting the adaptive threshold adjustment method based on isolation forest algorithm to analyze the data in the heartbeat packet; Compare the heartbeats according to the threshold. When a node's heartbeat is abnormal, determine the fault location based on the globally unique signature code. Generate a compensation message containing the faulty node feature code and send it to the compensation queue. After the application service obtains the compensation message from the compensation queue, it queries the progress of the task currently being processed by the faulty node. According to the task progress information of the faulty node, select the breakpoint location to continue executing the unfinished task.

2. The method according to claim 1, characterized in that The heartbeat packet contains the following information: Machine IP: used to identify the location of the node in the network; Service name: used to describe the type of service provided by the node; Startup time: records the timestamp of node startup, used to track the life cycle of the node; Distributed ID: A globally unique identifier assigned to a node; Current time: the timestamp when the heartbeat packet is sent, used to determine whether the heartbeat has timed out; Verification code: calculated by hash function based on the feature code and current time, used to prevent the heartbeat packet from being forged; Message body: used to transmit characteristic information related to fault detection.

3. The method according to claim 2, characterized in that The machine IP, service name, startup time and distributed ID constitute a globally unique feature code; The message body includes the following information: CPU utilization, memory usage, network latency, and the operating indicators of the service itself.

4. The method according to claim 1, characterized in that: The preprocessing of the collected heartbeat data includes: removing noise, filling missing values ​​and standardizing values.

5. The method according to claim 1, characterized in that The construction process of the isolated tree in the isolation forest algorithm is as follows: (1) Randomly select a feature: randomly select a feature from all features; (2) Randomly select a split point: randomly select a split point on the selected feature, and the split point is any value between the minimum and maximum values ​​of the feature; (3) Segmenting the data set: The feature data is divided into two parts according to the segmentation point, and the above segmentation process is recursively repeated for the two parts of data until any of the following stopping conditions is met: Stop condition a: When there is only one sample left in the data subset; Stop condition b: When the preset maximum tree depth is reached.

6. The method according to claim 1, characterized in that The adaptive threshold adjustment method includes: learning a normal heartbeat pattern through an isolation forest algorithm and adjusting the threshold accordingly; when a heartbeat signal is detected to deviate from a normal range, dynamically adjusting the detection threshold of the heartbeat signal according to the degree of deviation.

7. The method according to claim 1, characterized in that The method also includes a step of dynamic threshold adjustment calculation, including: During the training phase, a baseline threshold is set according to the distribution of normal heartbeat signals; When the heartbeat signal is detected to deviate from the normal range, the threshold is adjusted according to the degree of deviation; Continuously optimize the threshold adjustment strategy through online learning or incremental learning.

8. A distributed system fault intelligent processing device, characterized in that: The device comprises: Task startup module: used to start the application service, generate a heartbeat packet, and the heartbeat packet contains a globally unique feature code; start the heartbeat service, and send the heartbeat packet to the heartbeat queue at a set time interval; Data processing module: used to receive and save the data of the heartbeat queue; continuously collect the data of the heartbeat queue from each node, pre-process the collected heartbeat data, extract the features related to fault detection from the pre-processed data, and then use the isolation forest model for model training; Data analysis module: used to analyze the data in the heartbeat packet using an adaptive threshold adjustment method based on the isolation forest algorithm; compare the heartbeat according to the threshold, and when an abnormal heartbeat of a node is found, determine the fault location based on the globally unique feature code; Task compensation module: used to generate a compensation message containing the characteristic code of the faulty node and send it to the compensation queue. After the application service obtains the compensation message from the compensation queue, it queries the progress of the task currently being processed by the faulty node; based on the task progress information of the faulty node, it selects the breakpoint location to continue executing the unfinished task.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the distributed system fault intelligent processing method according to any one of claims 1 to 7 are implemented.

10. An electronic device, characterized in that: include: Memory and processor; Memory: used to store computer programs; Processor: used to execute the computer program to implement the steps of the distributed system fault intelligent processing method as described in any one of claims 1-7.

Images