Cluster fault processing method and related equipment

Abstract

The invention provides a cluster fault processing method and related equipment. The cluster fault processing method comprises the following steps: acquiring data traffic in a cluster in real time; determining a first node with a fault in the cluster according to the data traffic; performing isolation operation on the first node according to the fault level of the first node, and determining a to-be-processed task of the first node; distributing the to-be-processed task to a second node in the cluster based on a preset load balancing strategy, wherein the second node indicates other nodes except the first node in the cluster; determining corresponding task data according to the to-be-processed task; and distributing the task data to the second node according to the storage priority of the second node. According to the invention, the fault processing accuracy of the nodes in the cluster can be improved.

Description

Technical Field

[0001] This application relates to the field of equipment operation and maintenance technology, specifically to the field of cluster fault handling technology, and particularly to a cluster fault handling method and related equipment. Background Technology

[0002] With the development of industrial technology, in order to improve the stability of communication clusters during operation, enterprises and organizations are increasingly inclined to monitor and maintain the faults of nodes in the cluster in real time to ensure that the cluster failure rate is reduced and the stability of cluster operation is guaranteed. At present, the operating status of each node in the cluster is usually monitored by the cluster's operation logs. However, due to the time lag of the operation logs or monitoring data obtained in real time, it is impossible to predict the faulty nodes in the cluster in a timely manner, and it is impossible to accurately determine the time of failure. This usually causes cluster downtime, resulting in low accuracy in cluster fault handling. Summary of the Invention

[0003] In view of the above, it is necessary to propose a cluster fault handling method and related equipment to solve the technical problem of low accuracy in cluster fault handling. The related equipment includes a cluster fault handling device and electronic equipment.

[0004] This application provides a cluster fault handling method, the method comprising: acquiring data traffic in the cluster in real time; determining a first node in the cluster that has a fault based on the data traffic; isolating the first node according to the fault level of the first node, and determining the tasks to be processed on the first node; allocating the tasks to be processed to a second node in the cluster based on a preset load balancing strategy, wherein the second node indicates other nodes in the cluster besides the first node; determining the corresponding task data according to the tasks to be processed; and allocating the task data to the second node according to the storage priority of the second node.

[0005] In some embodiments, determining the first faulty node in the cluster based on the data traffic includes: determining the computing resource utilization, communication resource utilization, request response latency, and task throughput within a preset time period for any node in the cluster based on the data traffic; determining the performance indicators of any node based on the computing resource utilization, communication resource utilization, request response latency, and task throughput within a preset time period for any node; determining the heartbeat data, bit error rate, and task retry count of any node based on the data traffic; determining the health indicators of any node based on the heartbeat data, bit error rate, and task retry count of any node; determining the packet loss rate and cache response rate of the data center where the node is located based on the data traffic; determining the correlation indicators of any node based on the packet loss rate and cache response rate of the data center where the node is located; and determining the node as the first node if the performance indicators, the health indicators, and the correlation indicators meet a preset first condition.

[0006] In some embodiments, the method further includes determining the fault level of the first node, wherein determining the fault level of the first node includes: determining the performance indicators, health indicators, and correlation indicators of the first node within the first time period based on the data traffic of the cluster within the preset first time period; and determining the fault level of the first node using a pre-trained fault level prediction model based on the performance indicators, the health indicators, and the correlation indicators.

[0007] In some embodiments, the method further includes training the fault level prediction model, wherein training the fault level prediction model includes: determining, based on the data traffic of the cluster within a preset second time period, performance metrics, health metrics, and correlation metrics of any node in the cluster within the second time period; determining training samples based on the performance metrics, health metrics, and correlation metrics of any node in the cluster within the second time period; determining prediction data corresponding to the training samples using a pre-built initial model based on the training samples; the prediction data includes the prediction level corresponding to the training samples and the prediction probability corresponding to the prediction level; and determining, based on the data traffic of any node in the cluster within the second time period, performance metrics, health metrics, and correlation metrics of any node in the cluster within the second time period. The weight data of the training samples is determined by the performance metrics, health metrics, and correlation metrics of a node within the second time period; the weight data includes a first weight of the performance metrics, a second weight of the health metrics, and a third weight of the correlation metrics; the loss value of the initial model is determined based on the weight data, the prediction data, and the label data corresponding to the training samples; the label data is used to indicate the fault level corresponding to any node within the second time period; the initial model is updated based on the loss value, and the update of the initial model is stopped when the loss value meets a preset termination condition, thus obtaining a level prediction model trained to a convergent state.

[0008] In some embodiments, determining the weight data of the training sample based on the performance indicators, health indicators, and correlation indicators of any node within the second time period includes: determining the first weight based on the variance and mean of the performance indicators within the second time period; determining the second weight based on the variance and mean of the health indicators within the second time period; and determining the third weight of the training sample based on the variance and mean of the correlation indicators within the second time period.

[0009] In some embodiments, determining the loss value of the initial model based on the weight data, the prediction data, and the label data corresponding to the training samples includes: ; in, This represents the loss value of the initial model; This represents the first weight in the weighted data. This represents the second weight in the weighted data. Represents the third weight in the weighted data; i represents the index of the prediction level in the prediction data, and n represents the number of prediction levels in the prediction data; This represents the probability corresponding to the i-th fault level in the label data; This represents the prediction probability corresponding to the i-th prediction level in the prediction data.

[0010] In some embodiments, isolating the first node according to its fault level includes: stopping the allocation of data processing tasks to the first node when the fault level meets preset maintenance conditions; and sending a shutdown command to the first node after the data processing tasks in the first node have been completed.

[0011] In some embodiments, allocating the tasks to be processed to the second nodes in the cluster based on a preset load balancing strategy includes: determining the computing resource utilization rate, communication resource utilization rate, request response latency, and task throughput of any second node within a preset first time period based on the data traffic; determining the polling priority of the any second node based on the computing resource utilization rate, communication resource utilization rate, request response latency, and task throughput; and allocating the tasks to be processed to the second nodes in the cluster in descending order of the polling priority.

[0012] In some embodiments, determining the polling priority of any one of the second nodes based on the computing resource utilization, communication resource utilization, request response latency, and task throughput includes: ; Where R represents the polling priority of any second node; e represents a natural constant; s represents the request response latency; c represents the computing resource utilization rate; t represents the communication resource utilization rate; and num represents the task throughput.

[0013] In some embodiments, allocating the task data to the second node according to the storage priority of the second node includes: determining the heartbeat data, bit error rate, and task retry count of any second node within a preset first time period based on the data traffic; determining the storage priority of the any second node based on the heartbeat data, bit error rate, and task retry count; and allocating the task data to the second nodes in the cluster in descending order of the storage priority.

[0014] In some embodiments, determining the storage priority of any one of the second nodes based on the heartbeat data, bit error rate, and number of task retries includes: ; Where P represents the storage priority; e represents the natural constant; j represents the heartbeat data; k represents the bit error rate; and v represents the number of task retries.

[0015] This application embodiment also provides a cluster fault handling device, the device comprising: a monitoring module for acquiring data traffic in the cluster in real time; a determination module for determining a first node in the cluster that has a fault based on the data traffic; the determination module is further configured to isolate the first node according to the fault level of the first node and determine the tasks to be processed on the first node; an allocation module for allocating the tasks to be processed to a second node in the cluster based on a preset load balancing strategy, the second node indicating other nodes in the cluster besides the first node; the determination module is further configured to determine corresponding task data based on the tasks to be processed; the allocation module is further configured to allocate the task data to the second node according to the storage priority of the second node.

[0016] This application also provides an electronic device, which includes: a memory storing at least one instruction; and a processor executing the instructions stored in the memory to implement the cluster fault handling method.

[0017] This application also provides a computer-readable storage medium storing at least one instruction, which is executed by a processor in an electronic device to implement the cluster fault handling method.

[0018] As can be seen from the above technical solutions, the embodiments of this application identify faulty nodes by real-time data traffic, avoiding reliance on a single indicator threshold for predicting node faults and reducing false positives and false negatives. Comprehensive multi-dimensional indicators are used to determine faults, improving the accuracy of node fault identification. Furthermore, real-time data traffic collection ensures the timeliness of node fault identification, avoiding false negatives due to information lag. The mapping relationship between fault levels and isolation operations improves the accuracy of cluster fault handling. A node polling method based on node load and communication performance allocates tasks according to node hardware resources, ensuring that task allocation matches node processing capabilities, reducing the probability of task allocation errors, and avoiding cluster faults caused by task processing failures. Attached Figure Description

[0019] Figure 1 This is an application scenario diagram of a cluster fault handling method provided in an embodiment of this application.

[0020] Figure 2 This is a flowchart of a cluster fault handling method provided in an embodiment of this application.

[0021] Figure 3 This is a functional block diagram of a cluster fault handling device provided in an embodiment of this application.

[0022] Figure 4This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0023] To better understand the purpose, features, and advantages of this application, a detailed description of the application is provided below with reference to the accompanying drawings and specific embodiments. It should be noted that, unless otherwise specified, the embodiments and features described in the embodiments of this application can be combined with each other. Numerous specific details are set forth in the following description to provide a thorough understanding of this application; the described embodiments are only a part of the embodiments of this application, and not all of them.

[0024] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the stated features. In the description of this application, "a plurality of" means two or more, unless otherwise explicitly specified.

[0025] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. The term "and / or" as used herein includes any and all combinations of one or more of the associated listed items.

[0026] This application provides a cluster fault handling method that can be applied to one or more electronic devices. An electronic device is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions. Its hardware includes, but is not limited to, microprocessors, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), digital signal processors (DSPs), embedded devices, etc.

[0027] Electronic devices can be any electronic product that allows human-computer interaction with a customer, such as personal computers, tablets, smartphones, personal digital assistants (PDAs), game consoles, interactive network television (IPTV), smart wearable devices, etc.

[0028] Electronic devices may also include network devices and / or client devices. The network devices include, but are not limited to, a single network server, a server group consisting of multiple network servers, or a cloud based on cloud computing consisting of a large number of hosts or network servers.

[0029] The networks in which electronic devices are located include, but are not limited to, the Internet, wide area networks, metropolitan area networks, local area networks, and virtual private networks (VPNs).

[0030] like Figure 1 As shown, the cluster fault handling method provided in this application can be applied to an electronic device 100, which is communicatively connected to a server 200. The electronic device 100 is used to obtain data traffic in the cluster from the server 200 in real time; and to determine a first faulty node in the cluster based on the data traffic. The electronic device 100 is also used to isolate the first node according to its fault level, determine the tasks to be processed on the first node, and allocate the tasks to be processed to a second node in the cluster based on a preset load balancing strategy. The second node indicates other nodes in the cluster besides the first node. The electronic device 100 is also used to determine corresponding task data based on the tasks to be processed; and to allocate the task data to the second node according to its storage priority.

[0031] like Figure 2 The diagram shown is a flowchart of a cluster fault handling method according to an embodiment of this application. The order of the steps in the flowchart can be changed, and some steps can be omitted, depending on different requirements. The cluster fault handling method provided in this embodiment includes the following steps.

[0032] S20 acquires real-time data traffic within the cluster.

[0033] In one embodiment of this application, data traffic can reflect the operating status of nodes in the cluster, the task interaction process, and system stability. The information it contains may include one or more of the following: node interaction, task processing, system status, and network transmission.

[0034] In one embodiment of this application, node interaction information is used to reflect the communication status between nodes and between nodes and the outside world. Specifically, node interaction information includes the number of input / output requests of a node, used to indicate the number of external requests (such as API calls, data queries) received by each node per unit time and the number of internal requests (such as task collaboration, data synchronization requests) sent to other nodes; node interaction information also includes connection status data, used to indicate the number of active connections of a node, the frequency of connection establishment / disconnection, the number of connection timeouts, and the stability of communication links with other nodes (such as link jitter frequency); node interaction information also includes data transmission volume, used to indicate the size (number of bytes) of data packets received and sent by a node per unit time, including raw business data, node heartbeat packets (used for cluster liveness detection), metadata (such as task configuration information), etc.

[0035] In one embodiment of this application, task processing information can reflect the node's execution process of tasks. Specifically, task processing information includes task throughput, used to indicate the total number of tasks processed by the node per unit time (such as computation tasks and storage tasks), and the proportion of different types of tasks (such as real-time computation tasks and offline analysis tasks); task processing information also includes task response performance, used to indicate the time taken (response time) from receipt to completion of a single task, the latency distribution of task processing (such as P95 / P99 latency), and the amount of intermediate data generated during task execution; task processing information also includes task status logs, used to indicate the lifecycle status records of tasks (such as "pending", "in execution", "success", "failure"), and the error codes (such as insufficient resources, incorrect data format) and descriptions of failure reasons for failed tasks.

[0036] In one embodiment of this application, system status information is used to reflect the resource consumption and health of nodes. Specifically, the system status information includes resource usage data, which indicates the node's CPU utilization, memory utilization, disk I / O rate, network bandwidth utilization, etc., when processing traffic, directly related to traffic processing capacity; the system status information also includes abnormal event records, which indicate the error logs (such as memory overflow, disk overflow), warning logs (such as resource utilization approaching the threshold) generated by the node, and system-level events (such as process crash, restart records); the system status information also includes load fluctuation trends, which indicate the change curve of node load (such as request volume, resource usage) per unit time, including the time, duration and frequency of sudden peaks.

[0037] In one embodiment of this application, network transmission information is used to reflect the transmission quality of data at the network layer. Specifically, network transmission information includes data packet characteristics, used to indicate the transmission protocol type (such as TCP, UDP), data packet loss rate, number of retransmissions, and number of data packet verification failures (reflecting data integrity); network transmission information also includes network latency data, used to indicate the round-trip time (RTT) of data transmission between nodes, the number of routing hops for cross-node communication, and the differences in transmission efficiency between different network partitions (such as within a data center or across data centers).

[0038] S21, determine the first faulty node in the cluster based on the data traffic.

[0039] In one embodiment of this application, determining the first faulty node in the cluster based on the data traffic includes: determining the computing resource utilization rate, communication resource utilization rate, request response latency, and task throughput within a preset time for any node in the cluster based on the data traffic; determining the performance indicators of any node based on the computing resource utilization rate, communication resource utilization rate, request response latency, and task throughput within a preset time for any node; determining the heartbeat data, bit error rate, and task retry count of any node based on the data traffic; determining the health indicators of any node based on the heartbeat data, bit error rate, and task retry count of any node; determining the packet loss rate and cache response rate of the data center where the node is located based on the data traffic; determining the correlation indicators of any node based on the packet loss rate and cache response rate of the data center where the node is located; and determining the any node as the first node if the performance indicators, the health indicators, and the correlation indicators meet a preset first condition.

[0040] The first condition may include any one or more of the following combinations: performance indicators are less than or equal to preset performance thresholds; health indicators are less than or equal to preset health thresholds; and correlation indicators are less than or equal to preset correlation thresholds.

[0041] In one embodiment of this application, the method further includes determining the fault level of the first node. Determining the fault level of the first node includes: determining the performance indicators, health indicators, and correlation indicators of the first node within the first time period based on the data traffic of the cluster within the preset first time period; and determining the fault level of the first node using a pre-trained fault level prediction model based on the performance indicators, the health indicators, and the correlation indicators.

[0042] In one embodiment of this application, the method further includes training the fault level prediction model. Training the fault level prediction model includes determining, based on the data traffic of the cluster within a preset second time period, the performance indicators, health indicators, and correlation indicators of any node in the cluster within the second time period; determining training samples based on the performance indicators, health indicators, and correlation indicators of any node in the cluster within the second time period; determining prediction data corresponding to the training samples using a pre-constructed initial model based on the training samples; the prediction data includes the prediction level corresponding to the training samples and the prediction probability corresponding to the prediction level; and determining the prediction data based on the data traffic of any node in the cluster within the second time period. The weight data of the training samples is determined by the performance metrics, health metrics, and correlation metrics of a node within the second time period; the weight data includes a first weight of the performance metrics, a second weight of the health metrics, and a third weight of the correlation metrics; the loss value of the initial model is determined based on the weight data, the prediction data, and the label data corresponding to the training samples; the label data is used to indicate the fault level corresponding to any node within the second time period; the initial model is updated based on the loss value, and the update of the initial model is stopped when the loss value meets a preset termination condition, thus obtaining a level prediction model trained to a convergent state.

[0043] In one embodiment of this application, determining the weight data of the training sample based on the performance indicators, health indicators, and correlation indicators of any node within the second time period includes: determining the first weight based on the variance and mean of the performance indicators within the second time period; determining the second weight based on the variance and mean of the health indicators within the second time period; and determining the third weight of the training sample based on the variance and mean of the correlation indicators within the second time period. Specifically, the first weight may be the sum of the variance and mean of the performance indicators within the second time period; the second weight may be the sum of the variance and mean of the health indicators within the second time period; and the third weight may be the sum of the variance and mean of the correlation indicators within the second time period. The higher the first weight, the greater the fluctuation of the performance metric within the second time period, which in turn leads to greater fluctuation in the performance of the corresponding node, resulting in a higher loss value when training the initial model based on the training samples of that node. Similarly, the higher the second weight, the greater the fluctuation of the health metric within the second time period, which in turn leads to greater fluctuation in the health of the corresponding node, resulting in a higher loss value when training the initial model based on the training samples of that node. Finally, the higher the third weight, the greater the fluctuation of the correlation metric within the second time period, which in turn leads to greater fluctuation in the packet loss rate and cache response rate of the data center where the corresponding node is located, resulting in a higher loss value when training the initial model based on the training samples of that node.

[0044] In one embodiment of this application, determining the loss value of the initial model based on the weight data, the prediction data, and the label data corresponding to the training samples includes: ; in, This represents the loss value of the initial model; This represents the first weight in the weighted data. This represents the second weight in the weighted data. Represents the third weight in the weighted data; i represents the index of the prediction level in the prediction data, and n represents the number of prediction levels in the prediction data; This represents the probability corresponding to the i-th fault level in the label data; This represents the prediction probability corresponding to the i-th prediction level in the prediction data. Higher first, second, and third weights indicate greater volatility in the training samples, resulting in lower confidence levels and consequently lower accuracy when training the initial model based on these samples. Therefore, the loss value of the initial model can be adjusted based on the weight data. Higher first, second, and third weights increase the loss value, ensuring the initial model can avoid information from highly volatile training samples, thereby improving the performance of the resulting generative model.

[0045] S22, Isolate the first node according to the fault level of the first node, and determine the tasks to be processed for the first node.

[0046] In one embodiment of this application, the isolation operation of the first node according to the fault level of the first node includes: stopping the allocation of data processing tasks to the first node when the fault level meets the preset maintenance conditions; and sending a shutdown command to the first node after the data processing tasks in the first node have been completed.

[0047] The preset maintenance conditions may include: the fault level being a preset level threshold. For example, the fault levels, from low to high, can be low, medium, and high. If the level threshold is medium, and the fault level is also medium, then the fault level meets the preset maintenance conditions, and the allocation of data processing tasks to the first node can be stopped; after the data processing tasks in the first node are completed, a shutdown command is sent to the first node.

[0048] S23, the task to be processed is allocated to the second node in the cluster based on a preset load balancing strategy, and the second node indicates other nodes in the cluster besides the first node.

[0049] In one embodiment of this application, the step of allocating the task to be processed to a second node in the cluster based on a preset load balancing strategy includes: determining the computing resource utilization rate, communication resource utilization rate, request response latency, and task throughput of any second node within a preset first time period based on the data traffic; determining the polling priority of the any second node based on the computing resource utilization rate, communication resource utilization rate, request response latency, and task throughput; and allocating the task to be processed to the second nodes in the cluster in descending order of the polling priority.

[0050] In one embodiment of this application, determining the polling priority of any one of the second nodes based on the computing resource utilization rate, communication resource utilization rate, request response latency, and task throughput includes: ; Where R represents the polling priority of any second node; e represents a natural constant; s represents the request response latency; c represents the computing resource utilization rate; t represents the communication resource utilization rate; and num represents the task throughput.

[0051] S24, determine the corresponding task data based on the task to be processed.

[0052] In one embodiment of this application, task data can indicate a set of information about the tasks running in a node throughout their entire lifecycle (creation, execution, scheduling, migration, archiving).

[0053] Specifically, task data can include task identifiers and basic attribute data, used to identify tasks within the cluster and define their core characteristics. The task identifier can be a globally unique identifier generated by electronic devices in the cluster, used to track the flow of tasks across the entire cluster; the basic attribute data can include the task's business attributes and priority level, used to indicate the order of task scheduling and migration priority in case of node failure.

[0054] Specifically, task data can also include task input and output data, which serve as the core basis for task execution. Task input data can indicate the storage location, data format, data size, and retrieval method of the raw data required by the task; task output data can indicate the storage target, output format, expected output volume, and whether subsequent tasks need to be triggered for the task processing results.

[0055] Specifically, task data can also include task processing data, used to record the dynamic status of task execution. The current processing status of a task can include "awaiting resource allocation," "loading data," "executing (30% progress)," "failed (waiting for retry)," and "completed." Resource consumption records for a task can include data such as the amount of computing resources used during task execution, peak memory usage, disk read / write speed, and network bandwidth usage percentage. This data is used to assess the task's resource requirements on nodes and provides a basis for load balancing across nodes in the cluster.

[0056] Specifically, task data can also include task dependency and association data to maintain logical consistency between tasks. Task dependency data indicates the identifiers (e.g., task IDs) and dependency conditions (e.g., execution only if task A succeeds and outputs more than or equal to 1GB) of the preceding tasks that currently exist in a node, preventing tasks from being incorrectly scheduled when dependencies are not met. Association data for each node in the cluster can indicate the identifiers (e.g., node IDs) of the nodes currently running the task, as well as the identifiers of historically running nodes, enabling analysis of differences in task execution efficiency across different nodes and the dedicated resource nodes required by the task.

[0057] Specifically, task data can also include task security and permission data, used to control permissions for tasks involving sensitive business operations, ensuring the compliance of task processing. Permission data can indicate whether the task initiator has permission to access the input data source, the encryption method of the output data, and the storage location of the decryption key. Task security data can include the executor, operation time, and operation content of tasks such as creation, modification, suspension, and termination, and can be used to trace task anomalies caused by unauthorized or erroneous operations.

[0058] S25, the task data is allocated to the second node according to the storage priority of the second node.

[0059] In one embodiment of this application, the step of allocating the task data to the second node according to the storage priority of the second node includes: determining the heartbeat data, bit error rate, and task retry count of any second node within a preset first time period based on the data traffic; determining the storage priority of the any second node based on the heartbeat data, bit error rate, and task retry count; and allocating the task data to the second nodes in the cluster according to the storage priority from high to low.

[0060] In one embodiment of this application, determining the storage priority of any one of the second nodes based on the heartbeat data, bit error rate, and number of task retries includes: ; Where P represents the storage priority; e represents the natural constant; j represents the heartbeat data; k represents the bit error rate; and v represents the number of task retries. Specifically, a higher heartbeat data rate at the second node indicates a higher number of times the second node responds to communication requests, signifying normal communication operation and thus a higher storage priority. A higher bit error rate at the second node indicates a higher proportion of errors in the total output data, thus a lower storage priority. A higher number of retries at the second node indicates a higher probability of task restarts or timeouts during data processing, also resulting in a lower storage priority.

[0061] In one embodiment of this application, task data can be allocated to the second node in the cluster according to the storage priority from high to low, thereby ensuring that the second node with higher storage priority stores the task data in the faulty node first, which can ensure the integrity and security of task data storage.

[0062] As can be seen from the above technical solutions, the embodiments of this application identify faulty nodes by real-time data traffic, avoiding reliance on a single indicator threshold for predicting node faults and reducing false positives and false negatives. Comprehensive multi-dimensional indicators are used to determine faults, improving the accuracy of node fault identification. Furthermore, real-time data traffic collection ensures the timeliness of node fault identification, avoiding false negatives due to information lag. The mapping relationship between fault levels and isolation operations improves the accuracy of cluster fault handling. A node polling method based on node load and communication performance allocates tasks according to node hardware resources, ensuring that task allocation matches node processing capabilities, reducing the probability of task allocation errors, and avoiding cluster faults caused by task processing failures.

[0063] Please see Figure 3 , Figure 3 This is a functional block diagram of a cluster fault handling device according to an embodiment of this application. A cluster fault handling device 31 includes a monitoring module 310, a determination module 311, and an allocation module 312. The module / unit referred to in this application refers to a series of computer-readable instruction segments that can be executed by the processor 13 and perform a fixed function, and which are stored in the memory 12. In this embodiment, the functions of each module / unit will be described in detail in subsequent embodiments.

[0064] The monitoring module 310 is used to acquire data traffic in the cluster in real time.

[0065] The determining module 311 is used to determine the first faulty node in the cluster based on the data traffic.

[0066] The determining module 311 is further configured to isolate the first node according to the fault level of the first node, and determine the tasks to be processed by the first node.

[0067] The allocation module 312 is used to allocate the task to be processed to a second node in the cluster based on a preset load balancing strategy, wherein the second node indicates other nodes in the cluster besides the first node.

[0068] The determining module 311 is further configured to determine the corresponding task data based on the task to be processed.

[0069] The allocation module 312 is further configured to allocate the task data to the second node according to the storage priority of the second node.

[0070] Please see Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. The electronic device 100 includes a memory 12 and a processor 13. The memory 12 is used to store computer-readable instructions, and the processor 13 executes the computer-readable instructions stored in the memory to implement the cluster fault handling method described in any of the above embodiments.

[0071] In one embodiment of this application, the electronic device 100 further includes a bus and a computer program, such as a cluster fault handler, stored in the memory 12 and executable on the processor 13.

[0072] Figure 4 Only an electronic device 100 with memory 12 and processor 13 is shown; those skilled in the art will understand that... Figure 4 The structure shown does not constitute a limitation on the electronic device 100, and may include fewer or more components than shown, or combine certain components, or have different component arrangements.

[0073] Combination Figure 2 The memory 12 in the electronic device 100 stores a plurality of computer-readable instructions to implement a cluster fault handling method. The processor 13 can execute the plurality of instructions to achieve: real-time acquisition of data traffic in the cluster; determining a first node in the cluster that has a fault based on the data traffic; isolating the first node according to the fault level of the first node and determining the task to be processed for the first node; allocating the task to be processed to a second node in the cluster based on a preset load balancing strategy, wherein the second node indicates other nodes in the cluster besides the first node; determining the corresponding task data according to the task to be processed; and allocating the task data to the second node according to the storage priority of the second node.

[0074] Specifically, the processor 13's implementation method for the above instructions can be found in [reference needed]. Figure 2 The descriptions of the relevant steps in the corresponding embodiments are not repeated here.

[0075] Those skilled in the art will understand that the schematic diagram is merely an example of the electronic device 100 and does not constitute a limitation on the electronic device 100. The electronic device 100 may be a bus-type structure or a star-type structure. The electronic device 100 may also include more or fewer other hardware or software than shown in the diagram, or different component arrangements. For example, the electronic device 100 may also include input / output devices, network access devices, etc.

[0076] It should be noted that electronic device 100 is only an example. Other existing or future electronic products that are suitable for this application should also be included within the scope of protection of this application and are incorporated herein by reference.

[0077] The memory 12 includes at least one type of readable storage medium, which can be non-volatile or volatile. The readable storage medium includes flash memory, portable hard drives, multimedia cards, card-type memory (e.g., SD or DX memory), magnetic storage, magnetic disks, optical disks, etc. In some embodiments, the memory 12 can be an internal storage unit of the electronic device 100, such as a portable hard drive of the electronic device 100. In other embodiments, the memory 12 can be an external storage device of the electronic device 100, such as a plug-in portable hard drive, smart media card (SMC), secure digital (SD) card, flash card, etc., equipped on the electronic device 100. The memory 12 can be used not only to store application software and various types of data installed on the electronic device 100, such as the code of a cluster fault handling program, but also to temporarily store data that has been output or will be output.

[0078] In some embodiments, the processor 13 may be composed of integrated circuits, such as a single packaged integrated circuit or multiple integrated circuits packaged with the same or different functions, including combinations of one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and various control chips. The processor 13 is the control unit of the electronic device 100, connecting various components of the electronic device 100 via various interfaces and lines. It executes programs or modules stored in the memory 12 (e.g., executing cluster fault handling programs) and calls data stored in the memory 12 to perform various functions of the electronic device 100 and process data.

[0079] The processor 13 executes the operating system of the electronic device 100 and various installed applications. The processor 13 executes the applications to implement the steps in the various cluster fault handling method embodiments described above, for example... Figure 2 The steps are shown.

[0080] For example, the computer program may be divided into one or more modules / units, which are stored in the memory 12 and executed by the processor 13 to complete this application. The one or more modules / units may be a series of computer-readable instruction segments capable of performing a specific function, which describe the execution process of the computer program in the electronic device 100. For example, the computer program may be divided into a monitoring module 310, a determination module 311, and an allocation module 312.

[0081] The integrated unit implemented as a software functional module described above can be stored in a computer-readable storage medium. This software functional module, stored in a storage medium, includes several instructions to cause a computer device (which may be a personal computer, computer equipment, or network device, etc.) or processor to execute portions of the cluster fault handling methods described in the various embodiments of this application.

[0082] If the modules / units integrated in the electronic device 100 are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments can also be implemented by a computer program instructing related hardware devices. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above.

[0083] The computer program includes computer program code, which may be in the form of source code, object code, executable file, or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording media, USB flash drive, portable hard drive, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory, and other memory.

[0084] Furthermore, the computer-readable storage medium may primarily include a stored program area and a stored data area, wherein the stored program area may store the operating system, an application program required for at least one function, etc.; and the stored data area may store data created based on the use of blockchain nodes, etc.

[0085] The bus can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This bus can be divided into address bus, data bus, control bus, etc. For ease of representation, in... Figure 4 The symbol is represented by only one arrow, but this does not indicate that there is only one bus or one type of bus. The bus is configured to enable communication between the memory 12 and at least one processor 13, etc.

[0086] This application also provides a computer-readable storage medium (not shown), which stores computer-readable instructions that are executed by a processor in an electronic device to implement the cluster fault handling method described in any of the above embodiments.

[0087] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and other division methods may be used in actual implementation.

[0088] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0089] Furthermore, the functional modules in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or in the form of hardware plus software functional modules.

[0090] Furthermore, it is clear that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. Multiple units or devices described in the specification may also be implemented by a single unit or device through software or hardware. Terms such as "first," "second," etc., are used to indicate names and do not indicate any specific order.

[0091] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application and are not intended to limit it. Although this application has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of this application without departing from the spirit and scope of the technical solutions of this application.

Claims

1. A cluster fault handling method, characterized in that, The method includes: Real-time acquisition of data traffic in the cluster; The first faulty node in the cluster is determined based on the data traffic. Isolate the first node according to its fault level and determine the tasks to be processed on the first node. The task to be processed is assigned to the second node in the cluster based on a preset load balancing strategy, and the second node indicates the other nodes in the cluster besides the first node; Determine the corresponding task data based on the task to be processed; The task data is allocated to the second node according to the storage priority of the second node.

2. The cluster fault handling method as described in claim 1, characterized in that, The step of determining the first faulty node in the cluster based on the data traffic includes: Based on the data traffic, determine the computing resource utilization rate, communication resource utilization rate, request response latency, and task throughput within a preset time for any node in the cluster; based on the computing resource utilization rate, communication resource utilization rate, request response latency, and task throughput within a preset time for any node, determine the performance indicators of any node. Based on the data traffic, determine the heartbeat data, bit error rate, and task retry count of any given node; based on the heartbeat data, bit error rate, and task retry count of any given node, determine the health index of any given node; Based on the data traffic, determine the packet loss rate and cache response rate of the data center where any one node is located; based on the packet loss rate and cache response rate of the data center where any one node is located, determine the associated indicators of any one node. If the performance index, the health index, and the correlation index meet the preset first condition, then any one of the nodes is determined as the first node.

3. The cluster fault handling method as described in claim 2, characterized in that, The method further includes determining the fault level of the first node, wherein determining the fault level of the first node includes: Based on the data traffic of the cluster within a preset first time period, determine the performance indicators, health indicators, and correlation indicators of the first node within the first time period. Based on the performance indicators, health indicators, and correlation indicators, the fault level of the first node is determined using a pre-trained fault level prediction model.

4. The cluster fault handling method as described in claim 3, characterized in that, The method further includes training the fault level prediction model, wherein training the fault level prediction model includes: Based on the data traffic of the cluster within a preset second time period, determine the performance indicators, health indicators, and correlation indicators of any node in the cluster within the second time period. Training samples are determined based on the performance metrics, health metrics, and correlation metrics of any node in the cluster during the second time period. Based on the training samples, using a pre-built initial model, the prediction data corresponding to the training samples is determined; the prediction data includes the prediction level corresponding to the training samples and the prediction probability corresponding to the prediction level. The weight data of the training sample is determined based on the performance index, health index, and correlation index of any node within the second time period; the weight data includes the first weight of the performance index, the second weight of the health index, and the third weight of the correlation index. The loss value of the initial model is determined based on the weight data, the prediction data, and the label data corresponding to the training samples; the label data is used to indicate the fault level of any node within the second time period. The initial model is updated based on the loss value. When the loss value meets the preset termination condition, the update of the initial model is stopped, and a level prediction model trained to a convergent state is obtained.

5. The cluster fault handling method as described in claim 4, characterized in that, The step of determining the weight data of the training samples based on the performance indicators, health indicators, and correlation indicators of any node within the second time period includes: The first weight is determined based on the variance and mean of the performance indicators within the second time period; The second weight is determined based on the variance and mean of the health index within the second time period; The third weight of the training samples is determined based on the variance and mean of the correlation indicators within the second time period.

6. The cluster fault handling method as described in claim 4, characterized in that, Determining the loss value of the initial model based on the weight data, the predicted data, and the label data corresponding to the training samples includes: ； in, This represents the loss value of the initial model; This represents the first weight in the weighted data. This represents the second weight in the weighted data. Represents the third weight in the weighted data; i represents the index of the prediction level in the prediction data, and n represents the number of prediction levels in the prediction data; This represents the probability corresponding to the i-th fault level in the label data; This represents the prediction probability corresponding to the i-th prediction level in the prediction data.

7. The cluster fault handling method as described in claim 1, characterized in that, The isolation operation of the first node based on the fault level of the first node includes: If the fault level meets the preset maintenance conditions, stop assigning data processing tasks to the first node; after the data processing tasks in the first node are completed, send a shutdown command to the first node.

8. The cluster fault handling method as described in claim 1, characterized in that, The method of allocating the task to be processed to the second node in the cluster based on a preset load balancing strategy includes: Based on the data traffic, determine the computing resource utilization rate, communication resource utilization rate, request response latency, and task throughput of any second node within a preset first time period. The polling priority of any second node is determined based on the computing resource utilization rate, communication resource utilization rate, request response latency, and task throughput. The tasks to be processed are assigned to the second node in the cluster according to the polling priority from high to low.

9. The cluster fault handling method as described in claim 8, characterized in that, Determining the polling priority of any second node based on the computing resource utilization rate, communication resource utilization rate, request response latency, and task throughput includes: ； Where R represents the polling priority of any second node; e represents the natural constant; s represents the request response latency; c represents the computing resource utilization rate; t represents the communication resource utilization rate; and num represents the task throughput.

10. The cluster fault handling method as described in claim 1, characterized in that, The step of allocating the task data to the second node according to the storage priority of the second node includes: Based on the data flow, determine the heartbeat data, bit error rate, and number of task retries for any second node within a preset first time period; Based on the heartbeat data, bit error rate, and number of task retries, determine the storage priority of any one of the second nodes; The task data is allocated to the second node in the cluster according to the storage priority from high to low.

11. The cluster fault handling method as described in claim 10, characterized in that, The step of determining the storage priority of any one of the second nodes based on the heartbeat data, bit error rate, and number of task retries includes: ； Where P represents the storage priority; e represents the natural constant; j represents the heartbeat data; k represents the bit error rate; and v represents the number of task retries.

12. A cluster fault handling device, characterized in that, The apparatus includes a module that implements the cluster fault handling method as described in any one of claims 1 to 11, the apparatus comprising: The monitoring module is used to acquire data traffic in the cluster in real time; The determination module is used to determine the first faulty node in the cluster based on the data traffic. The determining module is further configured to isolate the first node according to the fault level of the first node, and determine the tasks to be processed by the first node. The allocation module is used to allocate the task to be processed to a second node in the cluster based on a preset load balancing strategy, wherein the second node indicates other nodes in the cluster besides the first node; The determining module is further configured to determine corresponding task data based on the task to be processed; The allocation module is further configured to allocate the task data to the second node according to the storage priority of the second node.

13. An electronic device, characterized in that, The electronic device includes a processor and a memory, the processor being configured to implement the cluster fault handling method as described in any one of claims 1 to 11 when executing a computer program stored in the memory.

Images