Management method for nodes in distributed management system

Abstract

The invention discloses a node management method in a distributed management system, relates to the technical field of distributed systems, and aims to solve the problems of load scheduling rigidness, low fault positioning efficiency and poor cross-node collaboration consistency in existing node management. The method comprises a node state sensing module, a dynamic load scheduling module, an intelligent fault processing module and a consistency maintenance module, accurate distribution of node loads is realized through an improved weighted adaptive load algorithm, a dynamic dependency graph is constructed in combination with a global TraceID penetration technology and a graph convolutional network model, and rapid positioning and self-healing of fault root causes are realized. A lease mechanism is innovatively introduced to optimize node state consensus, and reliability and high efficiency of cross-node data synchronization are guaranteed through a hierarchical consistency protocol. The method breaks through the limitation of traditional static scheduling and manual troubleshooting, improves the adaptive capacity, fault tolerance performance and operation and maintenance efficiency of the system, is suitable for a large-scale complex distributed cluster scene, and has wide engineering application value.

Description

A method for managing nodes in a distributed management system Technical Field

[0001] This invention relates to the field of distributed system technology, specifically to a method for managing nodes in a distributed management system. Background Technology

[0002] With the rapid development of internet technology, distributed systems have become the core infrastructure for building large-scale web applications, big data analytics platforms, and microservice architectures. By distributing tasks across multiple independent nodes for collaborative processing, they enhance system scalability, reliability, and availability. As the core component of a distributed system, the management efficiency of nodes directly determines the overall system performance. Node management encompasses key aspects such as load balancing, status monitoring, fault handling, and data consistency maintenance, and is a crucial guarantee for the stable operation of distributed systems.

[0003] Existing distributed node management methods suffer from several technical shortcomings that urgently need to be addressed. In terms of load scheduling, traditional methods often employ round-robin, weighted round-robin, or fixed-threshold scheduling algorithms. These algorithms allocate tasks based solely on static node weights or single load metrics, failing to adapt to dynamic changes in node load in real time. This can easily lead to some nodes being overloaded while others remain idle, resulting in low system resource utilization. Furthermore, existing load algorithms do not consider service dependencies between nodes, and scheduling decisions lack a global perspective, further exacerbating the load imbalance problem. Regarding fault handling, the "black box effect" of distributed systems leads to low efficiency in fault location and recovery. Traditional monitoring suffers from siloed metrics, cross-node logs cannot be effectively correlated, and reliance on manual log checks is insufficient for handling large-scale cluster failure scenarios. Moreover, blind spots in third-party interface monitoring can delay root cause location, prolonging system fault recovery time.

[0004] Regarding node state consensus and data consistency maintenance, existing methods either employ high-overhead Byzantine fault-tolerant algorithms, unsuitable for typical enterprise-level distributed scenarios, or simple master-slave replication mechanisms, which are prone to data inconsistency issues when nodes fail or networks partition. Furthermore, existing node management methods lack dynamic adaptability; when cluster size expands or service dependencies change, management strategies are difficult to adjust flexibly, requiring manual configuration, increasing operational costs and reducing system scalability. These technical shortcomings lead to performance bottlenecks, slow fault recovery, and data synchronization anomalies in large-scale distributed systems, hindering their further application and development. Summary of the Invention

[0005] To address the shortcomings of existing technologies, this invention provides a node management method in a distributed management system, which solves the problems mentioned in the background section.

[0006] To achieve the above objectives, the present invention provides the following technical solution: a node management method in a distributed management system, comprising the following steps: S1, node status awareness: capturing the running status data and service call information of each node through smart probes deployed at the core network egress of the system, generating a globally unique TraceID that runs through each service call link, and constructing a real-time running status dataset of nodes; S2, dynamic load scheduling: based on the status dataset obtained in step S1, calculating the scheduling weight of each node using an improved weighted adaptive load algorithm, and allocating tasks to be processed according to the weight; S3, intelligent fault handling: constructing a dynamic service dependency graph based on the full-link data associated with TraceID, using a graph convolutional network model to infer the probability of node anomalies, locating the root cause of the fault and triggering an adaptive self-healing strategy; S4, node consistency maintenance: introducing a lease mechanism to achieve node status consensus, and ensuring cross-node data synchronization and status consistency through a layered consistency protocol.

[0007] Optionally, the smart probe described in step S1 does not require modification of the business code and can automatically capture call data of mainstream protocols such as HTTP, gRPC, and Kafka. The real-time running status dataset of the node includes node CPU utilization, memory usage, disk I / O rate, network bandwidth usage, service response time, and error rate. At the same time, it records the node interaction sequence, time consumption, and return status of each call link through TraceID, so as to achieve full-link data traceability.

[0008] Optionally, the improved weighted adaptive load algorithm described in step S2 introduces a dynamic load adjustment factor and a service dependency coefficient based on the traditional weighted round-robin algorithm. The algorithm formula is as follows: in, For the first The final scheduling weight of each node, The static base weights of the nodes are determined by the node hardware configuration. The real-time load rate of the node has a value range of [0,1] and is calculated by weighting indicators such as CPU utilization and memory usage. The service dependency coefficient reflects the degree of dependence between this node and other nodes, and is derived based on historical call data statistics. It is the adjustment coefficient, and The system dynamically optimizes the values ​​through self-learning, ensuring that weight allocation adapts to both node load status and service dependencies. During scheduling, the system updates the values ​​of each node every preset time window. and Recalculate The task allocation strategy is adjusted so that when the node load rate is detected to exceed the preset threshold, its scheduling weight is automatically reduced, and tasks are preferentially allocated to nodes with lower load and better compatibility to avoid node overload.

[0009] Optionally, the dynamic service dependency graph construction process in step S3 is as follows: using nodes as graph nodes and service call relationships as edges, the graph structure is updated in real time based on the full-link data associated with TraceID, and the weight of the edges is determined by the call frequency and time consumption; the graph convolutional network model extracts and infers features from the dependency graph and outputs the anomaly probability of each node. ,when When the error exceeds the preset abnormal threshold, it is marked as a suspicious node. Further tracing of the TraceID link data associated with the node will locate the specific call link and root cause path where the failure occurred.

[0010] The adaptive self-healing strategy includes a tiered processing mechanism: for minor faults, it triggers local repair operations such as service restart and connection reconnection; for severe faults, it automatically isolates the faulty node, migrates its tasks to healthy nodes, and updates the load scheduling weights and dependency graph. After the faulty node recovers, tasks are gradually migrated back through a canary release method to avoid secondary impact on the system. Furthermore, for third-party interface faults, it extends the tracking of third-party interface response data through TraceID to accurately locate the root cause of the fault and trigger corresponding degradation strategies or backup interface switching.

[0011] Optionally, the implementation process of the lease mechanism in step S4 is as follows: the system selects a master node to be responsible for distributing leases, and each slave node applies for a lease from the master node. The lease includes the validity period and node status information. The master node verifies the survival status of the slave node through a heartbeat mechanism. When the slave node's heartbeat times out or its status is abnormal, the master node immediately revokes its lease and prohibits the node from participating in task scheduling and data synchronization. When the master node fails, a new master node is elected based on the pre-stored candidate master node list and the lease priority, to ensure that the lease mechanism continues to be effective and avoid node status consensus chaos caused by single point of failure.

[0012] Optionally, the layered consistency protocol described in step S4 divides data synchronization into a core data layer and a non-core data layer: the core data layer adopts the majority principle, requiring more than half of the nodes to confirm the synchronization before it can be committed, ensuring data consistency; the non-core data layer adopts an asynchronous synchronization method, prioritizing data transmission efficiency, and correcting possible data deviations through a timed verification mechanism; the layered strategy can dynamically adjust the data range of each layer according to business needs, achieving a dynamic balance between consistency and efficiency.

[0013] Optionally, the real-time load rate of the node The calculation uses a weighted summation formula: in, For CPU utilization, For memory usage, Disk I / O load rate, Network bandwidth utilization; Let be the weight of each indicator, with a value range of [0,1], and It can be dynamically configured according to the business type, increasing the capacity for compute-intensive businesses. Weighting increases for storage-intensive services. Weights.

[0014] Optionally, the training process of the graph convolutional network model includes: collecting historical fault data and normal operation data of the distributed system, and constructing a training dataset labeled with fault nodes and root cause paths; converting the dynamic service dependency graph into a model input feature matrix, the feature matrix containing features such as node load indicators, call frequency, and time consumption; optimizing model parameters through iterative training, so that the model can accurately identify the abnormal node features corresponding to different fault types and improve the accuracy of fault root cause localization.

[0015] Optionally, it also includes node lifecycle management steps: when a node joins, it automatically completes identity authentication, configuration synchronization, and basic weight initialization, and presets the initial service dependency coefficient based on historical data of similar nodes; when a node leaves, it triggers data migration and task handover processes to ensure that the tasks and data carried by the node are smoothly transferred to other nodes, while updating the service dependency graph and load scheduling strategy; after a node goes offline, the system retains its configuration information for a preset period of time. If it comes back online during this period, the node status and task allocation can be quickly restored, reducing configuration overhead.

[0016] This invention provides a node management method in a distributed management system, offering the following advantages: First, the improved weighted adaptive load balancing algorithm of this invention overcomes the limitations of traditional static scheduling. By introducing a dynamic load adjustment factor and a service dependency coefficient, combined with dynamically optimized adjustment coefficients, task allocation can adapt to both the real-time load status of nodes and service dependencies, achieving global load balancing. Compared to traditional round-robin or fixed-weight algorithms, this algorithm can dynamically respond to changes in node load and adjustments to service dependencies, avoiding node overload and resource idleness, and significantly improving system resource utilization. Simultaneously, the algorithm supports dynamically configuring load indicator weights according to business type, adapting to different business scenarios such as compute-intensive and storage-intensive tasks, enhancing the flexibility and applicability of the method.

[0017] Secondly, this invention combines global TraceID penetration technology with a graph convolutional network model to construct a dynamic service dependency graph, enabling rapid location of fault root causes and hierarchical self-healing. The single-node deployment of the smart probe requires no modification to business code, reducing deployment and maintenance costs. TraceID's full-link penetration can associate call data across nodes and third-party interfaces, completely solving the problems of isolated metrics and blind spots in third-party interface monitoring inherent in traditional monitoring. The graph convolutional network model, through intelligent reasoning of the dependency graph, can accurately identify fault nodes and root cause paths. Combined with a hierarchical self-healing strategy, it achieves closed-loop processing of local repair of minor faults and automatic isolation and migration of severe faults, significantly shortening fault recovery time, reducing manual maintenance intervention, and transforming the system from passive firefighting to proactive prevention, significantly improving system fault tolerance and operational efficiency.

[0018] Secondly, this invention introduces a lease mechanism to optimize node state consensus. Through master-slave node lease management and a master node election mechanism in case of failure, it avoids state chaos caused by single point of failure. Compared with the high-overhead Byzantine fault-tolerant algorithm, it significantly reduces the communication and computational overhead of the consensus process, adapting to the performance requirements of enterprise-level distributed systems. The layered consistency protocol improves the synchronization efficiency of non-core data while ensuring the reliability of core data through differentiated synchronization strategies for core and non-core data, achieving a dynamic balance between consistency and efficiency.

[0019] Furthermore, the node lifecycle management and multi-system heterogeneous adaptation capabilities of this invention further enhance the method's versatility and scalability. Automated handling of node joining, leaving, and offline restart reduces manual configuration costs and adapts to dynamic cluster scaling. Network support for heterogeneous nodes across multiple systems breaks down management barriers between nodes operating different operating systems, providing a unified management view suitable for complex heterogeneous distributed cluster scenarios. In summary, this invention comprehensively improves the adaptive capabilities, fault tolerance, operational efficiency, and adaptability of distributed system node management, stably supporting the operation of large-scale complex distributed clusters, and possesses broad engineering application prospects and promotional value. Attached Figure Description

[0020] Figure 1 is a schematic diagram of the method of the invention. Detailed Implementation

[0021] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments.

[0022] Please refer to Figure 1. This invention provides a technical solution: a node management method in a distributed management system, comprising the following steps: S1, node status awareness: Smart probes deployed at the system's core network egress capture the operational status data and service call information of each node, generating a globally unique TraceID that runs through each service call link, constructing a real-time node operational status dataset. The smart probes do not require modification of the business code and can automatically capture call data from mainstream protocols such as HTTP, gRPC, and Kafka. The real-time node operational status dataset includes node CPU utilization, memory usage, disk I / O rate, network bandwidth usage, service response time, and error rate. Simultaneously, the TraceID records the node interaction sequence, time consumption, and return status of each call link, achieving full-link data traceability. S2, dynamic load scheduling: Based on the status dataset obtained in step S1, an improved weighted adaptive load algorithm is used to calculate the scheduling weight of each node, and tasks are allocated according to the weight. The improved weighted adaptive load algorithm introduces a dynamic load adjustment factor and a service dependency coefficient on the basis of the traditional weighted round-robin algorithm. The algorithm formula is as follows: in, For the first The final scheduling weight of each node, The static base weights of the nodes are determined by the node hardware configuration. The real-time load rate of the node has a value range of [0,1] and is calculated by weighting indicators such as CPU utilization and memory usage. The service dependency coefficient reflects the degree of dependence between this node and other nodes, and is derived based on historical call data statistics. It is the adjustment coefficient, and The system dynamically optimizes the values ​​through self-learning, ensuring that weight allocation adapts to both node load status and service dependencies. During scheduling, the system updates the values ​​of each node every preset time window. and Recalculate The task allocation strategy is adjusted. When a node's load rate exceeds a preset threshold, its scheduling weight is automatically reduced, and tasks are preferentially allocated to nodes with lower load and better dependency compatibility to avoid node overload. S3, Intelligent Fault Handling: A dynamic service dependency graph is constructed based on the full-link data associated with TraceID. A graph convolutional network model is used to infer the probability of node anomalies, locate the root cause of the fault, and trigger an adaptive self-healing strategy. The dynamic service dependency graph construction process is as follows: nodes are used as graph nodes, and service call relationships are used as edges. The graph structure is updated in real time based on the full-link data associated with TraceID. The edge weights are determined by the call frequency and time consumption. The graph convolutional network model extracts and infers features from the dependency graph, outputting the anomaly probability of each node. ,when When the error exceeds the preset abnormal threshold, it is marked as a suspicious node. Further tracing of the TraceID link data associated with the node will locate the specific call link and root cause path where the failure occurred.

[0023] The adaptive self-healing strategy includes a tiered processing mechanism: for minor faults, it triggers local repair operations such as service restart and connection reconnection; for severe faults, it automatically isolates the faulty node, migrates the tasks it carries to healthy nodes, and updates the load scheduling weights and dependency graph. After the faulty node recovers, the tasks are gradually migrated back through a canary release method to avoid secondary impact on the system. Furthermore, for third-party interface failures, TraceID is used to extend the tracking of the response data of the third-party interface, enabling precise location of the root cause of the failure and triggering corresponding degradation strategies or backup interface switching. The training process of the graph convolutional network model includes: collecting historical failure data and normal operation data of the distributed system to construct a training dataset labeled with faulty nodes and root cause paths; converting the dynamic service dependency graph into a model input feature matrix, which includes features such as node load indicators, call frequency, and time consumption; optimizing model parameters through iterative training to enable the model to accurately identify the abnormal node features corresponding to different failure types and improve the accuracy of fault root cause location; S4, node consistency maintenance, introducing a lease mechanism to achieve node state consensus, and ensuring cross-node data synchronization and state consistency through a layered consistency protocol; the implementation process of the lease mechanism is as follows: the system selects a master node to be responsible for distributing leases, and each slave node applies to the master node for a lease. Please establish a lease, which includes the validity period and node status information. The master node verifies the liveness of slave nodes through a heartbeat mechanism. When a slave node's heartbeat times out or its status becomes abnormal, the master node immediately revokes its lease, prohibiting the node from participating in task scheduling and data synchronization. When the master node fails, a new master node is elected based on a pre-stored list of candidate master nodes, according to lease priority, ensuring the lease mechanism remains effective and avoiding node status consensus chaos caused by single-point failures. The layered consistency protocol divides data synchronization into a core data layer and a non-core data layer: the core data layer adopts a majority-based principle, requiring more than half of the nodes to confirm the synchronization before committing, ensuring data consistency; the non-core data layer uses an asynchronous synchronization method, prioritizing data transmission efficiency and correcting potential data deviations through a periodic verification mechanism. The layered strategy can dynamically adjust the data range of each layer according to business needs, achieving a dynamic balance between consistency and efficiency. Node real-time load rate. The calculation uses a weighted summation formula: in, For CPU utilization, For memory usage, Disk I / O load rate, Network bandwidth utilization; Let be the weight of each indicator, with a value range of [0,1], and It can be dynamically configured according to the business type, increasing the capacity for compute-intensive businesses. Weighting increases for storage-intensive services. Weights; the method also includes node lifecycle management steps: when a node joins, it automatically completes identity authentication, configuration synchronization and basic weight initialization, and presets the initial service dependency coefficient based on historical data of similar nodes; when a node leaves, it triggers data migration and task handover processes to ensure that the tasks and data carried by the node are smoothly transferred to other nodes, while updating the service dependency graph and load scheduling strategy; after a node goes offline, the system retains its configuration information for a preset period of time. If it comes back online during this period, the node status and task allocation can be quickly restored, reducing configuration overhead.

[0024] This method is applicable to multi-system heterogeneous node clusters. It enables network management of nodes with different operating systems such as HarmonyOS, Euler, and Ubuntu through a custom protocol, provides a unified node management view, supports load scheduling, fault handling, and data synchronization across system nodes, and is compatible with the interface specifications and data formats of different systems, thereby improving the versatility and adaptability of the method.

[0025] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A method for managing nodes in a distributed management system, characterized in that, Includes the following steps: S1. Node Status Awareness: Smart probes deployed at the system's core network egress capture the operational status data and service call information of each node, generating a globally unique TraceID that runs through each service call link, and constructing a real-time node operational status dataset. S2. Dynamic Load Scheduling: Based on the status dataset obtained in step S1, an improved weighted adaptive load algorithm is used to calculate the scheduling weight of each node, and tasks to be processed are allocated according to the weight. S3. Intelligent Fault Handling: A dynamic service dependency graph is constructed based on the full-link data associated with TraceIDs. A graph convolutional network model is used to infer the probability of node anomalies, locate the root cause of the fault, and trigger an adaptive self-healing strategy. S4. Node consistency maintenance: A lease mechanism is introduced to achieve node state consensus, and a layered consistency protocol is used to ensure cross-node data synchronization and state consistency.

2. The method according to claim 1, characterized in that, The smart probe described in step S1 does not require modification of the business code. The real-time running status dataset of the node includes node CPU utilization, memory usage, disk I / O rate, network bandwidth usage, service response time and error rate. At the same time, TraceID records the node interaction sequence, time consumption and return status of each call link to achieve full-link data traceability.

3. The method according to claim 1, characterized in that, The improved weighted adaptive load algorithm described in step S2 introduces a dynamic load adjustment factor and a service dependency coefficient based on the traditional weighted round-robin algorithm. The algorithm formula is as follows: in, For the first The final scheduling weight of each node, The static base weights of the nodes are determined by the node hardware configuration. The real-time load rate of the node has a value range of [0,1] and is calculated by weighting indicators such as CPU utilization and memory usage. The service dependency coefficient reflects the degree of dependence between this node and other nodes, and is derived based on historical call data statistics. It is the adjustment coefficient, and The system dynamically optimizes the values ​​through self-learning, ensuring that weight allocation adapts to both node load status and service dependencies. During scheduling, the system updates the values ​​of each node every preset time window. and Recalculate The task allocation strategy is adjusted so that when the node load rate is detected to exceed the preset threshold, its scheduling weight is automatically reduced and the task is preferentially allocated to nodes with lower load and better compatibility.

4. The method according to claim 1, characterized in that, The dynamic service dependency graph construction process described in step S3 is as follows: using nodes as graph nodes and service call relationships as edges, the graph structure is updated in real time based on the full-link data associated with TraceID, and the weight of the edges is determined by the call frequency and time consumption; the graph convolutional network model extracts and infers features from the dependency graph and outputs the anomaly probability of each node. ,when When an anomaly exceeds a preset threshold, it is marked as a suspicious node. Further tracing of the TraceID link data associated with this node allows for the identification of the specific call stage and root cause path of the failure. The adaptive self-healing strategy includes a tiered processing mechanism: for minor faults, local repair operations such as service restart and connection reconnection are triggered; for severe faults, the faulty node is automatically isolated, its tasks are migrated to healthy nodes, and the load scheduling weights and dependency graph are updated. After the faulty node recovers, tasks are gradually migrated back through a canary release method to avoid secondary impact on the system. Furthermore, for third-party interface faults, the response data of the third-party interface is traced through TraceID extension to achieve precise location of the root cause of the fault, triggering corresponding degradation strategies or backup interface switching.

5. The method according to claim 1, characterized in that, The implementation process of the lease mechanism described in step S4 is as follows: the system selects a master node to be responsible for distributing leases, each slave node applies for a lease from the master node, the lease includes the validity period and node status information, the master node verifies the survival status of the slave node through the heartbeat mechanism, when the slave node's heartbeat times out or its status is abnormal, the master node immediately revokes its lease and prohibits the node from participating in task scheduling and data synchronization. When the master node fails, a new master node is elected based on the pre-stored list of candidate master nodes and the lease priority, ensuring that the lease mechanism remains effective.

6. The method according to claim 1, characterized in that, The layered consistency protocol described in step S4 divides data synchronization into a core data layer and a non-core data layer: the core data layer adopts the majority principle, requiring more than half of the nodes to confirm the synchronization before it can be committed, ensuring data consistency; the non-core data layer adopts an asynchronous synchronization method, prioritizing data transmission efficiency, and correcting possible data deviations through a timed verification mechanism; the layered strategy can dynamically adjust the data range of each layer according to business needs, achieving a dynamic balance between consistency and efficiency.

7. The method according to claim 3, characterized in that, The node's real-time load rate The calculation uses a weighted summation formula: in, For CPU utilization, For memory usage, Disk I / O load rate, Network bandwidth utilization; Let be the weight of each indicator, with a value range of [0,1], and It can be dynamically configured according to the business type, increasing the capacity for compute-intensive businesses. Weighting increases for storage-intensive services. Weights.

8. The method according to claim 4, characterized in that, The training process of the graph convolutional network model includes: collecting historical fault data and normal operation data of the distributed system, constructing a training dataset labeled with fault nodes and root cause paths; converting the dynamic service dependency graph into a model input feature matrix; and optimizing the model parameters through iterative training so that the model can accurately identify the abnormal node features corresponding to different fault types and improve the accuracy of fault root cause localization.

9. The method according to claim 1, characterized in that, It also includes node lifecycle management steps: when a node is added, it automatically completes identity authentication, configuration synchronization and basic weight initialization, and presets the initial service dependency coefficient based on historical data of similar nodes; When a node exits, a data migration and task handover process is triggered to ensure that the tasks and data carried by the node are smoothly transferred to other nodes, while updating the service dependency graph and load scheduling strategy. When a node goes offline, the system retains its configuration information for a preset period of time. If it comes back online during this period, the node status and task allocation can be quickly restored, reducing configuration overhead.

Images