Automated system configuration and monitoring method and system based on Python

Abstract

The present invention provides a Python-based automated system configuration and monitoring method and system, which relates to the automated system configuration and monitoring technical field, including constructing a multi-level service dependency model and a service node affinity matrix, generating a configuration parameter priority queue, and completing the system initial configuration through a distributed transaction coordinator. When the system is running, a monitoring agent collects system operation data, calculates node health indicators, and shares them through a mesh sharing channel. The core node performs data association analysis and updates the node load value and health indicator. When the node health is lower than a preset threshold, an alarm processing module is triggered, and the optimal node is selected based on a node health status assessment report to perform reconfiguration, update a service registration center, and broadcast a configuration update message. The present invention can realize automated system configuration and monitoring, and improve system reliability and stability.

Description

Technical Field

[0001] The present invention relates to automation system configuration and monitoring technology, and in particular to an automation system configuration and monitoring method and system based on Python. Background Art

[0002] In current computer system management, system configuration and monitoring are usually completed by manually operating command line tools or using some dedicated system management software. Manually operating command line tools requires administrators to have high professional knowledge and rich experience, and the operation process is cumbersome and prone to errors. Although dedicated system management software simplifies the operation to a certain extent, it often has functional limitations and cannot adapt well to complex and changing system environments and diverse monitoring needs. For example, for some customized system configuration parameter adjustments and monitoring indicators related to specific business logic, existing software may not meet the requirements.

[0003] In addition, although some automated configuration tools can perform service deployment and monitoring, most tools fail to take into account the affinity of service nodes, load distribution, and dependencies between nodes, and lack load balancing and configuration optimization strategies based on service dependency models, which makes it difficult to meet efficient and accurate service configuration requirements when dealing with large-scale, highly complex systems. Therefore, there is an urgent need for a configuration and monitoring method based on automated scripts and multi-level service dependency models to improve the system's adaptability, optimize configuration efficiency, and enhance the system's health assessment and load balancing capabilities.

[0004] Therefore, there is an urgent need for an innovative Python-based automated system configuration and monitoring method that can automatically generate and optimize configuration parameters based on service dependencies and load information, monitor system health in real time, and achieve dynamic adjustment. Summary of the invention

[0005] The embodiment of the present invention provides a Python-based automated system configuration and monitoring method and system, which can solve the problems in the prior art.

[0006] According to a first aspect of the embodiments of the present invention,

[0007] Provides a Python-based automated system configuration and monitoring method, including:

[0008] A Python script is used to read service information in a service registration center, extract service access paths and call relationships, map service access paths to graph nodes, map call relationships to weighted connection edges, and build a multi-level service dependency model with a directed acyclic graph structure based on node load calculation values. The Python script is used to analyze the connectivity and load distribution of nodes in the multi-level service dependency model, establish a service node affinity matrix, calculate the dependency depth and association strength of configuration parameters, generate a configuration parameter priority queue that considers load balancing, read configuration information based on the configuration parameter priority queue, write configuration parameters into a Shell template in groups according to the service affinity matrix, create an atomic transaction unit with a rollback point through a distributed transaction coordinator, and generate a Shell execution instruction sequence to complete the initial configuration of the system.

[0009] Determine the deployment location of the monitoring agent based on the multi-level service dependency model and the service node affinity matrix, the monitoring agent sets a data sampling period according to the load value, collects system operation data and calculates the node health index, transmits the node health index to adjacent nodes through a publish-subscribe message bus to build a mesh shared channel, and determines the data transmission path according to the service node affinity matrix, performs data preprocessing at the edge node and transmits the preprocessing result to the core node, the core node performs data association analysis based on the multi-level service dependency model, and generates an updated node load value and an updated node health index;

[0010] When the node health index is lower than the preset threshold, the Python alarm processing module is automatically triggered to broadcast an alarm message to the adjacent nodes through the mesh shared channel, obtain the local node health index fed back by the adjacent nodes to complete the alarm confirmation, analyze the local node health index, and generate a node health status assessment report; based on the node health status assessment report, a consistency algorithm is used to select the node with the best health status as the execution node, the Python alarm processing module reads the updated node load value on the execution node, generates a service reconfiguration template based on the service node affinity matrix, assembles the service reconfiguration template and the updated node health index into Shell script parameters, calls the distributed transaction coordinator to execute the Shell script to complete the system reconfiguration, generates a configuration result log, and updates the configuration result log to the service registration center, broadcasts the configuration update message through the mesh shared channel, and triggers the adjacent nodes to execute the configuration update script.

[0011] In an optional embodiment,

[0012] The Python script is used to analyze the connectivity and load distribution of nodes in the multi-level service dependency model, establish a service node affinity matrix, calculate the dependency depth and association strength of configuration parameters, and generate a configuration parameter priority queue that considers load balancing, including:

[0013] Collect real-time monitoring data of each node in the multi-level service dependency model, calculate the CPU usage and memory occupancy of the node based on the monitoring data, set load thresholds to classify the nodes and obtain load distribution information;

[0014] A depth-first search algorithm is used to traverse the multi-level service dependency model, the shortest path distance between any two nodes is calculated to obtain connectivity information, the physical deployment coordinates of the nodes are obtained, the number of physical hops between the nodes is calculated based on the network topology structure to obtain deployment location information, the load distribution information, connectivity information and deployment location information are normalized and weightedly combined, the affinity coefficients between node pairs are calculated, and a service node affinity matrix is ​​constructed;

[0015] Taking the root node of the multi-level service dependency model as the starting point, a breadth-first search algorithm is used to obtain all reachable paths to each configuration parameter node, the number of edges contained in each reachable path is counted, and the maximum number of edges is selected as the dependency depth of the configuration parameter. Based on the service node affinity matrix, a set of associated parameters that directly have a calling relationship with each configuration parameter is identified, and the importance is calculated according to the calling frequency and resource occupancy rate of the associated parameters. The importance of all associated parameters corresponding to each configuration parameter is multiplied by the corresponding affinity coefficient and the average value is calculated to obtain the association strength of the current configuration parameter;

[0016] The dependency depth and association strength of each configuration parameter are standardized, and the weight coefficients of the dependency depth and association strength are dynamically adjusted based on the current load status of the system. The adjusted dependency depth and association strength are linearly combined to obtain the priority value of the configuration parameter. The configuration parameters are sorted in descending order according to the priority value, and configuration parameter pairs with rollback dependencies are identified and ensured to be placed adjacent to each other in the queue; the real-time load status of the node is detected, and when an overloaded node exceeding the preset load threshold is detected, the order of the configuration parameters deployed on the overloaded node is postponed backward; the configuration parameters on the same node are clustered, and the clustering results are placed adjacent to each other while ensuring the rollback dependency, and finally a configuration parameter priority queue that meets the load balancing constraint is generated.

[0017] In an optional embodiment,

[0018] The multi-level service dependency model is traversed using a depth-first search algorithm, the shortest path distance between any two nodes is calculated to obtain connectivity information, the physical deployment coordinates of the nodes are obtained, and the number of physical hops between nodes is calculated based on the network topology structure to obtain deployment location information including:

[0019] Construct an adjacency matrix of a multi-level service dependency model, wherein the adjacency matrix represents the call relationship between service nodes, wherein the matrix elements represent the call weights between service nodes, an access tag array is set to record the access status of the service nodes, and a distance matrix is ​​initialized to store the shortest path distance between the service nodes;

[0020] A depth-first search algorithm is used to traverse the multi-level service dependency model, recursively visit the unvisited adjacent service nodes of the current service node, maintain the path distance from the starting service node to the current service node, and when the target service node is found, the shortest path distance value corresponding to the distance matrix is ​​updated to obtain connectivity information;

[0021] Acquire the physical deployment coordinates of the service node, where the physical deployment coordinates include a data center identifier, a rack identifier, and a server identifier;

[0022] A hop weight matrix is ​​constructed based on the network topology structure, wherein the hop weight matrix includes communication weight within the same server, communication weight between servers in the same rack, communication weight between racks in the same data center, and communication weight across data centers; the physical hops between the service nodes are calculated according to the physical deployment coordinates and the hop weight matrix to obtain deployment location information.

[0023] In an optional embodiment,

[0024] Reading configuration information based on the configuration parameter priority queue, writing configuration parameters into the Shell template in groups according to the service affinity matrix, creating an atomic transaction unit with a rollback point through a distributed transaction coordinator, and generating a Shell execution instruction sequence to complete the initial configuration of the system include:

[0025] Based on the execution order of parameters in the configuration parameter priority queue, read affinity coefficients between parameters, and divide configuration parameters whose affinity coefficients exceed a preset affinity threshold into configuration parameter groups; construct a Shell template for the configuration parameter groups, and the Shell template generates system resource verification information, configuration operation sequence, and configuration verification information;

[0026] Creating a distributed transaction coordinator based on the system resource verification information and generating a transaction identifier, acquiring the initial state of each configuration parameter in the configuration parameter group to generate a rollback point, registering the node executing the configuration parameter group as a transaction participant and generating participant information; the distributed transaction coordinator sends a prepare instruction to the transaction participant based on the participant information, and generates a commit instruction or a rollback instruction according to the received execution status information;

[0027] Writing the transaction identifier, rollback point and configuration operation sequence into the Shell template, merging the same node operations based on the execution status information and marking the parallelizable configuration items, and generating an execution instruction sequence with checkpoints;

[0028] Execute the execution instruction sequence in the order of the configuration parameter group, record the execution progress and resource occupancy status, trigger the rollback instruction to return to the rollback point when the execution progress shows an abnormality, record the execution status as a transaction log and update the node configuration according to the transaction log, check the parameter effectiveness status and service functions based on the configuration verification information, and confirm the completion of the system initial configuration according to the inspection results.

[0029] In an optional embodiment,

[0030] The monitoring agent sets a data sampling period according to the load value, collects system operation data and calculates node health indicators, and transmits the node health indicators to adjacent nodes through a publish-subscribe message bus to build a mesh sharing channel, including:

[0031] The monitoring agent compares the load value with a preset load interval threshold to obtain a current load level, calculates a sampling coefficient corresponding to the current load level based on an exponential backoff algorithm, multiplies a base sampling interval by the sampling coefficient to obtain a current sampling interval, increases the sampling coefficient when the load value corresponds to a higher load level, and decreases the sampling coefficient when the load value corresponds to a lower load level;

[0032] According to the current sampling interval, periodically collect system operation status data to obtain original monitoring data, perform normalization calculation on the original monitoring data to obtain a standardized index value, calculate a resource dimension score and a service dimension score based on the standardized index value, and synthesize the resource dimension score and the service dimension score through dynamic weight to obtain a node health index;

[0033] Construct a health indicator message including the node health indicator, original monitoring data, message priority and maximum propagation hop number, send the node detection message through a multicast detection packet with a random identifier and a timeout, obtain neighbor node information according to the detection response message, and establish a bidirectional publish-subscribe relationship on the publish-subscribe message bus based on the neighbor node information;

[0034] A priority message routing table is constructed according to the bidirectional publish-subscribe relationship, wherein the routing table includes a subscription node list, forwarding rules, and routing priorities. The health indicator message is published to the publish-subscribe message bus according to the priority. The publish-subscribe message bus selects a forwarding path according to the routing table, removes duplicate messages through a message cache based on a sliding time window, and controls the message forwarding range in combination with the maximum number of propagation hops and the node physical distance threshold, thereby forming a mesh shared channel with load balancing and fault tolerance capabilities.

[0035] In an optional embodiment,

[0036] Determining the data transmission path according to the service node affinity matrix, performing data preprocessing at the edge node and transmitting the preprocessing result to the core node, the core node performing data association analysis based on the multi-level service dependency model, and generating updated node load values ​​and node health indicators include:

[0037] Calculate a load balancing factor based on the data transmission load between service nodes, introduce the load balancing factor into the shortest path algorithm to obtain an improved shortest path algorithm, and use the improved shortest path algorithm to process the service node affinity matrix to construct a data transmission path;

[0038] Receive original data at an edge node, process the original data for outliers and missing values ​​to obtain cleaned data, perform dimensionality reduction processing on the cleaned data to obtain feature data, and adaptively compress the feature data according to network bandwidth to obtain preprocessed data;

[0039] The pre-processed data is transmitted to the core node through the data transmission path, and the pre-processed data is input into the constructed multi-level service dependency model of the directed acyclic graph structure, wherein the multi-level service dependency model includes a service layer, an application layer and a resource layer, wherein the service layer extracts service response time, call frequency and error rate from the pre-processed data to construct service call data; the application layer extracts message queue length, processing delay and throughput from the pre-processed data to construct component interaction data; the resource layer extracts CPU utilization, memory occupancy and disk IO from the pre-processed data to construct resource usage data;

[0040] Calculate the connectivity of nodes in the directed acyclic graph and the shortest path between nodes, perform inter-layer data fusion on the service call data, component interaction data and resource usage data according to the node connectivity relationship, and obtain the correlation analysis result between service quality and resource utilization;

[0041] The node health score weight is determined based on the service response time, call frequency and error rate in the association analysis results, and the node health score weight is used to calculate the updated node health index. The resource usage data, the service call data and the component interaction data are combined, and the load change trend is predicted through an adaptive learning algorithm to obtain an updated node load value.

[0042] In an optional embodiment,

[0043] Selecting the node with the best health status as the execution node using a consistency algorithm based on the node health status assessment report includes:

[0044] Obtain a health status assessment report of the candidate node, the health status assessment report including a node health score and a health change trend, and calculate the election priority according to the sum of the product of the node health score and a first weight coefficient and the product of the health change trend and a second weight coefficient;

[0045] The health status assessment report, election priority and node identification information are encapsulated into a pre-election request message and broadcasted, the receiving node parses the pre-election request message to obtain the election priority, and compares it with the highest priority recorded locally, and when the election priority is higher than the highest priority, updates the highest priority and returns a pre-election response message containing the receiving node identification to the candidate node;

[0046] Counting the number of the pre-election response messages received, when the number of the pre-election response messages exceeds a preset message threshold, the candidate node generates a voting request message including a health status assessment report and an election round identifier;

[0047] The receiving node parses the voting request message to obtain a health status assessment report, verifies whether the node health score in the health status assessment report exceeds the execution node threshold, and returns a voting response message including a receiving node identifier and an election round identifier to the candidate node when the node health score exceeds the execution node threshold;

[0048] The candidate node verifies that the election round identifier in the received voting response message is consistent with the current election round identifier. When the number of voting response messages that are verified to be consistent exceeds a preset response message threshold, the candidate node is confirmed to be the execution node with the best health status.

[0049] A second aspect of an embodiment of the present invention provides a Python-based automated system configuration and monitoring system, comprising:

[0050] The first unit is used to use a Python script program to read service information in a service registration center, extract service access paths and call relationships, map service access paths into graph nodes, map call relationships into weighted connection edges, build a multi-level service dependency model with a directed acyclic graph structure based on node load calculation values, use the Python script program to analyze the connectivity and load distribution of nodes in the multi-level service dependency model, establish a service node affinity matrix, calculate the dependency depth and association strength of configuration parameters, generate a configuration parameter priority queue considering load balancing, read configuration information based on the configuration parameter priority queue, write the configuration parameters into a Shell template in groups according to the service affinity matrix, create an atomic transaction unit with a rollback point through a distributed transaction coordinator, and generate a Shell execution instruction sequence to complete the initial configuration of the system;

[0051] The second unit is used to determine the deployment location of the monitoring agent based on the multi-level service dependency model and the service node affinity matrix, the monitoring agent sets the data sampling period according to the load value, collects system operation data and calculates the node health index, transmits the node health index to the adjacent nodes through the publish-subscribe message bus to build a mesh shared channel, and determines the data transmission path according to the service node affinity matrix, performs data preprocessing at the edge node and transmits the preprocessing result to the core node, the core node performs data association analysis based on the multi-level service dependency model, and generates an updated node load value and an updated node health index;

[0052] The third unit is used to automatically trigger the Python alarm processing module to broadcast an alarm message to the adjacent nodes through the mesh shared channel when the node health index is lower than a preset threshold, obtain the local node health index fed back by the adjacent nodes to complete the alarm confirmation, analyze the local node health index, and generate a node health status assessment report; based on the node health status assessment report, a consistency algorithm is used to select the node with the best health status as the execution node, the Python alarm processing module reads the updated node load value on the execution node, generates a service reconfiguration template based on the service node affinity matrix, assembles the service reconfiguration template and the updated node health index into Shell script parameters, calls the distributed transaction coordinator to execute the Shell script to complete the system reconfiguration, generates a configuration result log, and updates the configuration result log to the service registration center, broadcasts the configuration update message through the mesh shared channel, and triggers the adjacent nodes to execute the configuration update script.

[0053] A third aspect of the embodiments of the present invention

[0054] An electronic device is provided, comprising:

[0055] processor;

[0056] a memory for storing processor-executable instructions;

[0057] The processor is configured to call the instructions stored in the memory to execute the aforementioned method.

[0058] A fourth aspect of the embodiments of the present invention is:

[0059] A computer-readable storage medium is provided, on which computer program instructions are stored. When the computer program instructions are executed by a processor, the aforementioned method is implemented.

[0060] In this embodiment, Python scripts are used to automatically read service information, build models, generate configuration parameters, execute configuration and monitoring, reduce manual intervention, and improve configuration efficiency and accuracy. Load balancing configuration and monitoring based on multi-level service dependency models and service node affinity matrices can effectively avoid single point failures, and through real-time monitoring and alarm mechanisms of node health indicators, potential problems can be discovered and handled in a timely manner to ensure stable operation of the system. Configuration parameters and monitoring strategies can be dynamically adjusted according to the system operation status, such as adjusting the data sampling period according to the load value, triggering alarms and reconfiguration according to node health indicators, etc., to achieve adaptive configuration and monitoring, and improve system resource utilization and operation efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] Figure 1 This is a flow chart of an automated system configuration and monitoring method based on Python according to an embodiment of the present invention;

[0062] Figure 2 This is a flowchart of the automation system configuration and monitoring system based on Python according to an embodiment of the present invention;

[0063] Figure 3 It is a schematic diagram of the structure of the Python-based automation system configuration and monitoring system according to an embodiment of the present invention. DETAILED DESCRIPTION

[0064] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0065] The technical solution of the present invention is described in detail with specific embodiments below. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described in detail in some embodiments.

[0066] Figure 1 FIG. 1 is a flow chart of an automated system configuration and monitoring method based on Python according to an embodiment of the present invention. Figure 1 As shown, the method includes:

[0067] S101. Use Python script program to read service information in the service registration center, extract service access path and call relationship, map service access path to graph node, map call relationship to weighted connection edge, build a multi-level service dependency model with directed acyclic graph structure based on node load calculation value, use the Python script program to analyze the connectivity and load distribution of nodes in the multi-level service dependency model, establish a service node affinity matrix, calculate the dependency depth and association strength of configuration parameters, generate a configuration parameter priority queue considering load balancing, read configuration information based on the configuration parameter priority queue, write configuration parameters into Shell template according to the service affinity matrix, create an atomic transaction unit with a rollback point through a distributed transaction coordinator, and generate a Shell execution instruction sequence to complete the initial configuration of the system;

[0068] S102. Determine the deployment location of the monitoring agent based on the multi-level service dependency model and the service node affinity matrix. The monitoring agent sets a data sampling period according to the load value, collects system operation data and calculates the node health index. The node health index is transmitted to adjacent nodes through a publish-subscribe message bus to build a mesh shared channel. At the same time, the data transmission path is determined according to the service node affinity matrix. Data preprocessing is performed at the edge node and the preprocessing result is transmitted to the core node. The core node performs data association analysis based on the multi-level service dependency model to generate an updated node load value and an updated node health index.

[0069] S103. When the node health index is lower than the preset threshold, the Python alarm processing module is automatically triggered to broadcast an alarm message to the adjacent nodes through the mesh shared channel, obtain the local node health index fed back by the adjacent nodes to complete the alarm confirmation, analyze the local node health index, and generate a node health status assessment report; based on the node health status assessment report, a consistency algorithm is used to select the node with the best health status as the execution node, the Python alarm processing module reads the updated node load value on the execution node, generates a service reconfiguration template based on the service node affinity matrix, assembles the service reconfiguration template and the updated node health index into Shell script parameters, calls the distributed transaction coordinator to execute the Shell script to complete the system reconfiguration, generates a configuration result log, and updates the configuration result log to the service registration center, broadcasts the configuration update message through the mesh shared channel, and triggers the adjacent nodes to execute the configuration update script.

[0070] The service registry usually stores metadata for all services in the system, including service name, port number, access path, and call relationship between services. By parsing this information, the access path and call relationship of each service can be extracted. The access path of each service will be used as a node in the graph, and the call relationship between services will be used as the connection edge between nodes. These connection edges will be given weights based on factors such as the service call frequency and data transmission volume to represent the transmission load of the edge.

[0071] After building the service dependency graph, use Python scripts to further analyze the connectivity and load distribution of each node in the graph. Connectivity refers to the strength of the connection between a service node and other service nodes, and load distribution describes the resource consumption of the service node during operation. By analyzing the connectivity of the nodes, the dependencies and critical paths between services can be clearly identified.

[0072] Based on the load value, the affinity matrix of the service nodes is further constructed. The affinity matrix is ​​used to describe the resource sharing relationship between service nodes, for example, the possibility of two services being on the same physical machine, or the frequency of data exchange between them.

[0073] By obtaining service configuration information from the service registry, the service that needs to be configured first is selected based on the generated priority queue. The configuration parameters are grouped according to the affinity matrix of the service node to generate a Shell template. This template is used for initial configuration of the service, where each configuration item matches the dependency and load value of the service to ensure load balancing and efficient operation of the system. Configuration parameters are sorted according to the load level, and services with heavier loads are configured first to reduce system pressure.

[0074] In an optional implementation, using the Python script program to analyze the connectivity and load distribution of nodes in the multi-level service dependency model, establish a service node affinity matrix, calculate the dependency depth and association strength of configuration parameters, and generate a configuration parameter priority queue considering load balancing includes:

[0075] Collect real-time monitoring data of each node in the multi-level service dependency model, calculate the CPU usage and memory occupancy of the node based on the monitoring data, set load thresholds to classify the nodes and obtain load distribution information;

[0076] A depth-first search algorithm is used to traverse the multi-level service dependency model, the shortest path distance between any two nodes is calculated to obtain connectivity information, the physical deployment coordinates of the nodes are obtained, the number of physical hops between the nodes is calculated based on the network topology structure to obtain deployment location information, the load distribution information, connectivity information and deployment location information are normalized and weightedly combined, the affinity coefficients between node pairs are calculated, and a service node affinity matrix is ​​constructed;

[0077] Taking the root node of the multi-level service dependency model as the starting point, a breadth-first search algorithm is used to obtain all reachable paths to each configuration parameter node, the number of edges contained in each reachable path is counted, and the maximum number of edges is selected as the dependency depth of the configuration parameter. Based on the service node affinity matrix, a set of associated parameters that directly have a calling relationship with each configuration parameter is identified, and the importance is calculated according to the calling frequency and resource occupancy rate of the associated parameters. The importance of all associated parameters corresponding to each configuration parameter is multiplied by the corresponding affinity coefficient and the average value is calculated to obtain the association strength of the current configuration parameter;

[0078] The dependency depth and association strength of each configuration parameter are standardized, and the weight coefficients of the dependency depth and association strength are dynamically adjusted based on the current load status of the system. The adjusted dependency depth and association strength are linearly combined to obtain the priority value of the configuration parameter. The configuration parameters are sorted in descending order according to the priority value, and configuration parameter pairs with rollback dependencies are identified and ensured to be placed adjacent to each other in the queue; the real-time load status of the node is detected, and when an overloaded node exceeding the preset load threshold is detected, the order of the configuration parameters deployed on the overloaded node is postponed backward; the configuration parameters on the same node are clustered, and the clustering results are placed adjacent to each other while ensuring the rollback dependency, and finally a configuration parameter priority queue that meets the load balancing constraint is generated.

[0079] Exemplarily, first, collect real-time monitoring data of each node in the multi-level service dependency model. For example, obtain the CPU usage, memory occupancy, network traffic and other data of each node through the monitoring system. Assume that the CPU usage of node A is 70%, the memory usage is 80%, and the CPU usage of node B is 30%, and the memory usage is 40%. Set a load threshold, for example, CPU usage is 75% and memory usage is 85%. Classify the nodes according to the monitoring data and load threshold to obtain load distribution information. For example, node A exceeds the set CPU usage threshold and is marked as a high-load node, and node B is below the threshold and is marked as a low-load node.

[0080] Next, the multi-level service dependency model is traversed using a depth-first search algorithm. For example, starting from the root node, its child nodes are traversed in sequence until the leaf node. The nodes and edges passed during the traversal process are recorded, and the shortest path distance between any two nodes is calculated. For example, the shortest path distance from node A to node B is 3, and the connectivity information is obtained. At the same time, the physical deployment coordinates of the nodes are obtained, for example, node A is deployed in data center 1 and node B is deployed in data center 2. The number of physical hops between nodes is calculated based on the network topology, for example, the number of physical hops from node A to node B is 2, and the deployment location information is obtained. The load distribution information, connectivity information, and deployment location information are normalized separately, for example, all values ​​are scaled to between 0 and 1. Then the normalized data is weighted and combined, for example, the load distribution weight is 0.5, the connectivity weight is 0.3, and the deployment location weight is 0.2. The affinity coefficient between the node pairs is calculated based on the result of the weighted combination, for example, the affinity coefficient between node A and node B is 0.6. Finally, the affinity coefficients of all node pairs are constructed into a service node affinity matrix.

[0081] Then, starting from the root node of the multi-level service dependency model, a breadth-first search algorithm is used to obtain all reachable paths to each configuration parameter node. For example, configuration parameter P1 has two reachable paths, path 1 contains 3 edges, and path 2 contains 2 edges. The number of edges contained in each reachable path is counted, and the maximum number of edges is selected as the dependency depth of the configuration parameter. For example, the dependency depth of configuration parameter P1 is 3. Based on the service node affinity matrix, the set of associated parameters that directly have a call relationship with each configuration parameter is identified. For example, the set of associated parameters of configuration parameter P1 is {P2, P3}. The importance is calculated based on the call frequency and resource occupancy rate of the associated parameters. For example, the call frequency of P2 is 100 times / minute, the resource occupancy rate is 60%, and the call frequency of P3 is 50 times / minute, and the resource occupancy rate is 40%. The importance of all associated parameters corresponding to each configuration parameter is multiplied by the corresponding affinity coefficient and averaged to obtain the association strength of the current configuration parameter.

[0082] Finally, the dependency depth and association strength of each configuration parameter are standardized. The weight coefficients of the dependency depth and association strength are dynamically adjusted based on the current load status of the system. For example, when the current load of the system is high, the weight coefficient of the dependency depth is increased. The adjusted dependency depth and association strength are linearly combined to obtain the priority value of the configuration parameter. The configuration parameters are sorted in descending order according to the priority value. Identify configuration parameter pairs with rollback dependencies, such as P4 and P5 have rollback dependencies, and ensure that they are placed adjacent to each other in the queue. Detect the real-time load status of the node. When an overloaded node exceeding the preset load threshold is detected, the order of the configuration parameters deployed on the overloaded node is postponed. Cluster the configuration parameters on the same node, and place the clustering results adjacent to each other while ensuring the rollback dependency. Finally, a configuration parameter priority queue that meets the load balancing constraint is generated.

[0083] In this embodiment, by giving priority to adjusting configuration parameters with small dependency depth and low correlation strength, the risk of configuration adjustment is reduced, cascading failures are avoided, and system stability is improved. By considering the load distribution of nodes, configuration parameters are deployed preferentially on low-load nodes to avoid further deterioration of overloaded nodes, thereby optimizing resource utilization and improving overall system performance. By generating a priority queue, the adjustment order of configuration parameters is guided, and unnecessary adjustment operations are reduced, thereby improving configuration efficiency and shortening configuration time.

[0084] In an optional implementation, using a depth-first search algorithm to traverse the multi-level service dependency model, calculating the shortest path distance between any two nodes to obtain connectivity information, obtaining the physical deployment coordinates of the nodes, and calculating the number of physical hops between nodes based on the network topology to obtain deployment location information includes:

[0085] Construct an adjacency matrix of a multi-level service dependency model, wherein the adjacency matrix represents the call relationship between service nodes, wherein the matrix elements represent the call weights between service nodes, an access tag array is set to record the access status of the service nodes, and a distance matrix is ​​initialized to store the shortest path distance between the service nodes;

[0086] A depth-first search algorithm is used to traverse the multi-level service dependency model, recursively visit the unvisited adjacent service nodes of the current service node, maintain the path distance from the starting service node to the current service node, and when the target service node is found, the shortest path distance value corresponding to the distance matrix is ​​updated to obtain connectivity information;

[0087] Acquire the physical deployment coordinates of the service node, where the physical deployment coordinates include a data center identifier, a rack identifier, and a server identifier;

[0088] A hop weight matrix is ​​constructed based on the network topology structure, wherein the hop weight matrix includes communication weight within the same server, communication weight between servers in the same rack, communication weight between racks in the same data center, and communication weight across data centers; the physical hops between the service nodes are calculated according to the physical deployment coordinates and the hop weight matrix to obtain deployment location information.

[0089] Exemplarily, first, construct an adjacency matrix of a multi-level service dependency model. The adjacency matrix is ​​used to represent the calling relationship between service nodes. The rows and columns of the matrix represent different service nodes, and the values ​​of the matrix elements represent the calling weights between service nodes. For example, if service node A calls service node B 5 times, the element value of row A and column B in the matrix is ​​5. At the same time, create an access mark array to record the access status of each service node during the depth-first search process, and the initial status is set to unvisited. In addition, create a distance matrix to store the shortest path distance between service nodes, and the initial value is set to infinity.

[0090] Next, the depth-first search algorithm is used to traverse the multi-level service dependency model. Starting from the starting service node, recursively visit its unvisited adjacent service nodes. During the visit process, the path distance from the starting service node to the current service node is maintained, which is the sum of the weights of all edges on the path from the starting node to the current node. When the target service node is visited, the current path distance is compared with the corresponding value in the distance matrix. If the current path distance is shorter, the corresponding shortest path distance value in the distance matrix is ​​updated. For example, assuming that the starting service node is A and the target service node is C, there are two paths A->B->C and A->D->C, the sum of the weights of path A->B->C is 3, and the sum of the weights of path A->D->C is 4, then the value of row A and column C in the distance matrix will be updated to 3. Through the depth-first search algorithm, all service nodes are traversed, and the shortest path distance between all service nodes is finally obtained, thereby obtaining connectivity information.

[0091] Then, obtain the physical deployment coordinates of each service node. The physical deployment coordinates include the data center identifier, rack identifier, and server identifier. For example, the physical deployment coordinates of service node A are (data center 1, rack 2, server 3), indicating that service node A is deployed on server 3 of rack 2 of data center 1.

[0092] Finally, a hop weight matrix is ​​constructed based on the network topology. The hop weight matrix is ​​used to calculate the physical hops between service nodes. The matrix takes into account the communication costs under different network environments, including the communication weight within the same server, the communication weight between servers in the same rack, the communication weight between racks in the same data center, and the communication weight across data centers. For example, the communication weight within the same server is set to 1, the communication weight between servers in the same rack is set to 2, the communication weight between racks in the same data center is set to 3, and the communication weight across data centers is set to 4. According to the physical deployment coordinates of the service nodes and the hop weight matrix, the physical hops between service nodes are calculated. For example, if the physical deployment coordinates of service node A are (data center 1, rack 2, server 3) and the physical deployment coordinates of service node B are (data center 1, rack 2, server 4), the physical hops between service nodes A and B are the communication weight between servers in the same rack, that is, 2. By calculating the physical hops between all service nodes, the deployment location information is finally obtained.

[0093] For example, suppose there are three service nodes A, B, and C. Service node A calls B once with a weight of 1, and service node B calls C twice with a weight of 2. The physical deployment coordinates of A are (data center 1, rack 1, server 1), the physical deployment coordinates of B are (data center 1, rack 1, server 2), and the physical deployment coordinates of C are (data center 2, rack 1, server 1). According to the above method, it can be obtained that the shortest path distance from A to C is 3, the number of physical hops from A to B is 2, and the number of physical hops from B to C is 4.

[0094] In this embodiment, by traversing the multi-level service dependency model through the depth-first search algorithm, the shortest path distance between service nodes can be accurately calculated, thereby obtaining more accurate connectivity information. By obtaining the physical deployment coordinates of the service nodes and calculating the number of physical hops between the service nodes, the faulty nodes can be quickly located, shortening the fault diagnosis time. By analyzing the connectivity and deployment location information between service nodes, the service deployment strategy can be optimized, the network communication cost can be reduced, and the service performance can be improved.

[0095] In an optional implementation, configuration information is read based on the configuration parameter priority queue, configuration parameters are grouped and written into a Shell template according to a service affinity matrix, an atomic transaction unit with a rollback point is created through a distributed transaction coordinator, and a Shell execution instruction sequence is generated to complete the initial configuration of the system, including:

[0096] Based on the execution order of parameters in the configuration parameter priority queue, read affinity coefficients between parameters, and divide configuration parameters whose affinity coefficients exceed a preset affinity threshold into configuration parameter groups; construct a Shell template for the configuration parameter groups, and the Shell template generates system resource verification information, configuration operation sequence, and configuration verification information;

[0097] Creating a distributed transaction coordinator based on the system resource verification information and generating a transaction identifier, acquiring the initial state of each configuration parameter in the configuration parameter group to generate a rollback point, registering the node executing the configuration parameter group as a transaction participant and generating participant information; the distributed transaction coordinator sends a prepare instruction to the transaction participant based on the participant information, and generates a commit instruction or a rollback instruction according to the received execution status information;

[0098] Writing the transaction identifier, rollback point and configuration operation sequence into the Shell template, merging the same node operations based on the execution status information and marking the parallelizable configuration items, and generating an execution instruction sequence with checkpoints;

[0099] Execute the execution instruction sequence in the order of the configuration parameter group, record the execution progress and resource occupancy status, trigger the rollback instruction to return to the rollback point when the execution progress shows an abnormality, record the execution status as a transaction log and update the node configuration according to the transaction log, check the parameter effectiveness status and service functions based on the configuration verification information, and confirm the completion of the system initial configuration according to the inspection results.

[0100] Exemplarily, first, the configuration parameter priority queue reading module reads the predefined configuration parameter priority queue and the affinity coefficient between the parameters. For example, the configuration parameter queue includes parameters A, B, C, D, E, and the affinity coefficient matrix is ​​as follows: AB: 0.8, AC: 0.2, AD: 0.9, AE: 0.3, BC: 0.7, BD: 0.2, BE: 0.6, CD: 0.1, CE: 0.9, DE: 0.2. The preset affinity threshold is 0.7.

[0101] Then, the configuration parameter grouping module performs grouping according to the affinity coefficient. According to the above data, the affinity coefficients of A, B, and D exceed the threshold and are grouped together; the affinity coefficients of C and E exceed the threshold and are grouped together.

[0102] Next, the Shell template building module builds a Shell template for each configuration parameter group. Taking groups A, B, and D as an example, the template contains the following: 1. System resource verification information: verify whether the disk space is sufficient, whether the required ports are available, etc. For example, check whether the remaining space in the / data directory is greater than 10GB, and whether port 8080 is occupied. 2. Configuration operation sequence: for example, install service A, configure service B, and start service D. 3. Configuration verification information: for example, check whether the configuration file of service A is correct, whether service B is listening on the specified port, and whether the process of service D is running.

[0103] Afterwards, the distributed transaction coordinator creation module creates a distributed transaction coordinator based on the system resource verification information and generates a globally unique transaction identifier, such as a UUID. The initial state of each configuration parameter in the configuration parameter group is obtained as a rollback point. For example, the initial state of services A, B, and D is recorded: not installed, not configured, and not started. The nodes that execute the configuration parameter group (such as Node1 and Node2) are registered as transaction participants, and participant information, such as participant ID, IP address, etc., is generated.

[0104] Subsequently, the distributed transaction coordinator sends preparation instructions to the transaction participants. For example, it sends an instruction to prepare to install service A to Node1 and an instruction to prepare to configure service B to Node2. The coordinator generates a commit instruction or a rollback instruction based on the received execution status information (such as success or failure).

[0105] Next, the execution instruction sequence generation module writes the transaction identifier, rollback point, and configuration operation sequence into the Shell template. For example, add the transaction ID, rollback operation, and configuration steps in the Shell script. Based on the execution status information, merge the operations of the same node and mark the parallel configuration items. For example, if Node1 needs to install service A and configure service B, merge them into one operation; if the installation configuration of service A and service B can be executed in parallel, mark them. Finally, generate the execution instruction sequence with checkpoints.

[0106] Finally, the configuration execution and status monitoring module executes the generated Shell instruction sequence in the order of the configuration parameter group. Record the execution progress and resource usage status, such as CPU usage, memory usage, etc. If the execution progress shows an abnormality (for example, a node execution times out), trigger the rollback instruction to restore the system status to the rollback point. Record the execution status as a transaction log, and update the node configuration information based on the transaction log. Check the parameter effectiveness status and service functions based on the configuration verification information. For example, check whether service A is running normally and whether service B is providing services. Confirm that the initial configuration of the system is completed based on the inspection results.

[0107] In this embodiment, the efficiency of the initial configuration of the system can be significantly improved by grouping and executing configuration parameters in parallel. The distributed transaction mechanism ensures that all configuration parameters are either all successful or all rolled back, avoiding system failures caused by inconsistent configurations. Automated Shell script generation and execution simplifies the operation of system administrators and reduces the risk of human error.

[0108] In an optional implementation, the monitoring agent sets a data sampling period according to the load value, collects system operation data and calculates a node health index, and transmits the node health index to adjacent nodes through a publish-subscribe message bus to build a mesh sharing channel, including:

[0109] The monitoring agent compares the load value with a preset load interval threshold to obtain a current load level, calculates a sampling coefficient corresponding to the current load level based on an exponential backoff algorithm, multiplies a base sampling interval by the sampling coefficient to obtain a current sampling interval, increases the sampling coefficient when the load value corresponds to a higher load level, and decreases the sampling coefficient when the load value corresponds to a lower load level;

[0110] According to the current sampling interval, periodically collect system operation status data to obtain original monitoring data, perform normalization calculation on the original monitoring data to obtain a standardized index value, calculate a resource dimension score and a service dimension score based on the standardized index value, and synthesize the resource dimension score and the service dimension score through dynamic weight to obtain a node health index;

[0111] Construct a health indicator message including the node health indicator, original monitoring data, message priority and maximum propagation hop number, send the node detection message through a multicast detection packet with a random identifier and a timeout, obtain neighbor node information according to the detection response message, and establish a bidirectional publish-subscribe relationship on the publish-subscribe message bus based on the neighbor node information;

[0112] A priority message routing table is constructed according to the bidirectional publish-subscribe relationship, wherein the routing table includes a subscription node list, forwarding rules, and routing priorities. The health indicator message is published to the publish-subscribe message bus according to the priority. The publish-subscribe message bus selects a forwarding path according to the routing table, removes duplicate messages through a message cache based on a sliding time window, and controls the message forwarding range in combination with the maximum number of propagation hops and the node physical distance threshold, thereby forming a mesh shared channel with load balancing and fault tolerance capabilities.

[0113] Exemplarily, first, the monitoring agent dynamically adjusts the data sampling period according to the system load. The agent presets multiple load interval thresholds, for example, the load is divided into four levels of "low", "medium", "high" and "very high", corresponding to the CPU usage intervals of 0-25%, 25-50%, 50-75%, and 75-100% respectively. The agent obtains the system load value in real time. For example, if the current CPU usage is 60%, it will be compared with the preset threshold to determine that the current load level is "high".

[0114] Then, the sampling factor is calculated based on the load level. The agent uses an exponential backoff algorithm and presets a baseline sampling interval, such as 60 seconds. The higher the load level, the larger the sampling factor; the lower the load level, the smaller the sampling factor. For example, the "Low" level corresponds to a sampling factor of 0.5, the "Medium" level corresponds to 1, the "High" level corresponds to 2, and the "Extreme" level corresponds to 4. If the current load level is "High", the sampling factor is 2. Multiply the baseline sampling interval by the sampling factor to get the current sampling interval, which is 60 seconds × 2 = 120 seconds. This means that under high load conditions, the agent collects data every 120 seconds. If the CPU usage drops to 40% and the load level becomes "Medium", the sampling factor becomes 1 and the sampling interval becomes 60 seconds.

[0115] Next, the agent program periodically collects system operation status data according to the calculated sampling interval. The collected data includes CPU usage, memory usage, disk I / O rate, network traffic, etc. Assume that in a certain sampling period, the collected raw monitoring data is: CPU usage 65%, memory usage 40%, disk I / O rate 50MB / s, network traffic 10Mbps.

[0116] After obtaining the raw monitoring data, the agent normalizes the data and converts it into a standardized indicator value between 0 and 1. For example, a CPU usage of 65% is converted to a standardized indicator value of 0.65, a memory usage of 40% is converted to 0.4, a disk I / O rate of 50MB / s is converted to 0.5 based on the historical peak value, and a network traffic of 10Mbps is converted to 0.1 based on the bandwidth limit.

[0117] Based on the standardized indicator values, the agent calculates the resource dimension score and the service dimension score respectively. The resource dimension score mainly reflects the usage of system resources, and the service dimension score mainly reflects the performance indicators of system services. For example, according to the preset weights, CPU usage, memory usage, and disk I / O rate account for 50%, 30%, and 20% of the resource dimension score respectively, so the resource dimension score is 0.65×0.5+0.4×0.3+0.5×0.2=0.545. Assuming that the service dimension score is calculated based on the service response time and error rate, the service dimension score is 0.8.

[0118] The agent combines the resource dimension score and the service dimension score through dynamic weights to obtain the node health index. For example, under high load, the resource dimension weight is 0.7 and the service dimension weight is 0.3, then the node health index is 0.545×0.7+0.8×0.3=0.6215.

[0119] Then, the agent constructs a health indicator message, which contains the node health indicator 0.6215, raw monitoring data (CPU usage 65%, memory usage 40%, disk I / O rate 50MB / s, network traffic 10Mbps), message priority (for example, setting the priority according to the health indicator, the lower the health, the higher the priority), and the maximum number of propagation hops (for example, set to 3 hops).

[0120] In order to pass health indicator messages to neighboring nodes, the agent first needs to discover neighbor nodes. It detects neighbor nodes by sending multicast probe packets with random identifiers and timeouts. The node that receives the probe packet replies with a probe response message containing its own node information. The agent builds a neighbor node information list based on the received probe response messages.

[0121] Based on the neighbor node information, the agent establishes a bidirectional publish-subscribe relationship on the publish-subscribe message bus. For example, if agent A discovers neighbor nodes B and C, A will subscribe to messages from B and C, and B and C will also subscribe to messages from A.

[0122] The agent builds a priority message routing table based on the bidirectional publish-subscribe relationship. The routing table contains a list of subscribed nodes, forwarding rules, and routing priorities. For example, the routing table records that node A subscribes to messages from B and C, B subscribes to messages from A and D, and C subscribes to messages from A and E, and specifies the message forwarding priority and rules.

[0123] The agent publishes the health indicator message to the publish-subscribe message bus according to the priority. The message bus selects the forwarding path according to the routing table and forwards the message to the subscribed node. In order to avoid repeated message propagation, the message bus uses a message cache mechanism based on a sliding time window to remove duplicate messages. At the same time, the message bus combines the maximum number of propagation hops and the node physical distance threshold to control the message forwarding range. For example, the message propagates up to 3 hops and is only forwarded to nodes within a certain physical distance range, eventually forming a mesh shared channel with load balancing and fault tolerance capabilities.

[0124] In this embodiment, by dynamically adjusting the sampling period, the data collection frequency can be adaptively adjusted according to the system load. In high-load conditions, the sampling frequency is increased to capture system state changes in a timely manner; in low-load conditions, the sampling frequency is reduced to reduce resource consumption, thereby improving system resource utilization. Through the sharing of node health indicators and the mesh topology, rapid detection and isolation of faults can be achieved. When a node fails, the adjacent nodes can perceive it in time and take corresponding measures, such as transferring the load to other healthy nodes, thereby avoiding a single point of failure that causes the entire system to crash and enhancing system stability. Efficient message delivery and processing can be achieved through the publish-subscribe message bus and priority message routing mechanism. Messages with higher priority can be processed faster, thereby improving monitoring efficiency and timely discovering and solving system problems.

[0125] In an optional implementation, determining a data transmission path according to the service node affinity matrix, performing data preprocessing at the edge node and transmitting the preprocessing result to the core node, the core node performing data association analysis based on the multi-level service dependency model, and generating an updated node load value and node health index include:

[0126] Calculate a load balancing factor based on the data transmission load between service nodes, introduce the load balancing factor into the shortest path algorithm to obtain an improved shortest path algorithm, and use the improved shortest path algorithm to process the service node affinity matrix to construct a data transmission path;

[0127] Receive original data at an edge node, process the original data for outliers and missing values ​​to obtain cleaned data, perform dimensionality reduction processing on the cleaned data to obtain feature data, and adaptively compress the feature data according to network bandwidth to obtain preprocessed data;

[0128] The pre-processed data is transmitted to the core node through the data transmission path, and the pre-processed data is input into the constructed multi-level service dependency model of the directed acyclic graph structure, wherein the multi-level service dependency model includes a service layer, an application layer and a resource layer, wherein the service layer extracts service response time, call frequency and error rate from the pre-processed data to construct service call data; the application layer extracts message queue length, processing delay and throughput from the pre-processed data to construct component interaction data; the resource layer extracts CPU utilization, memory occupancy and disk IO from the pre-processed data to construct resource usage data;

[0129] Calculate the connectivity of nodes in the directed acyclic graph and the shortest path between nodes, perform inter-layer data fusion on the service call data, component interaction data and resource usage data according to the node connectivity relationship, and obtain the correlation analysis result between service quality and resource utilization;

[0130] The node health score weight is determined based on the service response time, call frequency and error rate in the association analysis results, and the node health score weight is used to calculate the updated node health index. The resource usage data, the service call data and the component interaction data are combined, and the load change trend is predicted through an adaptive learning algorithm to obtain an updated node load value.

[0131] Exemplarily, first, determine the data transmission path. The system constructs a service node affinity matrix according to preset rules, and the matrix reflects the priority of data transmission between nodes. For example, the larger the value of the matrix element, the higher the priority of data transmission between two nodes.

[0132] The system improves the shortest path algorithm based on the load balancing factor. The load balancing factor takes into account the data transmission load between nodes. For example, the larger the amount of data transmitted between nodes, the smaller the load balancing factor. The load balancing factor is integrated into the traditional shortest path algorithm such as the Dijkstra algorithm to calculate the shortest path between nodes and finally construct the data transmission path. For example, the shortest path from node 1 to node 4 may be node 1->node 3->node 4.

[0133] Next, data preprocessing is performed at the edge node. The edge node receives the raw data and first performs data cleaning, for example, using the mean or median to fill missing values ​​and using the 3-sigma principle to remove outliers. Assume that the raw data contains three dimensions: temperature, humidity, and light intensity. The cleaned data still contains these three dimensions, but outliers and missing values ​​have been processed. Then, the cleaned data is subjected to dimensionality reduction processing, for example, the main features are extracted using the principal component analysis method. Assume that after dimensionality reduction processing, only two main features, temperature and humidity, are retained. Finally, the feature data is adaptively compressed according to the network bandwidth. For example, when the network bandwidth is low, a higher compression ratio is used. Assume that the compressed data size is 50% of the original data size.

[0134] Then, the preprocessed data is transmitted to the core node. The core node receives the preprocessed data from the edge node and inputs it into the multi-level service dependency model. The model includes the service layer, application layer, and resource layer. The service layer extracts information such as service response time, call frequency, and error rate from the preprocessed data to construct service call data. For example, the average response time of service A is 10ms, the call frequency is 100 times / second, and the error rate is 0.1%. The application layer extracts information such as message queue length, processing delay, and throughput from the preprocessed data to construct component interaction data. For example, the average length of message queue A is 100, the average processing delay is 5ms, and the throughput is 200 messages / second. The resource layer extracts information such as CPU utilization, memory usage, and disk IO from the preprocessed data to construct resource usage data. For example, the CPU utilization of node 1 is 70%, the memory usage is 80%, and the disk IO is 50MB / s.

[0135] The core node performs correlation analysis on the multi-level service dependency model. The connectivity of the nodes in the model and the shortest path between the nodes are calculated. The service call data, component interaction data, and resource usage data are integrated between layers according to the node connectivity relationship to obtain the correlation analysis results between service quality and resource utilization. For example, it is found that the response time of service A is highly correlated with the CPU utilization of node 1.

[0136] Finally, the updated node load value and node health index are generated. The node health score weight is determined based on the service response time, call frequency, and error rate in the association analysis results. For example, the service response time weight is 0.5, the call frequency weight is 0.3, and the error rate weight is 0.2. The updated node health index is calculated using the node health score weight. For example, the health index of node 1 is 0.8. Combining resource usage data, service call data, and component interaction data, the load change trend is predicted through an adaptive learning algorithm (for example, exponential smoothing method) to obtain the updated node load value. For example, it is predicted that the load of node 1 will increase by 10% in the next hour.

[0137] In this embodiment, by analyzing service dependencies and node loads, resource scheduling strategies can be optimized to avoid resource waste and improve overall resource utilization efficiency. By real-time monitoring of node health, potential risks can be discovered in a timely manner, and corresponding measures can be taken to prevent system failures and enhance system stability. By analyzing the correlation between service quality and resource utilization, performance bottlenecks can be identified, thereby improving service quality.

[0138] In an optional implementation, selecting the node with the best health status as the execution node using a consistency algorithm based on the node health status assessment report includes:

[0139] Obtain a health status assessment report of the candidate node, the health status assessment report including a node health score and a health change trend, and calculate the election priority according to the sum of the product of the node health score and a first weight coefficient and the product of the health change trend and a second weight coefficient;

[0140] The health status assessment report, election priority and node identification information are encapsulated into a pre-election request message and broadcasted, the receiving node parses the pre-election request message to obtain the election priority, and compares it with the highest priority recorded locally, and when the election priority is higher than the highest priority, updates the highest priority and returns a pre-election response message containing the receiving node identification to the candidate node;

[0141] Counting the number of the pre-election response messages received, when the number of the pre-election response messages exceeds a preset message threshold, the candidate node generates a voting request message including a health status assessment report and an election round identifier;

[0142] The receiving node parses the voting request message to obtain a health status assessment report, verifies whether the node health score in the health status assessment report exceeds the execution node threshold, and returns a voting response message including a receiving node identifier and an election round identifier to the candidate node when the node health score exceeds the execution node threshold;

[0143] The candidate node verifies that the election round identifier in the received voting response message is consistent with the current election round identifier. When the number of voting response messages that are verified to be consistent exceeds a preset response message threshold, the candidate node is confirmed to be the execution node with the best health status.

[0144] For example, first, obtain the health status assessment report of the candidate node. Each candidate node will generate its own health status assessment report, which includes two indicators: node health score and health change trend. For example, the health score of node A is 95, and the health change trend is rising; the health score of node B is 90, and the health change trend is stable; the health score of node C is 85, and the health change trend is falling.

[0145] Then, calculate the election priority. The election priority is calculated based on the node health score and health change trend. For example, set the weight coefficient of the health score to 0.8, the weight coefficient of the health change trend to 0.2, the "upward" trend is assigned a value of 1, the "stable" is assigned a value of 0.5, and the "downward" is assigned a value of 0.1. Then the election priority of node A is 95×0.8+1×0.2=76.2, the election priority of node B is 90×0.8+0.5×0.2=72.1, and the election priority of node C is 85×0.8+0.1×0.2=68.02.

[0146] Next, the health status assessment report, election priority, and node identification information are encapsulated into a pre-election request message and broadcast. After each node receives the pre-election request message, it parses the message to obtain the election priority and compares it with the highest priority recorded locally. Assume that the highest priority is 0 initially. Node A receives the pre-election request message from B, and the priority 72.1 is greater than 0, then the highest priority is updated to 72.1, and a pre-election response message containing the node A identification is returned to node B.

[0147] Subsequently, the number of pre-election response messages received is counted. Assume that the preset message threshold is half of the total number of nodes in the system. When the number of pre-election response messages received by Node B exceeds the preset message threshold, Node B generates a voting request message including a health status assessment report and an election round identifier (for example, the round identifier is 1).

[0148] Next, the receiving node parses the voting request message to obtain the health status assessment report and verifies whether the node health score exceeds the execution node threshold. Assume that the execution node threshold is 90. Node A receives the voting request message from node B, verifies that the health score of node B exceeds the execution node threshold of 90, and returns a voting response message containing the node A identifier and the election round identifier 1 to node B.

[0149] Finally, the candidate node verifies that the election round identifier in the received voting response message is consistent with the current election round identifier. When the number of voting response messages that are verified to be consistent exceeds the preset response message threshold (for example, half of the total number of nodes in the system), the candidate node is confirmed to be the execution node with the best health status. For example, if node B receives voting response messages from nodes A, C, and D, and the election round identifiers are all 1, which exceeds the preset threshold, node B is confirmed to be the execution node.

[0150] In this embodiment, by selecting the node with the best health status as the execution node, it is possible to avoid assigning tasks to nodes in poor status, thereby reducing the probability of task execution failure and improving the stability and reliability of the system. The method can dynamically select the most suitable node to execute tasks, make full use of the resources of each node in the cluster, avoid resource waste, and improve resource utilization. The method realizes the automatic selection of execution nodes, reduces the need for manual intervention, simplifies the operation and maintenance management of the system, and reduces management costs.

[0151] Figure 2 This is a flowchart of an automated system configuration and monitoring system based on Python according to an embodiment of the present invention. Figure 2 As shown, including:

[0152] Write Shell system configuration templates, including common configuration templates such as system network configuration, user permission configuration, and service startup configuration. These templates exist in the form of Shell script files, which contain standard operation commands and placeholders for receiving dynamic parameters. Use Python scripts to parse Shell templates. The Python script first reads the system environment variables and the configuration parameters entered by the user, and then replaces the placeholders in the Shell template according to these parameters to generate executable Shell commands.

[0153] When configuring the system, execute the Shell command generated by the Python script to complete the system configuration operation. Write a Python monitoring script. The Python monitoring script combines system calls and related monitoring libraries, such as the psutil library, to define monitoring indicators, such as CPU usage, memory usage, disk I / O, etc., and set corresponding thresholds.

[0154] Use Shell scripts to periodically execute Python monitoring scripts. For example, you can set a Python monitoring script to execute every 5 minutes. The Shell script starts the Python monitoring script through system calls and obtains monitoring data.

[0155] Send monitoring data to a specified storage location or analysis platform. You can use the network programming module in the Python script to send monitoring data to a remote server or local database.

[0156] If the monitoring data exceeds the threshold, the Python script takes corresponding measures according to the predefined rules. For example, if the CPU usage exceeds 80%, an alarm SMS is sent to the administrator or the system service configuration parameters are automatically adjusted, such as suspending some non-critical services.

[0157] In this embodiment, the advantages of Shell and Python are combined. Shell scripts provide concise and efficient system operation commands, while Python scripts provide powerful parsing and logic processing capabilities. Through this combination, the operation process of system configuration and monitoring is simplified, and the accuracy of configuration and the effectiveness of monitoring are improved. It can adapt to complex and changeable system environments and diverse monitoring requirements, and provide a unified and efficient solution for computer system management.

[0158] Figure 3 The following is a schematic diagram of the structure of an automated system configuration and monitoring system based on Python according to an embodiment of the present invention. Figure 3 As shown, the system comprises:

[0159] The first unit is used to use a Python script program to read service information in a service registration center, extract service access paths and call relationships, map service access paths into graph nodes, map call relationships into weighted connection edges, build a multi-level service dependency model with a directed acyclic graph structure based on node load calculation values, use the Python script program to analyze the connectivity and load distribution of nodes in the multi-level service dependency model, establish a service node affinity matrix, calculate the dependency depth and association strength of configuration parameters, generate a configuration parameter priority queue considering load balancing, read configuration information based on the configuration parameter priority queue, write the configuration parameters into a Shell template in groups according to the service affinity matrix, create an atomic transaction unit with a rollback point through a distributed transaction coordinator, and generate a Shell execution instruction sequence to complete the initial configuration of the system;

[0160] The second unit is used to determine the deployment location of the monitoring agent based on the multi-level service dependency model and the service node affinity matrix, the monitoring agent sets the data sampling period according to the load value, collects system operation data and calculates the node health index, transmits the node health index to the adjacent nodes through the publish-subscribe message bus to build a mesh shared channel, and determines the data transmission path according to the service node affinity matrix, performs data preprocessing at the edge node and transmits the preprocessing result to the core node, the core node performs data association analysis based on the multi-level service dependency model, and generates an updated node load value and an updated node health index;

[0161] The third unit is used to automatically trigger the Python alarm processing module to broadcast an alarm message to the adjacent nodes through the mesh shared channel when the node health index is lower than a preset threshold, obtain the local node health index fed back by the adjacent nodes to complete the alarm confirmation, analyze the local node health index, and generate a node health status assessment report; based on the node health status assessment report, a consistency algorithm is used to select the node with the best health status as the execution node, the Python alarm processing module reads the updated node load value on the execution node, generates a service reconfiguration template based on the service node affinity matrix, assembles the service reconfiguration template and the updated node health index into Shell script parameters, calls the distributed transaction coordinator to execute the Shell script to complete the system reconfiguration, generates a configuration result log, and updates the configuration result log to the service registration center, broadcasts the configuration update message through the mesh shared channel, and triggers the adjacent nodes to execute the configuration update script.

[0162] According to a third aspect of the embodiments of the present invention,

[0163] An electronic device is provided, comprising:

[0164] processor;

[0165] a memory for storing processor-executable instructions;

[0166] The processor is configured to call the instructions stored in the memory to execute the aforementioned method.

[0167] A fourth aspect of the embodiments of the present invention is:

[0168] A computer-readable storage medium is provided, on which computer program instructions are stored. When the computer program instructions are executed by a processor, the aforementioned method is implemented.

[0169] The present invention may be a method, an apparatus, a system and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for executing various aspects of the present invention.

[0170] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A Python-based automated system configuration and monitoring method, characterized in that: include: A Python script is used to read service information in a service registration center, extract service access paths and call relationships, map service access paths to graph nodes, map call relationships to weighted connection edges, and build a multi-level service dependency model with a directed acyclic graph structure based on node load calculation values. The Python script is used to analyze the connectivity and load distribution of nodes in the multi-level service dependency model, establish a service node affinity matrix, calculate the dependency depth and association strength of configuration parameters, generate a configuration parameter priority queue that considers load balancing, read configuration information based on the configuration parameter priority queue, write configuration parameters into a Shell template in groups according to the service affinity matrix, create an atomic transaction unit with a rollback point through a distributed transaction coordinator, and generate a Shell execution instruction sequence to complete the initial configuration of the system. Determine the deployment location of the monitoring agent based on the multi-level service dependency model and the service node affinity matrix, the monitoring agent sets a data sampling period according to the load value, collects system operation data and calculates the node health index, transmits the node health index to adjacent nodes through a publish-subscribe message bus to build a mesh shared channel, and determines the data transmission path according to the service node affinity matrix, performs data preprocessing at the edge node and transmits the preprocessing result to the core node, the core node performs data association analysis based on the multi-level service dependency model, and generates an updated node load value and an updated node health index; When the node health index is lower than the preset threshold, the Python alarm processing module is automatically triggered to broadcast an alarm message to the adjacent nodes through the mesh shared channel, obtain the local node health index fed back by the adjacent nodes to complete the alarm confirmation, analyze the local node health index, and generate a node health status assessment report; Based on the node health status assessment report, a consistency algorithm is used to select the node with the best health status as the execution node. The Python alarm processing module reads the updated node load value on the execution node, generates a service reconfiguration template based on the service node affinity matrix, assembles the service reconfiguration template and the updated node health index into Shell script parameters, calls the distributed transaction coordinator to execute the Shell script to complete system reconfiguration, generates a configuration result log, and updates the configuration result log to the service registration center. The configuration update message is broadcast through the mesh shared channel to trigger adjacent nodes to execute the configuration update script.

2. The method according to claim 1, characterized in that The Python script is used to analyze the connectivity and load distribution of nodes in the multi-level service dependency model, establish a service node affinity matrix, calculate the dependency depth and association strength of configuration parameters, and generate a configuration parameter priority queue that considers load balancing, including: Collect real-time monitoring data of each node in the multi-level service dependency model, calculate the CPU usage and memory occupancy of the node based on the monitoring data, set load thresholds to classify the nodes and obtain load distribution information; A depth-first search algorithm is used to traverse the multi-level service dependency model, the shortest path distance between any two nodes is calculated to obtain connectivity information, the physical deployment coordinates of the nodes are obtained, the number of physical hops between the nodes is calculated based on the network topology structure to obtain deployment location information, the load distribution information, connectivity information and deployment location information are normalized and weightedly combined, the affinity coefficients between node pairs are calculated, and a service node affinity matrix is ​​constructed; Taking the root node of the multi-level service dependency model as the starting point, a breadth-first search algorithm is used to obtain all reachable paths to each configuration parameter node, the number of edges contained in each reachable path is counted, and the maximum number of edges is selected as the dependency depth of the configuration parameter. Based on the service node affinity matrix, a set of associated parameters that directly have a calling relationship with each configuration parameter is identified, and the importance is calculated according to the calling frequency and resource occupancy rate of the associated parameters. The importance of all associated parameters corresponding to each configuration parameter is multiplied by the corresponding affinity coefficient and the average value is calculated to obtain the association strength of the current configuration parameter; The dependency depth and association strength of each configuration parameter are standardized, and the weight coefficients of the dependency depth and association strength are dynamically adjusted based on the current load status of the system. The adjusted dependency depth and association strength are linearly combined to obtain the priority value of the configuration parameter. The configuration parameters are sorted in descending order according to the priority value, and configuration parameter pairs with rollback dependencies are identified and ensured to be placed adjacent to each other in the queue; the real-time load status of the node is detected, and when an overloaded node exceeding the preset load threshold is detected, the order of the configuration parameters deployed on the overloaded node is postponed backward; the configuration parameters on the same node are clustered, and the clustering results are placed adjacent to each other while ensuring the rollback dependency, and finally a configuration parameter priority queue that meets the load balancing constraint is generated.

3. The method according to claim 2, characterized in that The multi-level service dependency model is traversed using a depth-first search algorithm, the shortest path distance between any two nodes is calculated to obtain connectivity information, the physical deployment coordinates of the nodes are obtained, and the number of physical hops between nodes is calculated based on the network topology structure to obtain deployment location information including: Construct an adjacency matrix of a multi-level service dependency model, wherein the adjacency matrix represents the call relationship between service nodes, wherein the matrix elements represent the call weights between service nodes, an access tag array is set to record the access status of the service nodes, and a distance matrix is ​​initialized to store the shortest path distance between the service nodes; A depth-first search algorithm is used to traverse the multi-level service dependency model, recursively visit the unvisited adjacent service nodes of the current service node, maintain the path distance from the starting service node to the current service node, and when the target service node is found, the shortest path distance value corresponding to the distance matrix is ​​updated to obtain connectivity information; Acquire the physical deployment coordinates of the service node, where the physical deployment coordinates include a data center identifier, a rack identifier, and a server identifier; A hop weight matrix is ​​constructed based on the network topology structure, wherein the hop weight matrix includes communication weight within the same server, communication weight between servers in the same rack, communication weight between racks in the same data center, and communication weight across data centers; the physical hops between the service nodes are calculated according to the physical deployment coordinates and the hop weight matrix to obtain deployment location information.

4. The method according to claim 1, characterized in that: Reading configuration information based on the configuration parameter priority queue, writing configuration parameters into the Shell template in groups according to the service affinity matrix, creating an atomic transaction unit with a rollback point through a distributed transaction coordinator, and generating a Shell execution instruction sequence to complete the initial configuration of the system include: Based on the execution order of parameters in the configuration parameter priority queue, read affinity coefficients between parameters, and divide configuration parameters whose affinity coefficients exceed a preset affinity threshold into configuration parameter groups; construct a Shell template for the configuration parameter groups, and the Shell template generates system resource verification information, configuration operation sequence, and configuration verification information; Creating a distributed transaction coordinator based on the system resource verification information and generating a transaction identifier, acquiring the initial state of each configuration parameter in the configuration parameter group to generate a rollback point, registering the node executing the configuration parameter group as a transaction participant and generating participant information; the distributed transaction coordinator sends a prepare instruction to the transaction participant based on the participant information, and generates a commit instruction or a rollback instruction according to the received execution status information; Writing the transaction identifier, rollback point and configuration operation sequence into the Shell template, merging the same node operations based on the execution status information and marking the parallelizable configuration items, and generating an execution instruction sequence with checkpoints; Execute the execution instruction sequence in the order of the configuration parameter group, record the execution progress and resource occupancy status, trigger the rollback instruction to return to the rollback point when the execution progress shows an abnormality, record the execution status as a transaction log and update the node configuration according to the transaction log, check the parameter effectiveness status and service functions based on the configuration verification information, and confirm the completion of the system initial configuration according to the inspection results.

5. The method according to claim 1, characterized in that The monitoring agent sets a data sampling period according to the load value, collects system operation data and calculates node health indicators, and transmits the node health indicators to adjacent nodes through a publish-subscribe message bus to build a mesh sharing channel, including: The monitoring agent compares the load value with a preset load interval threshold to obtain a current load level, calculates a sampling coefficient corresponding to the current load level based on an exponential backoff algorithm, multiplies a base sampling interval by the sampling coefficient to obtain a current sampling interval, increases the sampling coefficient when the load value corresponds to a higher load level, and decreases the sampling coefficient when the load value corresponds to a lower load level; According to the current sampling interval, periodically collect system operation status data to obtain original monitoring data, perform normalization calculation on the original monitoring data to obtain a standardized index value, calculate a resource dimension score and a service dimension score based on the standardized index value, and synthesize the resource dimension score and the service dimension score through dynamic weight to obtain a node health index; Construct a health indicator message including the node health indicator, original monitoring data, message priority and maximum propagation hop number, send the node detection message through a multicast detection packet with a random identifier and a timeout, obtain neighbor node information according to the detection response message, and establish a bidirectional publish-subscribe relationship on the publish-subscribe message bus based on the neighbor node information; A priority message routing table is constructed according to the bidirectional publish-subscribe relationship, wherein the routing table includes a subscription node list, forwarding rules, and routing priorities. The health indicator message is published to the publish-subscribe message bus according to the priority. The publish-subscribe message bus selects a forwarding path according to the routing table, removes duplicate messages through a message cache based on a sliding time window, and controls the message forwarding range in combination with the maximum number of propagation hops and the node physical distance threshold, thereby forming a mesh shared channel with load balancing and fault tolerance capabilities.

6. The method according to claim 1, characterized in that Determining a data transmission path according to the service node affinity matrix, performing data preprocessing at the edge node and transmitting the preprocessing result to the core node, the core node performing data association analysis based on the multi-level service dependency model, and generating updated node load values ​​and node health indicators include: Calculate a load balancing factor based on the data transmission load between service nodes, introduce the load balancing factor into the shortest path algorithm to obtain an improved shortest path algorithm, and use the improved shortest path algorithm to process the service node affinity matrix to construct a data transmission path; Receiving raw data at an edge node, performing outlier and missing value processing on the raw data to obtain cleaned data, performing dimensionality reduction processing on the cleaned data to obtain feature data, and adaptively compressing the feature data according to network bandwidth to obtain preprocessed data; The pre-processed data is transmitted to the core node through the data transmission path, and the pre-processed data is input into the constructed multi-level service dependency model of the directed acyclic graph structure, wherein the multi-level service dependency model includes a service layer, an application layer and a resource layer, wherein the service layer extracts service response time, call frequency and error rate from the pre-processed data to construct service call data; the application layer extracts message queue length, processing delay and throughput from the pre-processed data to construct component interaction data; the resource layer extracts CPU utilization, memory occupancy and disk IO from the pre-processed data to construct resource usage data; Calculate the connectivity of nodes in the directed acyclic graph and the shortest path between nodes, perform inter-layer data fusion on the service call data, component interaction data and resource usage data according to the node connectivity relationship, and obtain the correlation analysis result between service quality and resource utilization; The node health score weight is determined based on the service response time, call frequency and error rate in the association analysis results, and the node health score weight is used to calculate the updated node health index. The resource usage data, the service call data and the component interaction data are combined, and the load change trend is predicted through an adaptive learning algorithm to obtain an updated node load value.

7. The method according to claim 1, characterized in that Selecting the node with the best health status as the execution node using a consistency algorithm based on the node health status assessment report includes: Obtain a health status assessment report of the candidate node, the health status assessment report including a node health score and a health change trend, and calculate the election priority according to the sum of the product of the node health score and a first weight coefficient and the product of the health change trend and a second weight coefficient; The health status assessment report, election priority and node identification information are encapsulated into a pre-election request message and broadcasted, the receiving node parses the pre-election request message to obtain the election priority, and compares it with the highest priority recorded locally, and when the election priority is higher than the highest priority, updates the highest priority and returns a pre-election response message containing the receiving node identification to the candidate node; Counting the number of the pre-election response messages received, when the number of the pre-election response messages exceeds a preset message threshold, the candidate node generates a voting request message including a health status assessment report and an election round identifier; The receiving node parses the voting request message to obtain a health status assessment report, verifies whether the node health score in the health status assessment report exceeds the execution node threshold, and returns a voting response message including a receiving node identifier and an election round identifier to the candidate node when the node health score exceeds the execution node threshold; The candidate node verifies that the election round identifier in the received voting response message is consistent with the current election round identifier. When the number of voting response messages that are verified to be consistent exceeds a preset response message threshold, the candidate node is confirmed to be the execution node with the best health status.

8. A Python-based automated system configuration and monitoring system, used to implement the method described in any one of claims 1 to 7, characterized in that: include: The first unit is used to use a Python script program to read service information in a service registration center, extract service access paths and call relationships, map service access paths into graph nodes, map call relationships into weighted connection edges, build a multi-level service dependency model with a directed acyclic graph structure based on node load calculation values, use the Python script program to analyze the connectivity and load distribution of nodes in the multi-level service dependency model, establish a service node affinity matrix, calculate the dependency depth and association strength of configuration parameters, generate a configuration parameter priority queue considering load balancing, read configuration information based on the configuration parameter priority queue, write the configuration parameters into a Shell template in groups according to the service affinity matrix, create an atomic transaction unit with a rollback point through a distributed transaction coordinator, and generate a Shell execution instruction sequence to complete the initial configuration of the system; The second unit is used to determine the deployment location of the monitoring agent based on the multi-level service dependency model and the service node affinity matrix, the monitoring agent sets the data sampling period according to the load value, collects system operation data and calculates the node health index, transmits the node health index to the adjacent nodes through the publish-subscribe message bus to build a mesh shared channel, and determines the data transmission path according to the service node affinity matrix, performs data preprocessing at the edge node and transmits the preprocessing result to the core node, the core node performs data association analysis based on the multi-level service dependency model, and generates an updated node load value and an updated node health index; The third unit is used to automatically trigger the Python alarm processing module to broadcast an alarm message to the adjacent nodes through the mesh shared channel when the node health index is lower than a preset threshold, obtain the local node health index fed back by the adjacent nodes to complete the alarm confirmation, analyze the local node health index, and generate a node health status assessment report; Based on the node health status assessment report, a consistency algorithm is used to select the node with the best health status as the execution node. The Python alarm processing module reads the updated node load value on the execution node, generates a service reconfiguration template based on the service node affinity matrix, assembles the service reconfiguration template and the updated node health index into Shell script parameters, calls the distributed transaction coordinator to execute the Shell script to complete system reconfiguration, generates a configuration result log, and updates the configuration result log to the service registration center. The configuration update message is broadcast through the mesh shared channel to trigger adjacent nodes to execute the configuration update script.

9. An electronic device, characterized in that: include: processor; a memory for storing processor-executable instructions; The processor is configured to call the instructions stored in the memory to execute the method described in any one of claims 1 to 7.

10. A computer-readable storage medium having computer program instructions stored thereon, characterized in that: When the computer program instructions are executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images