A computer cluster monitoring and alarm method based on DQN algorithm

Abstract

The present invention discloses a cluster monitoring and alarm method based on a DQN algorithm, and belongs to the field of computer technology. The method includes: collecting monitoring data in a computing cluster state monitoring system, and preprocessing the data; extracting features effective for monitoring and alarming from the processed data, annotating the data set, and constructing an experience pool; designing the structure of a deep Q network and initializing it; using a loss function based on a TD error to train the deep Q network, so that the prediction of the action value by the main Q network will be closer and closer to the actual situation, so that the intelligent agent can select the optimal monitoring and alarm action when facing a specific state; the present invention is mainly used for the establishment of a computer cluster monitoring and alarm system, and by introducing deep reinforcement learning technology, the accuracy of monitoring and alarming is improved, and at the same time, it also provides strong support for coping with complex and dynamic system environments.

Description

Technical Field

[0001] The invention belongs to the field of computer technology. Background Art

[0002] Computer cluster monitoring and alarming are important means to ensure that servers, network devices, applications and services in the cluster remain up and running. With the popularity of data centers, cloud computing and high-performance computing clusters, monitoring and alarming systems have become essential. The operating environment of the cluster is complex, and efficient monitoring tools are needed to ensure the availability, performance and security of the system.

[0003] Computer clusters are usually composed of multiple servers (nodes) that jointly process large-scale computing tasks or service requests. In order to ensure the stable and efficient operation of the cluster, the monitoring system is used to continuously track the node's resource usage (CPU, memory, storage, network, etc.), application performance, service status, etc., and issue alarms in abnormal situations (such as node failure, resource exhaustion, network interruption, etc.).

[0004] The monitoring and alarm system is usually divided into two aspects: one is real-time monitoring, which refers to the continuous collection of performance and health status data of each node in the cluster, such as CPU load, memory usage, disk I / O, network traffic, etc. The other is alarm management, which means that when the system detects an abnormality (such as exceeding the preset threshold), it automatically triggers an alarm to notify the system administrator so that countermeasures can be taken quickly.

[0005] In general, computer cluster monitoring and alarm systems are core technologies to ensure the smooth operation of large-scale computing environments, but they still face many challenges. The current computer cluster monitoring and alarm systems have the following monitoring technologies: active monitoring: the system regularly sends requests to check the status of each node (such as Ping, HTTP requests, etc.), which is suitable for monitoring network equipment and server status; passive monitoring: the node actively sends heartbeat information or status reports to the monitoring system, which is suitable for real-time update of data streams; time series database (TSDB): such as TSDB in Prometheus, which is used to store and query time series data, helping to track historical performance indicators and analyze trends. In terms of alarm technology, the alarm is mainly issued through the threshold alarm method. The threshold alarm is to set upper and lower limits for a certain performance indicator, and an alarm is issued when the data exceeds the range.

[0006] However, these monitoring and alarm methods still have a series of problems in terms of adaptability, efficiency, and comprehensive decision-making, including:

[0007] 1. Traditional monitoring and alarm systems are prone to generate a large number of useless alarms, especially in large-scale clusters, which may lead to "alarm fatigue" and make it difficult for administrators to distinguish which are important alarms and which are noise.

[0008] 2. With the development of cloud computing and container technology, the scale and topology of clusters have become more dynamic. Traditional monitoring tools based on static configuration cannot adapt to changes in dynamic environments (such as elastic expansion) in a timely manner.

[0009] 3. Some alarm systems have high latency, which results in delayed alarms after a problem occurs and inability to respond quickly. This is especially important for monitoring high-performance computing or real-time services. Summary of the invention

[0010] In order to overcome the shortcomings of the above-mentioned prior art, the present invention provides a computer cluster monitoring and alarm method based on the DQN algorithm, introduces deep reinforcement learning technology, and comprehensively considers multiple factors in monitoring and alarming through interactive learning between the intelligent agent and the environment, such as key indicators of system status, task operation results, and current node status, etc. It can automatically learn and optimize alarm rules according to the node status and the status of tasks to be executed, so as to better improve the accuracy and efficiency of computer cluster monitoring and alarming, and can also be used for the system to realize automatic troubleshooting, resource scheduling and other functions.

[0011] In order to solve the above technical problems, the present invention proposes a computer cluster monitoring and alarm method based on the DQN algorithm, which is implemented by the following technical solutions:

[0012] Step 1: Collect monitoring data in real time in the computer cluster status monitoring system; and pre-process the collected data.

[0013] Step 2: Perform feature extraction based on the data collected and preprocessed in step 1 to extract features useful for alarm rules.

[0014] Step 3: Associate the monitoring data with the actual alarm time, that is, mark the time periods in which situations requiring alarms occurred, and finally complete the construction of the experience pool.

[0015] Step 4: Build a DQN network and initialize DQN parameters, where DQN stands for Deep Q-Network; input the data in the experience pool into the Q network for training, and optimize the DQN parameters by using the stochastic gradient descent algorithm until the Q value corresponding to the maximum Q value action tends to be stable, thus obtaining a trained Q network.

[0016] Step 5: Deploy the trained model to the monitoring alarm system, and output the Q value of each action under the current state s according to the Q network trained in step 4 to determine whether the monitoring alarm should be triggered.

[0017] Furthermore, in step 1, the computer cluster system needs to collect a large amount of monitoring data, including but not limited to performance indicators (such as CPU usage, memory occupancy, etc.), system logs, and network status. These data come from multiple nodes and components in the cluster, and are uniformly collected, cleaned, and preprocessed to ensure data quality and consistency. The preprocessing steps may include denoising, missing value filling, data normalization, etc.

[0018] Furthermore, when performing feature extraction in step 2, the characteristics of the tasks performed by the computer cluster and the main alarm risks faced can be focused on, so as to efficiently extract features useful for the alarm rules. These features may be direct monitoring indicators or statistics obtained through some transformation or aggregation. The selection and extraction of features in step 2 is crucial to the performance of the deep reinforcement learning model in the subsequent steps.

[0019] Furthermore, in the scenario of computer cluster monitoring alarms, the labeling task in step 3 may include identifying abnormal conditions that cause alarms, marking time periods of performance bottlenecks, classifying different types of warnings, and grading the severity of warnings. After completing the labeling task, the labeled data is used to build an experience pool. The basic unit of the experience pool is the experience tuple. The structure of the experience tuple includes the current state s, the executed action a, the next state s', the reward value r, and the flag done indicating whether the process is over. The construction process of the experience pool mainly includes state space construction, action space construction, and reward design.

[0020] Furthermore, the state space construction process is as follows:

[0021] Based on the computer cluster monitoring and alarm system built in step 1, the monitoring data of the computing nodes is extracted from the database on its control node, and the monitoring data is set as a binary array s[s1,s2,s3...s n ], where s1, s2, s3...s n Corresponding to the state s on each computer node respectively; the expression of the state s is Where n represents the nth node in the computer cluster, t n Indicates the corresponding time of state s on the current node; c, m, d, u, g respectively represent the current time t n The number of CPU cores, memory, hard disk, network bandwidth, and GPU cores available for the computing node. z represents the current time t n The remaining value coefficient of the current time t n The calculation formula of the remaining value coefficient z of the node resource is as follows:

[0022]

[0023] Among them, c′, m′, d′, u′, and g′ respectively represent the total number of CPU cores, memory, hard disk, network bandwidth, and GPU cores on the computing node. The content in s can be modified accordingly according to the feature values ​​extracted in step 2 to suit its corresponding cluster.

[0024] Furthermore, the action space is defined as follows:

[0025] The action space contains all the actions that the agent can perform in deep reinforcement learning. In this model, it can generally be set to the following structure:

[0026]

[0027] Among them, n1, n2, etc. represent nodes, and A, B, C, etc. represent the alarm level or no alarm. The action space can be adjusted accordingly according to the characteristics and work requirements of different clusters.

[0028] Furthermore, the reward design is designed as follows:

[0029] The formula for the reward value r of a single action is:

[0030]

[0031] Among them, t′ is the time when the system fault starts to appear, t is the time when the fault ends, e is the overall work efficiency of the system during the fault. In order to avoid the reward value being too large, the minimum value range for e can be manually determined, and a is the basic reward coefficient.

[0032] The process of training the deep Q network in step 4 is as follows:

[0033] Step 4.1: Build an appropriate deep Q network (DQN). The core is to design the network structure. The deep Q network usually consists of three layers, each of which has different functions to ensure that the network can accurately learn the mapping from the environment state to the action value. Specifically:

[0034] Input layer: The design of the input layer should closely correspond to the dimensionality of the environment state. The number of neurons in the input layer should be equal to the expanded dimensionality of the state space representation.

[0035] Hidden layer: The structure and configuration of the hidden layer directly affects the learning ability and efficiency of the network. The number of hidden layers and the number of neurons in each layer can be set according to actual needs.

[0036] Output layer: The number of neurons in the output layer should be consistent with the size of the action space. This means that the number of neurons in the output layer should be equal to the total number of possible scheduling actions in the platform. This design ensures that the network can generate a Q value, i.e. the expected reward of the action, for each possible scheduling decision, thus providing a basis for making the optimal scheduling decision.

[0037] After the network structure design is completed, two Q networks with the same structure are established, namely the main Q network and the target Q network, and their parameters are randomly initialized. The main Q network is used to generate the Q value prediction of the action, while the target Q network is used to calculate the target Q value in the loss function to help stabilize the training process.

[0038] Step 4.2: Use the temporal difference (TD) learning method, where the loss function is based on the TD error, which measures the difference between the model's current action value prediction and the actual reward value plus the expected reward based on future states. Specifically, for a certain experience tuple in the experience pool (state S, action A, reward R, next state S'), DQN uses the main network to estimate the Q value of the current state and action (Qreal), and uses the target network to calculate the maximum expected reward for the next state (Qtarget). The formula for calculating the loss function is as follows:

[0039]

[0040] Step 4.3: The system periodically executes the steps of extracting samples from the experience pool, calculating losses, and updating network parameters. Through such iterations, the temporal difference (TD) error gradually decreases, allowing DQN to more accurately learn which warning actions can bring the greatest expected reward in a given cluster state. As training progresses, the main Q network's prediction of the action value will become closer and closer to the actual situation, allowing the agent to choose the optimal warning action when facing a specific state.

[0041] Compared with the prior art, the present invention has the following beneficial effects:

[0042] 1. The present invention can automatically optimize the alarm strategy by continuously interacting with the monitoring system environment. In traditional systems, the alarm threshold is usually set manually, which is easy to fail in dynamically changing systems. Through reinforcement learning, the system can automatically adjust the alarm threshold based on historical alarm data and environmental feedback to reduce false positives and missed alarms. According to the performance and load conditions of the actual system, deep reinforcement learning can learn what kind of alarm threshold should be used in different time periods or usage scenarios.

[0043] 2. Reduce alarm noise (false positives). The present invention can automatically learn which alarms are important and which are noise based on historical data and feedback from the environment. In this way, the system can more accurately predict and judge abnormal situations, reducing the fatigue of administrators caused by too many unnecessary alarms. The present invention can capture the contextual information of system operation. For example, some alarms may be normal under high load, but may indicate potential problems under low load. It can learn more appropriate alarm strategies based on different contexts.

[0044] 3. Modern data centers, cloud computing platforms, and large-scale distributed systems are usually very complex. Deep reinforcement learning can mine complex patterns in high-dimensional data and nonlinear systems and adapt to this complexity. Therefore, the present invention can deal with nonlinear behaviors and emergencies in clusters and learn how to make effective warning decisions in complex and dynamic environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 is a flow chart of an embodiment of the present invention.

[0046] Figure 2 The basic structure of the experience pool for model training in the present invention.

[0047] Figure 3 This is the basic structure diagram of the deep Q network used in the present invention. DETAILED DESCRIPTION

[0048] The present invention is further explained below in conjunction with specific embodiments. It should be understood that these embodiments are only used to illustrate the present invention and are not used to limit the scope of the present invention. After reading the present invention, various equivalent forms of modifications to the present invention by those skilled in the art all fall within the scope defined by the claims attached to this application.

[0049] like Figure 1 As shown, the computer cluster monitoring and alarm method based on deep reinforcement learning proposed in this embodiment is divided into six steps:

[0050] Step 1: Collect monitoring data in real time in the computer cluster monitoring system;

[0051] The computer cluster monitoring system includes at least one control node and several computing nodes. Assume that a computer cluster has one control node and three computing nodes, wherein the computing nodes are represented as n1, n2, and n3, and each computing node includes node_exporter, mysql_exporter, and Pushgateway components. All components on the computing node collect the node operation status in real time and save the collected monitoring data to the prometheus database on the control node;

[0052] Step 2: Preprocess the data, unify the data format, obtain the relevant information and execution parameters of the task operation, and store the processed data in the MySQL database of the control node;

[0053] Step 3: Extract key feature values, focusing on the characteristics of the tasks performed by the computer cluster and the main alarm risks it faces, so as to efficiently extract features that are useful for alarm rules. These features may be direct monitoring indicators or statistics obtained through some transformation or aggregation. In this implementation, the node resource surplus value coefficient is a statistic obtained through transformation and aggregation, and its calculation formula is:

[0054]

[0055] Step 4: Label the data set and build an experience pool. The specific structure of the experience pool is as follows: Figure 2 shown.

[0056] In this implementation, the action space is defined as:

[0057]

[0058] Among them, n1, n2, n3, etc. represent nodes, and A, B, and C represent the alarm behaviors issued, where A represents no alarm, B represents a general alarm, and C represents a serious alarm. The alarm status of each computing node is represented by combining nodes and alarm behaviors.

[0059] For this embodiment, the formula for the reward value r of a single action is as follows:

[0060]

[0061] Among them, t′ is the time when the system fault starts to appear, t is the time when the fault ends, and e is the overall work efficiency of the system during the fault. In order to avoid the reward value being too large, the value range of e can be manually determined. In this embodiment, the initial value of the minimum value of e is set to 0.2, that is, when the current work efficiency is lower than 20% of the normal time work efficiency, the reward value deduction reaches the maximum. a is the basic reward coefficient, which can be adjusted later according to the model training effect.

[0062] Step 5: In order to effectively train the intelligent monitoring and alarm system, establishing an appropriate deep Q network (DQN) is the first step, and its core lies in carefully designing the network structure. The deep Q network is usually composed of three layers. The structure of DQN in this embodiment is as follows: Figure 3 As shown in the figure, it contains one input layer, one output layer and two hidden layers. Each layer has different functions to ensure that the network can accurately learn the mapping from environmental state to action value. Specifically:

[0063] (1) Input layer: The design of the input layer should closely correspond to the dimension of the environment state. In this system, the number of neurons in the input layer should be equal to the expanded dimension of the state space representation.

[0064] (2) Hidden layer: The hidden layer plays a vital role in the network, and its structural configuration directly affects the learning ability and efficiency of the network. The number of hidden layers and the number of neurons in each layer can be set according to actual needs. The number of neurons usually ranges from dozens to hundreds. Determining the appropriate hidden layer configuration usually requires experimental adjustment. In these layers, it is common to use nonlinear activation functions (such as ReLU and its variants Leaky ReLU, ELU, etc.). The introduction of these activation functions enables the network to capture more complex nonlinear relationships, thereby improving the depth and breadth of learning.

[0065] (3) Output layer: The number of neurons in the output layer should be consistent with the size of the action space. This means that the number of neurons in the output layer should be equal to the total number of possible alarm actions in the platform. This design ensures that the network can generate a Q value for each possible decision, that is, the expected reward of the action, thus providing a basis for making the optimal decision.

[0066] After the network structure design is completed, two Q networks with the same structure are established, namely the main Q network and the target Q network, and their parameters are randomly initialized. The main Q network is used to generate the Q value prediction of the action, while the target Q network is used to calculate the target Q value in the loss function to help stabilize the training process.

[0067] In the deep Q network (DQN) training process, the loss function is the core indicator for evaluating the difference between the model's predicted value and the true value. It guides the optimization process of the model parameters to reduce the prediction error and thus enhance the model performance. Specifically, for a certain experience tuple in the experience pool (state S, action A, reward R, next state S'), DQN uses the main network to estimate the Q value of the current state and action (Q reality), and uses the target network to calculate the maximum expected reward of the next state (Q target). TD error, as the core of the loss function, is the difference between these two Q values, reflecting the deviation between the estimated action value and the actual experience.

[0068] This TD error-based loss function not only enables the deep Q-network to accurately learn the optimal action-value function, but also ensures that the strategy is continuously optimized towards the goal of maximizing the accumulated future rewards, thereby achieving efficient task management in the monitoring and alarm system.

[0069] Step 6: Periodically copy the parameters of the main Q network to the target Q network. This step helps reduce volatility in training and ensures the stability of the learning goal by using a relatively stable set of parameters to calculate the target Q value. During the training cycle, the system periodically executes the steps of extracting samples from the experience pool, calculating losses, and updating network parameters. Through such iterations, the temporal difference (TD) error gradually decreases, allowing DQN to more accurately learn which monitoring and alarm actions can bring the greatest expected rewards under a given cluster state. As training progresses, the main Q network's prediction of the action value will become closer and closer to the actual situation, allowing the computer cluster monitoring and alarm system to select the optimal monitoring and alarm action when facing a specific state.

Claims

1. A computer cluster monitoring and alarm method based on DQN algorithm, characterized in that: Step 1: Collect monitoring data in real time in the computer cluster status monitoring system and pre-process the collected data; Step 2: Perform feature extraction based on the data collected and preprocessed in step 1 to extract features useful for alarm rules; Step 3: Associate the monitoring data with the actual alarm time, that is, mark the time periods in which situations requiring alarms occurred, and finally complete the construction of the experience pool; Step 4: Build a DQN network and initialize DQN parameters, where DQN stands for DeepQ-Network. Input the data in the experience pool into the Q network for training, and optimize the DQN parameters by using the stochastic gradient descent algorithm until the Q value corresponding to the maximum Q value action tends to be stable, thus obtaining a trained Q network. Step 5: Deploy the trained model to the monitoring alarm system, and output the Q value of each action under the current state s according to the Q network trained in step 4 to determine whether the monitoring alarm should be triggered.

2. According to the computer cluster monitoring and alarm method based on the DQN algorithm according to claim 1, it is characterized in that: The step 3 also includes: Step 3.1: Define the state space of the cluster using the data obtained in steps 1 and 2; Step 3.2: Define the action space of the cluster, covering all actions that the deep reinforcement learning agent can perform; Step 3.3: Design rewards. When the alarm system takes correct actions, it will get higher reward scores. The reward design is also related to the time and degree of impact of the system.

3. According to claim 2, a computer cluster monitoring and alarm method based on the DQN algorithm is characterized in that: The process of defining the state space in step 3.1 is as follows: Based on the computer cluster monitoring and alarm system built in step 1, the monitoring data of the computing nodes is extracted from the database on its control node, and the monitoring data is set as a binary array s[s1,s2,s3...s n ], where s1, s2, s3...s n Corresponding to the state s on each computer node respectively; the expression of the state s is Where n represents the nth node in the computer cluster, t n Indicates the corresponding time of state s on the current node; c, m, d, u, g respectively represent the current time t n The number of CPU cores, memory, hard disk, network bandwidth, and GPU cores available for the computing node. z represents the current time t n The remaining value coefficient of the current time t n The calculation formula of the remaining value coefficient z of the node resource is as follows: Among them, c′, m′, d′, u′, and g′ respectively represent the total number of CPU cores, memory, hard disk, network bandwidth, and GPU cores on the computing node. The content in s is modified accordingly according to the feature value extracted in step 2 to suit its corresponding cluster.

4. According to claim 2, a computer cluster monitoring and alarm method based on the DQN algorithm is characterized in that: The method for defining the action space in step 3.2 is as follows: The action space contains all the actions that the agent can perform in deep reinforcement learning and is set to the following structure: Where n1, n2, n t represents a node, A, B, …, X represents the alarm level issued or no alarm; the action space is adjusted accordingly according to the characteristics and work requirements of different clusters.

5. According to claim 2, a computer cluster monitoring and alarm method based on DQN algorithm is characterized in that: The method of reward design in step 3.3 is as follows: Among them, r is the reward value of a single action, t′ is the time when the system failure starts, t is the time when the failure ends, e is the overall work efficiency of the system during the failure, and the minimum value of e is manually determined based on the computer cluster to limit the reward value to a reasonable range. a is the basic reward coefficient.

Images