A distributed Bayesian cluster monitoring and early warning analysis method

Abstract

The present invention discloses a distributed Bayesian cluster monitoring early warning analysis method, including: a central server program executes a central early warning model training start instruction, and sends the instruction to a local cluster management node; when the cluster management node receives the instruction, it accesses the database deployed in the cluster management node, queries the identification field Tag, and determines whether to continue to execute; according to the business situation of a Node in the cluster managing multiple Pods, the log data of all Pods are classified, a Pod early warning model is constructed, and the Pod early warning probability is obtained; according to the business situation of a Master in the cluster managing multiple Nodes, the log data of all Nodes are analyzed, a Node early warning model is constructed, and the Node early warning probability is obtained; the Pod early warning probability and the Node early warning probability of each place are submitted to the central server, and the central early warning model is updated. The method of the present invention overcomes the model deviation problem and reduces the calculation complexity.

Description

Technical Field

[0001] The present invention belongs to the technical field of cluster monitoring and Bayesian early warning, and specifically relates to a distributed Bayesian cluster monitoring and early warning analysis method. Background Art

[0002] With the deepening of digital development, the number of operating equipment in various units has gradually increased, which is 10 to 100 times that of ten years ago. Even though operation and maintenance has been developing from manual operation and maintenance to tool operation and maintenance and platform operation and maintenance, the existing technology is difficult to solve the problem of accuracy of early warning of distributed cluster monitoring anomalies due to the large business scale, complex application relationships, multiple dependency levels and difficult problem troubleshooting in the computer room operation and maintenance scenarios. Summary of the invention

[0003] In view of the problems existing in the prior art, the present invention provides a distributed Bayesian cluster monitoring early warning analysis method. The Bayesian cluster monitoring early warning analysis method of the present invention improves the monitoring quality, can effectively overcome the model deviation problem, and reduce the computational complexity.

[0004] To achieve the above object, the present invention adopts the following technical solution: a distributed Bayesian cluster monitoring and early warning analysis method, which specifically includes the following steps:

[0005] Step 1: Execute the central early warning model training start instruction through the central server program deployed at the headquarters, and send the instruction to the local cluster management node;

[0006] Step 2: Build a Kubernetes-based cluster on the cluster management node. When the cluster management node receives the instruction, it accesses the database deployed on the cluster management node, queries the identification field Tag, and determines whether to execute step 3 based on the identification field Tag;

[0007] Step 3: According to the business situation of a Node managing multiple Pods in the Kubernetes cluster, the log data of all Pods managed on a Node is obtained by executing the program deployed on the Node local log server, the log data of all Pods are classified, a Pod warning model is built, and the Pod warning probability is obtained;

[0008] Step 4: According to the business situation of a Master managing multiple Nodes in the Kubernetes cluster, the log data of all Nodes managed by a Master is obtained by executing the program deployed on the Master local log server, the log data of all Nodes is analyzed, a Node warning model is built, and the Node warning probability is obtained;

[0009] Step 5: Submit the Pod warning probability and Node warning probability of each region to the central server and update the central warning model.

[0010] Furthermore, the Kubernetes-based cluster includes three objects: Master, Node, and Pod. The Master is used to control the machine of the Kubernetes node. The Node performs the assigned tasks under the control of the Master. The Pod is a collection of one or more containers, which is deployed as a whole on a Node.

[0011] Furthermore, in step 2, if the identification field Tag is not 0, it means that the central early warning model exists, and step three is executed; if the identification field Tag is 0, a request is sent to the central server to obtain the request instruction of the central early warning model. After receiving the request instruction, the central server sends the central early warning model to the cluster management node, and step three is executed.

[0012] Furthermore, the log data of the Pod includes: the CPU usage, memory usage, traffic size, and hard disk space size of the Pod.

[0013] Furthermore, the Pod early warning model is specifically:

[0014] Node(D|+) = Node (+|D) Node (D) / ( Node (+|D) Node (D)+ Node (+|N)Node (N))

[0015] Among them, Node (D) represents the probability that the historical alarm data of the Pod is close to the truth without considering the false alarm rate; Node (D|+) represents the probability that the historical alarm data of the Pod is close to the truth considering the false alarm rate; Node (+|D) represents the accuracy of the Pod alarm; Node (+|N) represents the false alarm rate of the Pod alarm data; Node (N) represents the reasonable probability of the Pod alarm.

[0016] Furthermore, the log data of the Node includes: the CPU usage, memory usage, traffic size, and hard disk space size of the Node.

[0017] Furthermore, the Node early warning model is specifically:

[0018] Master(D|+) = Master (+|D) Master (D) / ( Master (+|D) Master (D)+Master (+|N) Master (N))

[0019] Among them, Master (D) represents the probability that the historical alarm data of the Node is close to the truth without considering the false alarm rate; Master (D|+) represents the probability that the historical alarm data of the Node is close to the truth considering the false alarm rate; Master (+|D) represents the accuracy of the Node; Master (+|N) represents the false alarm rate of the Node's alarm data; Master (N) represents the reasonable probability of the Node alarm.

[0020] Compared with the prior art, the present invention has the following beneficial effects: the present invention, based on the distributed Bayesian cluster monitoring and early warning analysis method, highlights the algorithmic advantage of artificial intelligence in predicting the occurrence of alarms during cluster monitoring, improves the monitoring quality, and effectively solves the overall health status of the existing distributed cluster, as well as the network node equipment that may cause alarms, thereby reducing the losses caused to the cluster after a failure occurs. At the same time, it can also solve the problem that the distributed cluster monitors different local network services, and the privacy data is inconvenient to participate in the central early warning model training without affecting the effect of the central model training, and the distributed training can also greatly reduce the central early warning model training load. The present invention, based on the distributed Bayesian cluster monitoring and early warning analysis method, effectively overcomes the model bias problem and reduces the computational complexity. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 The present invention is a flow chart of the distributed Bayesian cluster monitoring and early warning analysis method. DETAILED DESCRIPTION

[0022] The technical solution of the present invention is further explained below in conjunction with the accompanying drawings.

[0023] The Bayesian cluster monitoring early warning analysis method of the present invention, before an abnormality occurs during cluster monitoring, analyzes the log data of the management node of the cluster, the cluster managed node server, and one or more POD points deployed on the node server, and monitoring indicators such as CPU, memory, and hard disk usage, and comprehensively obtains the node server early warning probability index, and then according to the index combined with the monitoring indicators such as CPU, memory, and hard disk usage of the node server, comprehensively obtains the cluster management node early warning probability index, specifically, the overall health status of the cluster is calculated by the Bayesian model twice, and the network node equipment that may be alarmed, thereby reducing the loss caused to the cluster after the failure occurs, making the model early warning probability more accurate and closer to the real business environment. Finally, the early warning probabilities of clusters in various places are uniformly uploaded to the central early warning model for training, and the entire distributed cluster monitoring early warning analysis is completed.

[0024] like Figure 1 The flowchart of the distributed Bayesian cluster monitoring and early warning analysis method of the present invention is as follows:

[0025] Step 1: Execute the central early warning model training start instruction through the central server program deployed at the headquarters, and send the instruction to the local cluster management node;

[0026] Step 2: Build a Kubernetes-based cluster on the cluster management node. The Kubernetes-based cluster in the present invention includes three objects: Master, Node, and Pod. The Master is used to control the machine of the Kubernetes node and is also used to create job tasks. The Node executes the assigned tasks under the control of the Master. The Pod is a collection of one or more containers, which are deployed as a whole on a Node. The containers in the same Pod share IP addresses, inter-process communication IPCs, host names, and other resources. The Pod abstracts the network and storage of the underlying container, making the migration of containers within the cluster more convenient. Considering the security of local privacy data, local exercise privacy data is used to perform associated data analysis and classification aggregation to form real and effective privacy analysis data. When the cluster management node receives the instruction, it accesses the database deployed on the cluster management node, queries the identification field Tag, and determines whether to execute step three according to the identification field Tag. Specifically, if the identification field Tag is not 0, it means that the central warning model exists, and step three is executed; if the identification field Tag is 0, a request is sent to the central server to obtain the request instruction of the central warning model. After receiving the request instruction, the central server sends the central warning model to the cluster management node and executes step three.

[0027] Since the structure of the distributed cluster is composed of different Node nodes, and the cluster business has a huge amount of data for business transactions through Pod. Therefore, in order to make the model training results more accurate and the warning effect closer to the real state, it is divided into two parts: Pod warning and Node warning. The overall health status of the cluster and the network node devices that may cause alarms are calculated through the secondary Bayesian model, thereby reducing the loss caused to the cluster after a failure occurs.

[0028] Step 3: According to the business situation of a Node managing multiple Pods in the Kubernetes cluster, the log data of all Pods managed on a Node is obtained by executing the program deployed on the Node local log server. The log data of the Pod in the present invention includes: the CPU usage rate, memory usage rate, traffic size, and hard disk space size of the Pod; the log data of all Pods are classified, a Pod early warning model is constructed, and the Pod early warning probability is obtained, which is responsible for predicting the running health of the Pod managed by each Node;

[0029] The Pod early warning model in the present invention is specifically:

[0030] Node(D|+) = Node (+|D) Node (D) / ( Node (+|D) Node (D)+ Node (+|N)Node (N))

[0031] Among them, Node (D) represents the probability that the historical alarm data of the Pod is close to the truth without considering the false alarm rate; Node (D|+) represents the probability that the historical alarm data of the Pod is close to the truth considering the false alarm rate; Node (+|D) represents the accuracy of the Pod alarm; Node (+|N) represents the false alarm rate of the Pod alarm data; Node (N) represents the reasonable probability of the Pod alarm.

[0032] Step 4: According to the business situation of a Master managing multiple Nodes in the Kubernetes cluster, the log data of all Nodes managed by a Master is obtained by executing the program deployed on the Master local log server. The log data of the Node in the present invention includes: the CPU usage rate, memory usage rate, traffic size, and hard disk space size of the Node. Analyze the log data of all Nodes, build a Node warning model, obtain the Node warning probability, and be responsible for predicting the operating health of each Node managed by the master;

[0033] The Node early warning model in the present invention is specifically:

[0034] Master(D|+) = Master (+|D) Master (D) / ( Master (+|D) Master (D)+Master (+|N) Master (N))

[0035] Among them, Master (D) represents the probability that the historical alarm data of the Node is close to the truth without considering the false alarm rate; Master (D|+) represents the probability that the historical alarm data of the Node is close to the truth considering the false alarm rate; Master (+|D) represents the accuracy of the Node; Master (+|N) represents the false alarm rate of the Node's alarm data; Master (N) represents the reasonable probability of the Node alarm.

[0036] The advantage of uploading only the warning probability of the local model to the central server through the above steps three and four is that, considering the security of local privacy data, some data do not want to be exposed to the central server. Therefore, local exercise privacy data is used for data analysis and calculation to form real and effective privacy analysis data protection.

[0037] Step 5: Submit the Pod warning probability and Node warning probability of each region to the central server and update the central warning model.

[0038] The present invention is based on a distributed Bayesian cluster monitoring and early warning analysis method to effectively solve the overall health status of existing distributed clusters and network node devices that may generate alarms, and realize rapid positioning and tracking when a fault occurs, thereby reducing the loss to the cluster after the fault occurs; at the same time, it can also solve the problem that distributed cluster monitoring has different local network services, and privacy data is inconvenient to participate in central early warning training without affecting the effect of central early warning model training. In addition, distributed training can also greatly reduce the training load of the central early warning model, effectively overcome the model bias problem, and reduce computational complexity.

[0039] The above are only preferred embodiments of the present invention, and the protection scope of the present invention is not limited to the above embodiments. All technical solutions under the concept of the present invention belong to the protection scope of the present invention. It should be pointed out that for ordinary technicians in this technical field, some improvements and modifications without departing from the principle of the present invention should be regarded as the protection scope of the present invention.

Claims

1. A distributed Bayesian cluster monitoring and early warning analysis method, characterized in that: The specific steps include: Step 1: Execute the central early warning model training start instruction through the central server program deployed at the headquarters, and send the instruction to the local cluster management node; Step 2: Build a Kubernetes-based cluster on the cluster management node. When the cluster management node receives the instruction, it accesses the database deployed on the cluster management node, queries the identification field Tag, and determines whether to execute step 3 based on the identification field Tag; Step 3: According to the business situation of a Node managing multiple Pods in the Kubernetes cluster, the log data of all Pods managed on a Node is obtained by executing the program deployed on the Node local log server, the log data of all Pods are classified, a Pod warning model is built, and the Pod warning probability is obtained; Step 4: According to the business situation of a Master managing multiple Nodes in the Kubernetes cluster, the log data of all Nodes managed by a Master is obtained by executing the program deployed on the Master local log server, the log data of all Nodes is analyzed, a Node warning model is built, and the Node warning probability is obtained; Step 5: Submit the Pod warning probability and Node warning probability of each region to the central server and update the central warning model.

2. The distributed Bayesian cluster monitoring and early warning analysis method according to claim 1 is characterized in that: The Kubernetes-based cluster includes three objects: Master, Node, and Pod. The Master is used to control the machine of the Kubernetes node. The Node performs the assigned tasks under the control of the Master. The Pod is a collection of one or more containers, which is deployed as a whole on a Node.

3. The distributed Bayesian cluster monitoring and early warning analysis method according to claim 1 is characterized in that: In step 2, if the identification field Tag is not 0, it means that the central warning model exists, and step 3 is executed; if the identification field Tag is 0, a request is sent to the central server to obtain the request instruction of the central warning model. After receiving the request instruction, the central server sends the central warning model to the cluster management node and step 3 is executed.

4. The distributed Bayesian cluster monitoring and early warning analysis method according to claim 1 is characterized in that: The log data of the Pod includes: the CPU usage, memory usage, traffic size, and hard disk space size of the Pod.

5. The distributed Bayesian cluster monitoring and early warning analysis method according to claim 1 is characterized in that: The Pod warning model is specifically: Node(D|+) = Node (+|D) Node (D) / ( Node (+|D) Node (D)+ Node (+|N) Node(N)) Among them, Node (D) represents the probability that the historical Pod alarm data is close to the truth without considering the false alarm rate; Node (D|+) represents the probability that the historical Pod alarm data is close to the truth considering the false alarm rate; Node (+|D) represents the accuracy of Pod alarms; Node (+|N) represents the false alarm rate of Pod alarm data; Node (N) represents the reasonable probability of Pod alarms.

6. The distributed Bayesian cluster monitoring and early warning analysis method according to claim 1 is characterized in that: The Node log data includes: the Node's CPU usage, memory usage, traffic size, and hard disk space size.

7. The distributed Bayesian cluster monitoring and early warning analysis method according to claim 1 is characterized in that: The Node early warning model is specifically: Master(D|+) = Master (+|D) Master (D) / ( Master (+|D) Master (D)+ Master(+|N) Master (N)) Among them, Master (D) represents the probability that the historical alarm data of the Node is close to the truth without considering the false alarm rate; Master (D|+) represents the probability that the historical alarm data of the Node is close to the truth considering the false alarm rate; Master (+|D) represents the accuracy of the Node; Master (+|N) represents the false alarm rate of the Node's alarm data; Master (N) represents the reasonable probability of the Node alarm.

Images