Monitoring method, device and computer-readable storage medium for containers in a cluster

Abstract

The present disclosure relates to a monitoring method, device and computer-readable storage medium for containers in a cluster, and to the field of cloud computing technology. The method of the present disclosure includes: in response to changes in relevant indicators of service nodes and containers in the cluster, obtaining indicators that have changed; comparing the indicators that have changed with the state tree of the cluster, and determining whether there is a container in the cluster whose state has changed to an abnormal state, wherein the state tree includes the state of each container under each service node in the cluster and the collection time of the state; in the case that there is a container in the cluster whose state has changed to an abnormal state, updating the state tree, and extracting the identifier of the container and the collection time of the abnormal state; obtaining the parameters of the container at the collection time of the abnormal state according to the identifier of the container, and the parameters of the container at each historical moment in the preset historical time period before the collection time of the abnormal state, to form time series data; according to the time series data, determining whether the container is abnormal and the type of abnormality.

Description

Technical Field

[0001] The present disclosure relates to the technical field of cloud computing, and in particular to a method, device, and computer-readable storage medium for monitoring containers in a cluster. Background Art

[0002] In the current cloud computing and cloud-to-cloud hosting scenarios where cloud-network integration and cloud-edge collaboration are integrated, operators have actually deployed multiple sets of core-edge cloud computing cluster scenarios, and deployed control nodes and control systems in the core to uniformly manage Kubernetes container clusters of different scenarios and architectures in the cloud-edge. According to business and operation and maintenance needs, monitoring the cluster status has become an inevitable and urgent need. As a CNCF (Cloud Native Computing Foundation) graduated project, Prometheus has actually been integrated into the Kubernetes cluster and become the de facto standard for monitoring components, providing comprehensive indicator collection and aggregation for cluster internal indicator monitoring.

[0003] At present, the scenarios of operators' business platforms are faced with the requirements of high peak access and high reliability of multiple services. The deployment methods of services include multiple edges, multiple clusters, and a mixture of bare metal and virtual machines. If there are problems with containers in the cluster, such as untimely response and unreachable containers when the container is automatically pulled up, the Prometheus monitoring component can be used to generate alarms, and then the cluster disaster recovery mechanism can be used, usually by deleting the node and restarting it. Summary of the invention

[0004] The inventors found that in multi-cluster scenarios, the indicators of clusters with different architectures are not universal, and it is difficult to accurately identify abnormal situations through unified abnormality identification criteria. When a container has a problem, the cluster automatically restarts the container, which causes information loss and makes it impossible to accurately analyze the abnormal situation of the container.

[0005] A technical problem to be solved by the present disclosure is: how to improve the accuracy of determining abnormal conditions of a container.

[0006] According to some embodiments of the present disclosure, a method for monitoring containers in a cluster is provided, comprising: in response to changes in relevant indicators of service nodes and containers in the cluster, obtaining the indicators that have changed; comparing the indicators that have changed with the status tree of the cluster, and determining whether there is a container in the cluster whose status has changed to an abnormal state, wherein the status tree includes the status of each container under each service node in the cluster and the collection time of the status; in the case that there is a container in the cluster whose status has changed to an abnormal state, updating the status tree, and extracting the identifier of the container and the collection time of the abnormal state; obtaining parameters of the container at the collection time of the abnormal state according to the identifier of the container, and parameters of the container at each historical time in a preset historical time period before the collection time of the abnormal state, to form time series data; and determining whether the container is abnormal and the type of abnormality according to the time series data.

[0007] In some embodiments, obtaining indicators that have changed in related indicators of service nodes and containers in the cluster includes: receiving alarm information from a Prometheus monitoring component; and determining the indicators that have changed according to the alarm information.

[0008] In some embodiments, comparing the indicator that has changed with the status tree of the cluster includes: searching for a tree node corresponding to the indicator that has changed in the status tree of the cluster according to the attributes of the indicator that has changed; and comparing the value of the indicator that has changed with the value of the corresponding tree node to determine whether there is a difference.

[0009] In some embodiments, determining whether there is a container in the cluster whose state has changed to an abnormal state includes: when there is a difference between the value of the indicator that has changed and the value of the corresponding tree node, comparing the value of the indicator that has changed with a preset abnormal judgment condition to determine whether the state of the container related to the indicator that has changed has changed to an abnormal state.

[0010] In some embodiments, obtaining the parameters of the container at the collection moment of the abnormal state and the parameters of the container at each historical moment in a preset historical time period before the collection moment of the abnormal state according to the identifier of the container includes: searching the parameters of the container at the collection moment of the abnormal state and the parameters of the container at each historical moment in the preset historical time period before the collection moment of the abnormal state from a parameter library generated after the Prometheus monitoring component collects all parameters of the cluster according to the identifier of the container.

[0011] In some embodiments, the method further includes: traversing each service node and each container in the cluster to obtain an identifier of each service node, an identifier of each container, and a status of each container; and constructing a status tree according to the identifier of each service node, the identifier of each container, and the status of each container and the time when the status was collected.

[0012] In some embodiments, constructing a state tree based on the identifier of each service node, the identifier of each container, and the state of each container and the time when the state is collected includes: generating a root node representing the cluster based on the identifier of the cluster; generating a tree node representing each service node based on the identifier of each service node and using it as a child node of the root node; for each service node, generating a tree node representing each container based on the identifier of each container under the service node and using it as a child node of the tree node corresponding to the service node; for each container, generating a child node of the tree node corresponding to the container based on the state of the container and the time when the state is collected.

[0013] In some embodiments, determining whether the container is abnormal and the type of abnormality based on the time series data includes: inputting the time series data into a pre-trained machine learning model to obtain an output result of whether the container is abnormal and the type of abnormality.

[0014] In some embodiments, the method further includes: storing the identification of the container, the abnormal state, the abnormal type, and the collection time of the abnormal state in association with each other.

[0015] In some embodiments, the parameter of the container includes at least one of: CPU usage, memory usage, disk write speed, and application program interface (API) throughput of the container.

[0016] According to some other embodiments of the present disclosure, a monitoring device for containers in a cluster is provided, including:

[0017] An indicator acquisition module is used to respond to changes in relevant indicators of service nodes and containers in the cluster and acquire the changed indicators; a comparison module is used to compare the changed indicators with the state tree of the cluster and determine whether there is a container in the cluster whose state has changed to an abnormal state, wherein the state tree includes the state of each container under each service node in the cluster and the collection time of the state; an update module is used to update the state tree when there is a container in the cluster whose state has changed to an abnormal state, and extract the identifier of the container and the collection time of the abnormal state; a parameter acquisition module is used to acquire the parameters of the container at the collection time of the abnormal state according to the identifier of the container, and the parameters of the container at each historical time in a preset historical time period before the collection time of the abnormal state, to form time series data; a determination module is used to determine whether the container is abnormal and the type of abnormality according to the time series data.

[0018] According to some further embodiments of the present disclosure, a device for monitoring containers in a cluster is provided, comprising: a processor; and a memory coupled to the processor, for storing instructions, which, when executed by the processor, causes the processor to execute a method for monitoring containers in a cluster as described in any of the aforementioned embodiments.

[0019] According to some further embodiments of the present disclosure, a transient computer-readable storage medium is provided, on which a computer program is stored, wherein when the program is executed by a processor, the method for monitoring containers in a cluster of any of the aforementioned embodiments is implemented.

[0020] In the method disclosed in the present invention, a state tree can be established for each cluster in a multi-cluster scenario. When the indicator that has changed is obtained, the indicator that has changed is compared with the state tree of the cluster, and it is determined whether there is a container in the cluster whose state has changed to an abnormal state. If so, the state tree is updated and the collection time of the abnormal state of the container and the parameters of the container at each historical moment in the previously preset historical time period are obtained to form time series data, and then the container is determined to be abnormal and the type of abnormality based on the time series data. Since each cluster is represented by a state tree and the state tree is dynamically updated, the abnormal situation of the container can be judged independently for each cluster, and the abnormal state of the container is analyzed in combination with the time series data within a period of historical time, which improves the accuracy of determining the abnormal situation of the container. Once the relevant indicators of the service node and the container change, the container in the abnormal state can be quickly located and the relevant parameters can be obtained according to the method disclosed in the present invention, and the abnormal situation can be identified to avoid data loss caused by container restart, thereby improving the accuracy of determining the abnormal situation of the container.

[0021] Other features and advantages of the present disclosure will become apparent from the following detailed description of exemplary embodiments of the present disclosure with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the embodiments of the present disclosure or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0023] Figure 1 A schematic flow chart showing a method for monitoring containers in a cluster according to some embodiments of the present disclosure.

[0024] Figure 2 A schematic diagram illustrating a state tree of a cluster of some embodiments of the present disclosure.

[0025] Figure 3 A schematic flow chart showing a method for monitoring containers in a cluster according to other embodiments of the present disclosure.

[0026] Figure 4 A schematic flow chart showing a method for monitoring containers in a cluster according to yet other embodiments of the present disclosure.

[0027] Figure 5A schematic diagram showing the structure of a monitoring device for containers in a cluster according to some embodiments of the present disclosure.

[0028] Figure 6 A schematic diagram showing the structure of a monitoring device for containers in a cluster according to other embodiments of the present disclosure.

[0029] Figure 7 A schematic diagram showing the structure of a monitoring device for containers in a cluster according to some other embodiments of the present disclosure. DETAILED DESCRIPTION

[0030] The technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are only part of the embodiments of the present disclosure, rather than all of the embodiments. The following description of at least one exemplary embodiment is actually only illustrative and is by no means intended to limit the present disclosure and its application or use. Based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present disclosure.

[0031] This disclosure proposes a method for monitoring containers in a cluster. Figures 1 to 4 Give a description.

[0032] Figure 1 The following is a flow chart of some embodiments of the method for monitoring containers in a cluster disclosed in the present invention. Figure 1 As shown, the method of this embodiment includes: steps S102 to S110.

[0033] In step S102, in response to changes in relevant indicators of service nodes and containers in the cluster, the changed indicators are obtained.

[0034] Each cluster includes multiple service nodes (Node), and one or more container groups (Pod) can be deployed on each service node, and each container group can include one or more containers (Container). In some embodiments, the alarm information of the Prometheus monitoring component is received, and the indicator that has changed is determined based on the alarm information. The alarm conditions in Prometheus can be set according to the information that actually needs to be monitored. When the alarm conditions are met, Prometheus issues an alarm message. For example, when the container cannot be connected, when the container is migrated, when the node is deleted, etc., Prometheus issues an alarm message.

[0035] The relevant indicators of service nodes and containers include but are not limited to: at least one of the number of service nodes (node_sum), the number of containers (container_sum), the status of service nodes (node_status), and the status of containers (container_status). The indicator that has changed and the value of the indicator can be determined based on the alarm information. For example, when the container cannot be connected, the indicator that has changed is determined to be the status of the container based on the alarm information issued by Prometheus, and the value of the indicator is unhealthy.

[0036] In step S104, the changed indicator is compared with the state tree of the cluster, and it is determined whether there is a container in the cluster whose state has changed to an abnormal state.

[0037] The cluster status tree includes the status of each container under each service node in the cluster and the time when the status was collected. Figure 2 As shown, the root node of the state tree represents the cluster, and the value of the root node can be set to the identifier of the cluster. The child nodes of the root node represent each service node, and the value of each child node of the root node can be set to the identifier of the corresponding service node. The child nodes of the tree node corresponding to each service node are the tree nodes corresponding to each container deployed by the service node, and the value of the tree node corresponding to each container can be set to the identifier of the container. The value of the tree node (leaf node) corresponding to each container includes the state of the container and the time when the state is collected.

[0038] In some embodiments, according to the attribute of the indicator that has changed, the tree node corresponding to the indicator that has changed is searched in the state tree of the cluster; the value of the indicator that has changed is compared with the value of the corresponding tree node to determine whether there is a difference. For example, the attribute of the indicator includes the identifier of the object to which the indicator belongs (the identifier of the service node or the identifier of the container). For example, when the indicator that has changed is the state of the container and the value of the indicator is unhealthy, the tree node of the container corresponding to the indicator that has changed is searched in the state tree, and the state of the container in the tree node is compared with the value of the indicator to determine whether there is a difference.

[0039] In some embodiments, when there is a difference between the value of the indicator that has changed and the value of the corresponding tree node, the value of the indicator that has changed is compared with the preset abnormal judgment condition to determine whether the state of the container related to the indicator that has changed has changed to an abnormal state. For example, if the value of the indicator is an unhealthy state and the state of the container in the state tree is normal, there is a difference, and then it is determined whether the unhealthy state is a preset abnormal state (the preset abnormal judgment condition, for example, includes the preset abnormal state). For another example, if the value of the indicator that has changed is a service node restart, and the service node restart meets the preset abnormal judgment condition, it is determined that the state of the container related to the service node has changed to an abnormal state.

[0040] The container status includes, for example, healthy, normal, restarting, paused, deleting, unhealthy, etc., but is not limited to the examples. For example, restarting, deleting, and unhealthy are preset abnormal states, while healthy, normal, and paused are not preset abnormal states.

[0041] In step S106, when there is a container in the cluster whose state is changed to an abnormal state, the state tree is updated, and the identifier of the container and the collection time of the abnormal state are extracted.

[0042] When updating the status tree, you only need to update the corresponding tree nodes according to the indicators that have changed, and there is no need to regenerate the status tree, which improves processing efficiency. If there are no containers in the cluster whose status has changed to an abnormal state, there is no need to update the status tree.

[0043] In addition to the situation where there is a container in the cluster whose state has changed to an abnormal state and the state tree needs to be updated, there are other situations in which the state tree can be updated. In some embodiments, when there is a difference between the value of the indicator that has changed and the value of the corresponding tree node, the value of the indicator that has changed is compared with the preset update condition. If the preset update condition is met, the state tree is updated. For example, if the value of the indicator that has changed is a new service node or a new container, etc., and the preset update condition is met, the state tree is updated. The preset update condition can be set according to the changes in the tree nodes in the actual state tree.

[0044] When there is a container in the cluster whose state changes to an abnormal state, the container identifier, the collection time of the abnormal state, and the abnormal state of the container can be thrown out after the state tree is updated. These data can be stored in the form of a list for subsequent application. The container identifier can uniquely identify a container. For example, the container can be uniquely identified by the cluster ID, service node ID and container ID. The container identifier can include the cluster ID, service node ID and container ID, and is not limited to the examples given.

[0045] In step S108, the parameters of the container at the collection time of the abnormal state and the parameters of the containers at each historical moment in a preset historical time period before the collection time of the abnormal state are obtained according to the identifier of the container to form time series data.

[0046] In some embodiments, the parameters of the container at the time of collection of the abnormal state are searched from the parameter library generated after the Prometheus monitoring component collects all the parameters of the cluster according to the identifier of the container, as well as the parameters of the containers at each historical moment in the preset historical time period before the collection moment of the abnormal state. The preset historical time period is, for example, one hour or 10 minutes before the collection moment of the abnormal state, and is not limited to the examples given. All parameters that can be queried by traversing the identifier of the container under Prometheus are divided according to the collection moment of the abnormal state and the preset historical time period, and the collection moment of the abnormal state and the parameters of the containers at each historical moment in the preset historical time period before the collection moment of the abnormal state are obtained, which can be output to s3 storage.

[0047] The container parameters include, for example, at least one of the container's CPU usage, memory usage, disk write speed, and application program interface (API) throughput.

[0048] In step S110, whether the container is abnormal and the type of abnormality are determined based on the time series data.

[0049] In some embodiments, the time series data is input into a pre-trained machine learning model to obtain the result of whether the output container is abnormal and the abnormal type. The collection time of the abnormal state and the parameters of the containers at each historical moment in the preset historical time period before the collection time of the abnormal state can be derived from s3 and input into the machine learning model. The machine learning model is, for example, VAE (variational autoencoder). The abnormal type is, for example, disk congestion, high CPU occupancy, etc., and the abnormal type can be set according to actual needs during model training, not limited to the examples given. The result output by the machine learning model is a numerical value, which can be semantically processed and converted into readable text. For example, the result output by the machine learning model is processed as API, the abnormal value is 0.45, and the abnormal interval is 2021-10-27-19-48-4430~2021-10-27-19-50-8700, that is, the API of the container is abnormal, and the abnormal interval represents the abnormal time period.

[0050] In some embodiments, the identification, abnormal state, abnormal type and collection time of the abnormal state of the container are stored in association, and data such as abnormal values ​​and abnormal intervals output by the machine learning model can also be stored in association.

[0051] In the method of the above embodiment, in a multi-cluster scenario, a state tree can be established for each cluster. When the indicator that has changed is obtained, the indicator that has changed is compared with the state tree of the cluster, and it is determined whether there is a container in the cluster whose state has changed to an abnormal state. If so, the state tree is updated and the collection time of the abnormal state of the container and the parameters of the container at each historical moment in the previously preset historical time period are obtained to form time series data, and then the container is determined to be abnormal and the type of abnormality according to the time series data. Since each cluster is represented by a state tree and the state tree is dynamically updated, the abnormal situation of the container can be judged independently for each cluster, and the abnormal state of the container is analyzed in combination with the time series data within a period of historical time, which improves the accuracy of determining the abnormal situation of the container. Once the relevant indicators of the service node and the container change, the container in the abnormal state can be quickly located and the relevant parameters can be obtained according to the method of the above embodiment, and the abnormal situation can be identified to avoid data loss caused by container restart, thereby improving the accuracy of determining the abnormal situation of the container.

[0052] Combine the following Figure 3 Other embodiments of the method for monitoring containers in a cluster of the present disclosure are described.

[0053] Figure 3 Flowcharts of other embodiments of the method for monitoring containers in a cluster disclosed herein. Figure 3 As shown, before step S102, the following steps are included: steps S302 to S304.

[0054] In step S302, each service node and each container in the cluster is traversed to obtain the identifier of each service node, the identifier of each container, and the status of each container.

[0055] In step S304, a state tree is constructed according to the identifier of each service node, the identifier of each container, the state of each container and the time of collecting the state.

[0056] For example, you can align the Prometheus data interface to obtain the number of service nodes and containers in the cluster, and traverse all service nodes and containers in the cluster to build a status tree based on the number of service nodes and containers. For example, generate a root node representing the cluster based on the cluster identifier; generate tree nodes representing each service node based on the identifier of each service node, and use them as child nodes of the root node; for each service node, generate tree nodes representing each container based on the identifier of each container under the service node, and use them as child nodes of the tree node corresponding to the service node; for each container, generate child nodes of the tree node corresponding to the container based on the container's status and the time when the status was collected. The status tree is as follows: Figure 2 shown.

[0057] Combine the following Figure 4Some application examples of the method for monitoring containers in a cluster disclosed in the present invention are described.

[0058] Figure 4 Flowcharts of some other embodiments of the method for monitoring containers in a cluster disclosed herein. Figure 4 As shown, the method of this embodiment includes: steps S402 to S418.

[0059] In step S402, the changed indicators are collected from Prometheus.

[0060] In step S404, the changed indicator is compared with the state tree of the cluster.

[0061] In step S406, the value of the changed indicator is compared with the preset abnormality judgment condition. If the preset abnormality judgment condition is met, step S408 is executed, otherwise step S409 is executed.

[0062] In step S408, the status tree is updated.

[0063] In step S409, the state tree is not updated, and the process returns to step S402.

[0064] In step S410, the identifier of the container whose state is changed to an abnormal state and the collection time of the abnormal state are thrown out.

[0065] In step S412, the parameters of the container at the collection time of the abnormal state and the parameters of the containers at each historical moment in a preset historical time period before the collection time of the abnormal state are searched from the parameter library according to the identifier of the container.

[0066] In step S414, the parameters of the collection moment container of the abnormal state and the parameters of each historical moment container in a preset historical time period before the collection moment of the abnormal state are input into the machine learning model.

[0067] In step S416, it is determined whether the container is abnormal and the type of abnormality and stored.

[0068] In step S418, the parameters of the collection time container of the abnormal state and the parameters of each historical time container in a preset historical time period before the collection time of the abnormal state are stored.

[0069] It can also store relevant data such as abnormal status of the container.

[0070] The disclosed solution performs tree topology modeling on subordinate service nodes and containers of the federated cluster, generates a state tree, and tracks the dynamic changes of the state tree; Prometheus-related monitoring parameters are marked according to the abnormal state in the state tree, and a machine learning model is imported to identify abnormal conditions of the container. The disclosed solution can track the dynamics of cluster topology, locate and throw abnormal containers or node positions, and effectively support abnormal location and abnormal processing. A time positioning mechanism is provided for abnormal containers that cannot continue to collect data, and past data is traced back for processing and analysis, time series data is actively screened, and all collectible data is processed to locate the time and type of abnormality. A high-performance state tree update mechanism, local updates to reduce computing loss, and O(1) complexity. The disclosed solution can provide support for the operation and maintenance of multi-container clusters, fine tracking of container-level accuracy, and unified management and control under massive applications. Provide positioning capabilities for perception of abnormalities. An abnormal state library can be established to save the occurrence of existing abnormalities so that subsequent intelligent operation and maintenance can conduct subsequent mining and analysis of data. For the intelligent operation and maintenance scenarios of data centers, a cluster-level fault occurrence library is provided, which can be continuously updated. In the field of stream processing, real-time data comparison is performed to provide real-time prediction capabilities for disaster recovery in existing cluster scenarios.

[0071] The disclosed solution is applied to a multi-container cluster management cloud platform to monitor network element services of physical network cards mapped to virtual network cards, locate network element application failures in a timely manner and reproduce problems, thereby greatly improving the performance of the cluster.

[0072] The present disclosure also provides a monitoring device for containers in a cluster. Figure 5 Give a description.

[0073] Figure 5 FIG. 1 is a structural diagram of some embodiments of the monitoring device for containers in the disclosed cluster. Figure 5 As shown, the device 50 of this embodiment includes: an indicator acquisition module 510, a comparison module 520, an update module 530, a parameter acquisition module 540, and a determination module 550.

[0074] The indicator acquisition module 510 is used to respond to changes in related indicators of service nodes and containers in the cluster and acquire the changed indicators.

[0075] In some embodiments, the indicator acquisition module 510 is used to receive alarm information from the Prometheus monitoring component; and determine the indicator that has changed based on the alarm information.

[0076] The comparison module 520 is used to compare the changed indicator with the state tree of the cluster and determine whether there is a container in the cluster whose state has changed to an abnormal state, wherein the state tree includes the state of each container under each service node in the cluster and the time when the state was collected.

[0077] In some embodiments, the comparison module 520 is used to search for a tree node corresponding to a changed indicator in the state tree of the cluster according to the attributes of the changed indicator; and compare the value of the changed indicator with the value of the corresponding tree node to determine whether there is a difference.

[0078] In some embodiments, the comparison module 520 is used to compare the value of the changed indicator with a preset abnormal judgment condition when there is a difference between the value of the changed indicator and the value of the corresponding tree node, to determine whether the state of the container related to the changed indicator has changed to an abnormal state.

[0079] The updating module 530 is used to update the state tree and extract the identifier of the container and the collection time of the abnormal state when there is a container whose state is changed to an abnormal state in the cluster.

[0080] The parameter acquisition module 540 is used to acquire the parameters of the container at the collection moment of the abnormal state according to the identifier of the container, and the parameters of each historical moment container in a preset historical time period before the collection moment of the abnormal state, to form time series data.

[0081] In some embodiments, the parameter acquisition module 540 is used to search for the parameters of the container at the collection moment of the abnormal state from the parameter library generated after the Prometheus monitoring component collects all the parameters of the cluster according to the identifier of the container, as well as the parameters of the containers at each historical moment in a preset historical time period before the collection moment of the abnormal state.

[0082] The determination module 550 is used to determine whether the container is abnormal and the type of abnormality according to the time series data.

[0083] In some embodiments, the determination module 550 is used to input the time series data into a pre-trained machine learning model to obtain an output result of whether the container is abnormal and the type of abnormality.

[0084] In some embodiments, the parameter of the container includes at least one of: CPU usage, memory usage, disk write speed, and application program interface (API) throughput of the container.

[0085] In some embodiments, the device 50 also includes: a construction module 560, which is used to traverse each service node and each container in the cluster, obtain the identification of each service node, the identification of each container and the status of each container; and construct a status tree according to the identification of each service node, the identification of each container, the status of each container and the time of state collection.

[0086] In some embodiments, the construction module 560 is used to generate a root node representing the cluster according to the identifier of the cluster; generate a tree node representing each service node according to the identifier of each service node, and use it as a child node of the root node; for each service node, generate a tree node representing each container according to the identifier of each container under the service node, and use it as a child node of the tree node corresponding to the service node; for each container, generate a child node of the tree node corresponding to the container according to the state of the container and the time when the state was collected.

[0087] In some embodiments, the device 50 further includes: a storage module 570 for storing the identification of the container, the abnormal state, the abnormal type and the collection time of the abnormal state in association with each other.

[0088] The monitoring device for containers in a cluster in the embodiments of the present disclosure can be implemented by various computing devices or computer systems. Figure 6 as well as Figure 5 Give a description.

[0089] Figure 6 FIG. 1 is a structural diagram of some embodiments of the monitoring device for containers in the disclosed cluster. Figure 6 As shown, the device 60 of this embodiment includes: a memory 610 and a processor 620 coupled to the memory 610, and the processor 620 is configured to execute the monitoring method of the container in the cluster in any of some embodiments of the present disclosure based on the instructions stored in the memory 610.

[0090] The memory 610 may include, for example, a system memory, a fixed non-volatile storage medium, etc. The system memory may store, for example, an operating system, an application program, a boot loader, a database, and other programs.

[0091] Figure 7 FIG. 1 is a structural diagram of some other embodiments of the monitoring device for containers in the cluster disclosed in the present invention. Figure 7 As shown, the device 70 of this embodiment includes: a memory 710 and a processor 720, which are similar to the memory 610 and the processor 620 respectively. It can also include an input and output interface 730, a network interface 740, a storage interface 750, etc. These interfaces 730, 740, 750 and the memory 710 and the processor 720 can be connected, for example, through a bus 760. Among them, the input and output interface 730 provides a connection interface for input and output devices such as a display, a mouse, a keyboard, and a touch screen. The network interface 740 provides a connection interface for various networked devices, for example, it can be connected to a database server or a cloud storage server. The storage interface 750 provides a connection interface for external storage devices such as SD cards and USB flash drives.

[0092] Those skilled in the art will appreciate that the embodiments of the present disclosure may be provided as methods, systems, or computer program products. Therefore, the present disclosure may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present disclosure may take the form of a computer program product implemented on one or more computer-usable non-transient storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0093] The present disclosure is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present disclosure. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0094] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0095] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0096] The above description is only a preferred embodiment of the present disclosure and is not intended to limit the present disclosure. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present disclosure should be included in the protection scope of the present disclosure.

Claims

1. A method for monitoring containers in a cluster, comprising: In response to changes in relevant indicators of service nodes and containers in the cluster, obtain the indicators that have changed; Compare the changed indicator with the status tree of the cluster, and determine whether there is a container in the cluster whose status has changed to an abnormal state, wherein the status tree includes the status of each container under each service node in the cluster and the time when the status was collected; In the case that there is a container whose state is changed to an abnormal state in the cluster, the state tree is updated, and the identifier of the container whose state is changed to an abnormal state and the collection time of the abnormal state are extracted, wherein when the state tree is updated, only the corresponding tree node is updated according to the indicator that has changed; Acquire, according to the identifier of the container whose state is changed to the abnormal state, the parameters of the container whose state is changed to the abnormal state at the time of collecting the abnormal state, and the parameters of the container whose state is changed to the abnormal state at each historical moment in a preset historical time period before the time of collecting the abnormal state, to form time series data; Determine, according to the time series data, whether the container whose state changes to the abnormal state is abnormal and the type of abnormality, including: inputting the time series data into a pre-trained machine learning model to obtain an output result of whether the container whose state changes to the abnormal state is abnormal and the type of abnormality.

2. The monitoring method according to claim 1, wherein: The indicators for obtaining changes include: Receive warning information from Prometheus monitoring components; An indicator of change is determined according to the alarm information.

3. The monitoring method according to claim 1, wherein: The comparing the changed indicator with the state tree of the cluster includes: According to the attribute of the changed indicator, searching the state tree of the cluster for a tree node corresponding to the changed indicator; The value of the changed indicator is compared with the value of the corresponding tree node to determine whether there is a difference.

4. The monitoring method according to claim 3, wherein: Determining whether there is a container in the cluster whose state has changed to an abnormal state includes: When there is a difference between the value of the changed indicator and the value of the corresponding tree node, the value of the changed indicator is compared with a preset abnormality judgment condition to determine whether the state of the container related to the changed indicator is changed to an abnormal state.

5. The monitoring method according to claim 1, wherein: The acquiring, according to the identifier of the container whose state is changed to the abnormal state, the parameters of the container whose state is changed to the abnormal state at the time of collecting the abnormal state, and the parameters of the container whose state is changed to the abnormal state at each historical moment in a preset historical time period before the time of collecting the abnormal state comprises: According to the identifier of the container whose state is changed to the abnormal state, the parameters of the container whose state is changed to the abnormal state at the time of collection of the abnormal state are searched from the parameter library generated after the Prometheus monitoring component collects all the parameters of the cluster, and the parameters of the container whose state is changed to the abnormal state at each historical moment in a preset historical time period before the time of collection of the abnormal state.

6. The monitoring method according to claim 1, further comprising: Traversing each service node and each container in the cluster, obtaining an identifier of each service node, an identifier of each container, and a status of each container; The state tree is constructed according to the identifier of each service node, the identifier of each container, the state of each container and the time when the state is collected.

7. The monitoring method according to claim 6, wherein: The step of constructing the state tree according to the identifier of each service node, the identifier of each container, the state of each container and the time of collecting the state includes: Generating a root node representing the cluster according to the identifier of the cluster; Generate a tree node representing each service node according to the identifier of each service node and use it as a child node of the root node; For each service node, a tree node representing each container is generated according to the identifier of each container under the service node, and the tree node is used as a child node of the tree node corresponding to the service node; For each container, a child node of the tree node corresponding to the container is generated according to the state of the container and the time when the state is collected.

8. The monitoring method according to any one of claims 1 to 7, further comprising: The identifier of the container whose state is changed to an abnormal state, the abnormal state, the abnormal type and the collection time of the abnormal state are associated and stored.

9. The monitoring method according to any one of claims 1 to 7, wherein: The parameter of the container whose state is changed to an abnormal state includes: at least one of the CPU usage, memory usage, disk writing speed, and application program interface API throughput of the container.

10. A monitoring device for containers in a cluster, comprising: An indicator acquisition module is used to respond to changes in related indicators of service nodes and containers in the cluster and obtain the indicators that have changed; A comparison module, used to compare the changed indicator with the state tree of the cluster, and determine whether there is a container in the cluster whose state has changed to an abnormal state, wherein the state tree includes the state of each container under each service node in the cluster and the time when the state was collected; An update module, configured to update the state tree and extract the identifier of the container whose state has changed to an abnormal state and the collection time of the abnormal state when there is a container whose state has changed to an abnormal state in the cluster, wherein when updating the state tree, only the corresponding tree node is updated according to the indicator that has changed; a parameter acquisition module, configured to acquire, according to the identifier of the container whose state changes to the abnormal state, the parameters of the container whose state changes to the abnormal state at the time of collection of the abnormal state, and the parameters of the container whose state changes to the abnormal state at each historical moment in a preset historical time period before the time of collection of the abnormal state, to form time series data; A determination module is used to determine whether the container whose state changes to an abnormal state is abnormal and the type of abnormality, including: inputting the time series data into a pre-trained machine learning model to obtain an output result of whether the container whose state changes to an abnormal state is abnormal and the type of abnormality.

11. A monitoring device for containers in a cluster, comprising: processor; as well as A memory coupled to the processor, for storing instructions, wherein when the instructions are executed by the processor, the processor executes the method for monitoring containers in a cluster according to any one of claims 1 to 9.

12. A non-transitory computer-readable storage medium having a computer program stored thereon, wherein: When the program is executed by a processor, the steps of the method according to any one of claims 1 to 9 are implemented.

Images