Multi-hierarchical cluster control device, multi-hierarchical cluster control method, and multi-hierarchical cluster control program

Abstract

To provide a multi-hierarchical cluster control device capable of appropriately arranging nodes of a plurality of clusters operating on a container cluster.SOLUTION: A multi-hierarchical cluster policy control device 100 is configured to: by a processor, receive, when deploying a processing cluster, deployment request information including requirements for processing cluster nodes (400, 500) and designation of associated clusters for each of multiple processing clusters; identify, on the basis of the requirements for the processing cluster nodes and the processing cluster nodes that constitute the associated clusters, a suitable container cluster node; transmit node identification information indicating the identified container cluster node as cluster policy information; identify a placement group that operates on the container cluster node within a predetermined range of the container cluster node on which the processing cluster node of the processing cluster and the associated cluster operate; and transmit information about the placement group to the processing cluster as process policy information.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a technology for controlling the placement of processing clusters configured on a container cluster that provides containers. [Background technology]

[0002] In recent years, there has been an increasing number of cases where application systems running on clouds or data centers are developed using containers, which are virtual environments that separate operating system (OS) processes using namespace technology. Furthermore, microservice-type architectures, which consist of multiple containers, communicate data between the containers, and process requested requests, are becoming more common. To run and manage multiple containers, there are an increasing number of cases where container clusters, such as Kubernetes (registered trademark), are used, which provide scalability and resilience by using multiple virtual machines (VMs) or servers.

[0003] Furthermore, with the expansion of AI (Artificial Intelligence) and big data analysis, there is a growing demand for running applications on container clusters to perform analysis. To perform analysis in a scalable manner, clusters for analysis, such as Apache Ray, which uses multiple nodes to execute analysis processes, are becoming more popular.

[0004] Container clusters are sometimes built across multiple availability zones for purposes such as HA (High-availability). In such cases, communication between containers must take place across availability zones, which can lead to problems such as high communication costs.

[0005] For example, techniques relating to the above problem are disclosed in Patent Documents 1 and 2 below.

[0006] Patent Document 1 describes a method for receiving a manifest of a container image of a container to be created, identifying a mapping index of a cluster of computing nodes, and selecting a computing node within the cluster of computing nodes for generating the container based on a comparison between the manifest and the mapping index.

[0007] Patent Document 2 discloses that "a method, an information processing system, and a computer program product manage server placement of virtual machines in an operating environment. A mapping of each virtual machine in a plurality of virtual machines to at least one server in a set of servers is determined. The mapping substantially satisfies a set of primary constraints associated with the set of servers. A plurality of virtual machine clusters are created. Each virtual machine cluster includes a set of virtual machines from the plurality of virtual machines. The server placement of one virtual machine in a cluster is interchangeable with the server placement of another virtual machine in the same cluster while satisfying a set of primary constraints. A server placement of the set of virtual machines in each virtual machine on at least one mapped server is generated for each cluster. The server placement substantially satisfies a set of secondary constraints." [Prior art documents] [Patent documents]

[0008] [Patent Document 1] US Patent Application Publication No. 2022 / 0147517 [Patent Document 2] U.S. Patent No. 8,478,878 Summary of the Invention [Problem to be solved by the invention]

[0009] In the technology of Patent Document 1, the node where a container is created is selected taking into consideration the container image, which is the content of the container. This makes it possible to reduce the communication costs required to move large-capacity containers. In addition, in Patent Document 2, the placement of VMs is changed taking into consideration constraints, making it possible to reduce communication costs.

[0010] However, these technologies cannot reduce communication costs in a system configuration where a cluster is built on top of a container cluster. For example, if an analysis cluster such as Ray (hereinafter referred to as the upper cluster) is built on a container cluster such as Kubernetes (hereinafter referred to as the lower cluster), the lower cluster manages which nodes in the lower cluster each container in the upper cluster runs on, but does not manage which containers run on the contents of the nodes in the upper cluster, i.e., the application processes and analysis processes of the upper cluster. On the other hand, the upper cluster manages which processes run on which nodes among the multiple nodes in the upper cluster that run as containers, but does not manage which nodes in the lower cluster each container in the upper cluster runs on. As a result, when multiple upper clusters run on top of a lower cluster and communicate with each other, the relative locations of the processes and data between the upper clusters are unknown.

[0011] Furthermore, when a container is restarted due to a process stop or a lack of central processing unit (CPU) on the node where it is running, or when the container is initially deployed, it is deployed to a worker node that satisfies the constraints. Because the data of each container is stored as a persistent volume and each container is virtualized, the upper cluster does not need to be aware of the lower cluster, and does not know that the node on which the container is running has changed.

[0012] As a result, it is not possible to determine the optimal location of containers in higher-level clusters or the optimal location of processes within higher-level clusters to reduce communication costs between higher-level clusters. Therefore, even if containers from two different higher-level clusters are running on the same or nearby nodes of lower-level clusters, depending on the situation, the processes may end up running on nodes of different or distant lower-level clusters. In such cases, the amount of long-distance communication between higher-level clusters becomes enormous, and response times become long.

[0013] The present invention has been made in view of the above circumstances, and an object of the present invention is to provide a technology that can appropriately arrange nodes of multiple clusters that operate on a container cluster. [Means for solving the problem]

[0014] In order to achieve the above object, a multi-hierarchical cluster control device according to one aspect is a multi-hierarchical cluster control device that controls a plurality of processing clusters that are operated on a container cluster, the multi-hierarchical cluster control device having a processor, and when deploying the processing cluster, the processor receives, for each of the plurality of processing clusters, deployment request information including requirements for the processing cluster nodes that constitute the processing cluster and specifications of associated clusters that are other processing clusters that have a connection relationship with the processing cluster, and identifies one or more container cluster nodes that are nodes of a container cluster that meet the requirements for the processing cluster nodes and the requirements for the processing cluster nodes that constitute the associated clusters, and transmits node identification information indicating the identified container cluster nodes to the container cluster that deploys the processing cluster as cluster policy information for determining a container cluster node when deploying the processing cluster in the container cluster, identifies one or more placement groups that are groups of processing cluster nodes that operate on container cluster nodes that are within a predetermined range of a container cluster node on which the processing cluster nodes of the processing cluster and the processing cluster nodes of the associated clusters operate, and transmits information on the identified placement groups to the processing cluster as process policy information for determining a processing cluster node on which a process is to be deployed. [Effects of the Invention]

[0015] According to the present invention, it is possible to appropriately arrange nodes of multiple clusters that operate on a container cluster. [Brief explanation of the drawings]

[0016] [Figure 1] FIG. 1 is a diagram showing the overall configuration of a computer system according to one embodiment. [Figure 2] FIG. 2 is a diagram illustrating an example of the configuration of node information of a lower cluster according to an embodiment. [Figure 3]FIG. 3 is a diagram illustrating an example of cluster deployment request information according to an embodiment. [Figure 4] FIG. 4 is a diagram illustrating another example of cluster deployment request information according to an embodiment. [Figure 5] FIG. 5 is a diagram illustrating an example of a cluster deploy manifest according to an embodiment. [Figure 6] FIG. 6 is a diagram illustrating another example of a cluster deploy manifest according to an embodiment. [Figure 7] FIG. 7 is a diagram illustrating a configuration of node mapping information according to an embodiment. [Figure 8] FIG. 8 is a diagram illustrating the configuration of process data location information according to an embodiment. [Figure 9] FIG. 9 is a configuration diagram of related cluster information according to an embodiment. [Figure 10] FIG. 10 is a configuration diagram of common node candidate information according to an embodiment. [Figure 11] FIG. 11 is a configuration diagram of process policy information according to an embodiment. [Figure 12] FIG. 12 is a sequence diagram of a cluster deployment process according to an embodiment. [Figure 13] FIG. 13 is a flowchart of a cluster policy information generation process according to an embodiment. [Figure 14] FIG. 14 is a flowchart of a process policy information generation process according to an embodiment. [Figure 15] FIG. 15 is a sequence diagram of an application deployment process according to an embodiment. [Figure 16] FIG. 16 is a sequence diagram of a container movement process according to an embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0017] The following description of the embodiments will be given with reference to the drawings. Note that the embodiments described below do not limit the scope of the invention as claimed, and not all of the elements and combinations thereof described in the embodiments are necessarily essential to the solution of the invention.

[0018] Fig. 1 is a diagram showing the overall configuration of a computer system according to one embodiment. Fig. 1 shows a state in which a higher-level cluster (processing cluster) is deployed and configured on a lower-level cluster, and if the higher-level cluster is not deployed, the configuration of the higher-level cluster does not exist.

[0019] The computer system 1 includes a multi-hierarchical cluster policy control device 100 as an example of a multi-hierarchical cluster control device, a lower-level cluster master node 200, one or more lower-level cluster worker nodes 300 (300-1, 300-2, ... 300-N), and a client 700. The multi-hierarchical cluster policy control device 100, the lower-level cluster master node 200, the lower-level cluster worker nodes 300, and the client 700 are connected via a computer network. The computer network is, for example, a LAN (Local Area Network) and / or WAN (Wide Area Network) connected by communication devices such as routers and switches.

[0020] The lower cluster master node 200 and the lower cluster worker node 300 configure a lower cluster. The lower cluster is, for example, a container cluster that provides containers, and can be configured using, for example, Kubernetes.

[0021] The lower cluster master node 200 is a node that manages a lower cluster. The lower cluster master node 200 may be configured, for example, as a physical server or as a VM (Virtual Machine). The lower cluster master node 200 includes a control manager 210, a scheduler 220, an API (Application Programming Interface) server 230, a data store 240, an agent 250, and a container runtime 260.

[0022] The control manager 210 grasps the resource usage rate, communication connection status, operating status of the host OS, operating status of the container, etc. of the lower cluster worker node 300, and determines whether to run a container and how many containers to run. The scheduler 220 determines the lower cluster worker node 300 on which to run a container. The API server 230 provides data stored in the data store 240 to the control manager 210, scheduler 220, agent 250, node management agent 330, etc. The data store 240 holds configuration information of the lower cluster, information on application systems running in the container cluster, node information (node ​​information), etc. The agent 250 grasps the status of the host OS of the lower cluster master node 200 and requests the container runtime 260 to start or stop the container. The container runtime 260 is an execution environment for running containers, and is configured, for example, by Docker (registered trademark) Daemon.

[0023] The lower cluster worker node 300 is a node that runs a container. The lower cluster worker node 300 may be configured as, for example, a physical server or a VM. The lower cluster worker node 300 includes a reverse proxy 310, a network control unit 320, a node management agent 330, a container runtime 340, and a host OS 350.

[0024] The reverse proxy 310 has a function of forwarding communication to a container to the appropriate container. The reverse proxy 310 is configured, for example, by Kube-proxy. The network control unit 320 controls IP tables and the like to set communication paths to the containers. The network control unit 320 is configured, for example, by Calico. The node management agent 330 grasps the status of the host OS 350 of the lower cluster worker node 300 and requests the container runtime 340 to start or stop the container. The container runtime 340 is an execution environment for running containers and is configured, for example, by Docker Daemon. The container runtime 340 runs various applications as containers in response to requests. For example, when running an upper cluster as a container, the container runtime 340 runs the containers that make up the upper cluster. For example, the container runtime 340 runs the upper cluster master node 400, upper cluster worker node 500, and the like as containers. The upper cluster master node 400 and the upper cluster worker node 500 make up the upper cluster. The upper cluster can be configured using, for example, Apache Ray, Kafka, Hadoop (registered trademark), Spark, etc. The host OS 350 is an operating system that controls the lower cluster worker nodes 300, and is, for example, Linux (registered trademark).

[0025] The upper cluster master node 400 is a node that manages the upper cluster and causes the upper cluster to execute requested source code as an application process. The upper cluster master node 400 runs as a container on the container runtime 340 of the lower cluster worker node 300. The upper cluster master node 400 includes a driver 410, a runtime 420, a scheduler 430, and a global control store 440.

[0026] The driver 410 determines the upper cluster master node 400 or upper cluster worker node 500 (these nodes are also called upper cluster nodes or processing cluster nodes) that will execute the application process 600 (also simply called an app or process), and assigns the application process 600 to the upper cluster node. The runtime 420 is the execution environment for the application process 600. The scheduler 430 executes the assigned application process 600 taking into account the CPU and memory usage rate on which the scheduler 430 operates. The global control store 440 stores configuration information for the upper cluster master node 400 and information (identifier) ​​of the upper cluster node on which the application process 600 operates.

[0027] The upper cluster worker node 500 is a node that executes source code requested by the upper cluster as an application process 600. The upper cluster worker node 500 runs as a container on the container runtime 340 of the lower cluster worker node 300. The upper cluster worker node 500 includes a runtime 520, a scheduler 530, and an object store 540. The runtime 520 is similar to the runtime 420, and the scheduler 530 is similar to the scheduler 430. The object store 540 is a storage area in which the application process 600 can store data. The application process 600 is a data processing process that can run on the upper cluster node. The application process 600 acquires data from a message broker such as Kafka, MQTT (Message Queuing Telemetry Transport), or AMQP (Advanced Message Queuing Protocol), performs processing such as analysis, and stores the results in the object store 540, a database (not shown), or the like. The application process 600 is started when the application developer sends an execution request including source code to the upper cluster master node 400, which selects the upper cluster node on which to execute the application, and the scheduler 430 (530) on the selected upper cluster node is executed on the runtime 420 (520).

[0028] The multi-hierarchical cluster policy control device 100 may be configured, for example, by a computer such as a PC or a server having a processor, memory, etc., or may be configured by a VM. The multi-hierarchical cluster policy control device 100 manages upper-level clusters by linking them with lower-level clusters, and when deploying an upper-level cluster, takes into account the requirements of other upper-level clusters (related clusters) with which it has a communication connection relationship, manages nodes on which multiple upper-level clusters can operate as a common node group, and deploys the upper-level cluster to a nearby node group from the perspective of communication by setting this as a policy when deploying the upper-level cluster, introduces the execution location when a process is executed in the upper-level cluster as a policy, and detects movement of the upper-level cluster node (container) and updates the process policy, thereby reducing the amount of communication between nodes located remotely.

[0029] The multi-tiered cluster policy control device 100 includes a common node identification unit 110, a manifest management unit 120, a container location identification unit 130, a process location identification unit 140, an ID map management unit 150, a policy setting unit 160, and an information storage unit 170. Here, the common node identification unit 110, the manifest management unit 120, the container location identification unit 130, the process location identification unit 140, the ID map management unit 150, and the policy setting unit 160 are functional units that are configured by a processor executing a program (a multi-tiered cluster control program) stored in memory.

[0030] When multiple upper clusters are deployed to the same lower cluster, the common node identification unit 110 identifies lower cluster worker nodes 300 that can be deployed by each upper cluster based on the requirements of each upper cluster, identifies upper clusters that have connection relationships between application processes 600 running on different upper clusters, and identifies lower worker nodes that satisfy the requirements of these upper clusters.

[0031] The manifest management unit 120 manages a deployment manifest (cluster deployment manifest) for deploying a higher-level cluster to a lower-level cluster. The cluster deployment manifest includes information such as requirements for deploying a higher-level cluster (for example, whether a required CPU, memory, or GPU (Graphics Processing Unit) is required (recommended or required)), network requirements such as exposed UDP and TCP ports, and container images for running the higher-level cluster.

[0032] The container location identification unit 130 identifies the lower cluster worker node 300 on which the container configured as the upper cluster master node 400 and the upper cluster worker node 500 runs. The process location identification unit 140 identifies the upper cluster node on which the application process 600 runs.

[0033] The ID map management unit 150 associates and manages the identifiers of the upper cluster nodes managed by the upper cluster master node 400 with the pod names of the upper cluster nodes managed by the lower cluster master node 200. Here, the pod name is unique information for identifying one or more containers that are treated as a group.

[0034] The policy setting unit 160 sets a policy (called a cluster policy) that specifies the lower cluster worker node to which the upper cluster will be deployed when the upper cluster is deployed, and also sets a policy (called a process policy) that specifies the upper cluster node that will execute the application process 600 within the upper cluster.

[0035] The information storage unit 170 is an example of a storage device, and is a storage area that stores programs and data used by the multi-tiered cluster policy control device 100. The information storage unit 170 stores, for example, a multi-tiered cluster control program that configures each functional unit, as well as node mapping information 1400 (see FIG. 7), process data location information 1500 (see FIG. 8), related cluster information 1600 (see FIG. 9), common node candidate information 1700 (see FIG. 10), process policy information 1800 (see FIG. 11), node information 2100 (see FIG. 2), cluster deployment request information 2200 (see FIGS. 3 and 4), and a cluster deployment manifest 2300 (see FIGS. 5 and 6). The information stored in the information storage unit 170 will be described later.

[0036] The client 700 is configured, for example, by a computer such as a PC, and receives various instructions from users who are application developers, infrastructure administrators of clusters, etc. The client 700 creates cluster deployment request information (see FIGS. 3 and 4) based on instructions from the user, for example, and sends it to the lower-level cluster master node 200.

[0037] Next, the node information 2100 will be described.

[0038] Fig. 2 is a diagram illustrating an example of the configuration of node information of a lower cluster according to one embodiment, Fig. 2 illustrates an example of node information for one node of a lower cluster.

[0039] The node information 2100 is managed by the lower cluster master node 200, and in this embodiment, the multi-hierarchical cluster policy control device 100 acquires it from the lower cluster master node 200. The node information 2100 is provided corresponding to each of the lower cluster master node 200 and the lower cluster worker node 300 (lower cluster node: container cluster node).

[0040] The node information 2100 includes labels indicating the management purpose of the corresponding lower cluster node (corresponding lower cluster node), annotations indicating supplementary information for the corresponding lower cluster node, conditions indicating the state of the target lower cluster node, addresses indicating network information such as IP address and host name, capacity indicating capacity information such as CPU and memory held by the target lower cluster node, allocatable indicating information such as surplus CPU and memory, system info indicating various information about the system that constitutes the corresponding lower cluster node, etc. The configuration of the node information 2100 is publicly known, so further explanation will be omitted.

[0041] Next, we will explain the cluster deployment request information 2200. The cluster deployment request information 2200 is information that is sent to the lower cluster master node 200 in order to deploy an upper cluster to a lower cluster, and is created by a user who wants to deploy an upper cluster, for example, by the client 700.

[0042] Fig. 3 is a diagram showing an example of cluster deployment request information according to an embodiment. Fig. 4 is a diagram showing another example of cluster deployment request information according to an embodiment. Cluster deployment request information 2200-1 in Fig. 3 is cluster deployment request information when deploying an Apache Ray cluster, and cluster deployment request information 2200-2 in Fig. 4 is cluster deployment request information when deploying a Kafka cluster.

[0043] The cluster deployment request information 2200 includes an API version (apiVersion) indicating the version of the API, a type (kind) indicating the type of request, metadata, and spec. The API version and type are publicly known, so a detailed description will be omitted.

[0044] The spec contains the requirements for deploying a higher-level cluster to a lower-level cluster.

[0045] The metadata includes management information for a lower cluster to manage a higher cluster. The metadata includes annotations. The annotations include a key "pool-hetero-stuck" and object-type information indicating the requirements for the nodes of the higher cluster (which can also be considered requirements for the lower cluster nodes to which the higher cluster nodes are deployed). The object-type information includes related services (related-services), node preferences (node_preference), cluster size (cluster_size), and resources (resources).

[0046] The related services are information indicating one or more related services (having a relationship (connection relationship) that connects communications), and include information indicating other upper clusters to which an application process deployed to an upper cluster is connected. The related services of the cluster deployment request information 2200-1 shown in FIG. 3 indicate that an application deployed to the upper cluster Ray Cluster corresponding to the cluster deployment request information 2200-1 has a connection relationship with the Kafka cluster, the Knative cluster, and the Influx cluster. The related services of the cluster deployment request information 2200-2 shown in FIG. 4 indicate that an application deployed to the upper cluster Kafka Cluster corresponding to the cluster deployment request information 2200-2 has a connection relationship with the Ray cluster, the Knative cluster, and the Influx cluster. Since the Kafka cluster is a cluster that performs PubSub (Publish-Subscribe), it is not directly connected to the Influx cluster. However, in this embodiment, not only clusters that have a direct connection relationship with the target upper cluster but also clusters that have an indirect connection relationship with the target upper cluster are included in the clusters that have a communication relationship. By including clusters having indirect connections in this way, it becomes possible to collectively deploy clusters having indirect connections to lower cluster nodes that are close in terms of communication distance.

[0047] Node preferences are information indicating preferences common to all clusters included in the related service. In these node preferences, it is possible to specify whether it is recommended (whether gpu-support is true) or required (whether required is true) that a GPU be installed on the lower cluster worker node 300 on which each cluster is deployed. This makes it possible to specify, for example, whether a GPU is required, and therefore, when a GPU is required, it is possible to appropriately prevent the target cluster from being deployed on a lower cluster worker node that does not have a GPU.

[0048] The cluster size is information indicating the minimum and maximum sizes of the upper cluster. The resources include a request indicating the minimum amount of resources (CPU, memory, etc.) required and a limit indicating the maximum amount of resources allowed (restricted amount).

[0049] Next, the cluster deploy manifest 2300 will be described. The cluster deploy manifest 2300 is a manifest used to deploy a higher-level cluster to a lower-level cluster. The lower-level cluster master node 200 deploys the higher-level cluster in accordance with this cluster deploy manifest 2300. The cluster deploy manifest 2300 is generated based on the cluster deploy request information 2200 in a cluster policy information generation process (see FIG. 13) described later.

[0050] Fig. 5 is a diagram illustrating an example of a cluster deployment manifest according to an embodiment. Fig. 6 is a diagram illustrating another example of a cluster deployment manifest according to an embodiment. Cluster deployment manifest 2300-1 in Fig. 5 is a cluster deployment manifest when deploying an Apache Ray cluster, and cluster deployment manifest 2300-2 in Fig. 6 is a cluster deployment manifest when deploying a Kafka cluster.

[0051] The cluster deploy manifest 2300 has a configuration in which affinity (node ​​identification information) is added under specs to the cluster deploy request information 2200 shown in Figures 3 and 4. The cluster deploy manifest 2300-1 has affinities 2301-1 and 2301-2 added to the cluster deploy request information 2200-1, and the cluster deploy manifest 2300-2 has affinity 2302-1 added to the cluster deploy request information 2200-2.

[0052] The affinity can include a required condition (requiredDuringSchedulingIgnoredDuringExecution) and a preferred condition (preferredDuringSchedulingIgnoredDuringExecution) for the nodes of the lower cluster to which the upper cluster is deployed. The conditions are configured with a key-value pair, and a key-value pair set as a required condition means that it must match the key-value pair of the label in the node information 2100 of the node, and a key-value pair set as a preferred condition means that a node that matches the key-value pair of the label in the node information 2100 of the node is preferentially selected.

[0053] For example, in the case of affinity 2301-1, the upper cluster can only be deployed to the lower cluster worker node 300 whose label key in the node information 2100 contains 50acd0ce-9b9a-4438-ab6a-a64f4656165c as a value for pool-hetero-stuck-id. If there are multiple such nodes, the node whose label key-value pair is gpu-support and true will be the preferred deployment destination.

[0054] Next, the node mapping information 1400 will be described.

[0055] FIG. 7 is a diagram illustrating a configuration of node mapping information according to an embodiment.

[0056] The node mapping information 1400 is information on the correspondence between upper cluster nodes and lower cluster pods, and is generated in the process policy information generation process (see FIG. 14). The node mapping information 1400 makes it possible to identify the lower cluster pods corresponding to the upper cluster nodes. The node mapping information 1400 stores an entry for each upper cluster node. The entry of the node mapping information 1400 includes fields for a cluster ID (cluster-id) 1401, a cluster node ID (cluster node id) 1402, a machine ID (machine id) 1403, and a pod name (pod name) 1404.

[0057] The identifier (cluster ID) of the upper cluster to which the upper cluster node corresponding to the entry belongs is stored in the cluster ID 1401. The cluster ID is, for example, a UUID (Universally Unique Identifier) ​​created by the multi-hierarchical cluster policy control device 100.

[0058] The identifier (cluster node ID) of the higher-level cluster node corresponding to the entry is stored in the cluster node ID 1402. The cluster node ID is the identifier of the higher-level cluster node that is created and managed by the higher-level cluster master node 400.

[0059] The identifier (machine ID) of the lower cluster worker node is stored in the machine ID 1403. The machine ID is the identifier of the lower cluster worker node created and managed by the lower cluster master node 200, and corresponds to the machine ID under the system information of the node information 2100.

[0060] The pod name 1404 stores the name of the pod (pod name) that includes the upper cluster node corresponding to the entry.

[0061] Next, the process data location information 1500 will be described.

[0062] FIG. 8 is a diagram illustrating the configuration of process data location information according to an embodiment.

[0063] The process data location information 1500 is information on the correspondence between processes in a higher-level cluster and higher-level cluster nodes, and is generated in step 5230 of FIG. 15. The process data location information 1500 makes it possible to know on which node in the higher-level cluster a process in a higher-level cluster is being executed. The process data location information 1500 stores an entry for each process. An entry of the process data location information 1500 includes fields for a cluster ID (cluster-id) 1501, a process ID (process id) 1502, and a cluster node ID (cluster node id) 1503.

[0064] Cluster ID 1501 stores the identifier (cluster ID) of the upper cluster in which the process corresponding to the entry is executed. Process ID 1502 stores the identifier (process ID) of the process corresponding to the entry. The process ID is the identifier of the application process 600 deployed to the upper cluster created and managed by the upper cluster master node 400. Cluster node ID 1503 stores the identifier (cluster node ID) of the node in the upper cluster in which the process 600 corresponding to the entry is executed. The cluster node ID is the identifier of the upper cluster node created and managed by the upper cluster master node 400.

[0065] Next, the related cluster information 1600 will be described.

[0066] FIG. 9 is a configuration diagram of related cluster information according to an embodiment.

[0067] The related cluster information 1600 is information for managing related (connected) upper clusters (related clusters), and is generated in step 7020 of FIG. 13. The related cluster information 1600 makes it possible to identify other upper clusters to which an application process running in an upper cluster connects. The related cluster information 1600 stores an entry for each cluster. An entry of the related cluster information 1600 includes fields for a pool ID (pool-id) 1601, a cluster ID (cluster-id) 1602, and a related cluster ID (related cluster id) 1603.

[0068] The pool ID stores an identifier (pool ID) indicating a group of higher-level clusters (pool) including higher-level clusters related to the higher-level cluster corresponding to the entry. The pool ID is a UUID created by the multi-hierarchical cluster policy control device 100. The cluster ID 1602 stores the cluster ID of the higher-level cluster corresponding to the entry. The related cluster ID 1603 stores the cluster ID of another higher-level cluster related to the higher-level cluster corresponding to the entry, that is, the cluster ID of another higher-level cluster to which an application process running in the higher-level cluster corresponding to the entry connects.

[0069] Next, the common node candidate information 1700 will be described.

[0070] FIG. 10 is a configuration diagram of common node candidate information according to an embodiment.

[0071] The common node candidate information 1700 is information for managing nodes of lower clusters that meet the requirements of a group of higher clusters (pools), and is generated in step 7040 of FIG. 13. The common node candidate information 1700 makes it possible to identify nodes that meet the requirements of multiple higher clusters with which there is a connection relationship. The common node candidate information 1700 stores an entry for each pool. An entry of the common node candidate information 1700 includes fields for a pool ID (pool-id) 1701 and a machine ID (machine id) 1702.

[0072] A pool ID indicating a pool that is a related upper cluster group is stored in the pool ID 1701. A machine ID of a lower cluster worker node 300 that can be allocated to the pool corresponding to the entry is stored in the machine ID 1702.

[0073] Next, the process policy information 1800 will be described.

[0074] FIG. 11 is a configuration diagram of process policy information according to an embodiment.

[0075] The process policy information 1800 is information for managing the correspondence between cluster names and placement groups, and is created in step 8070 of FIG. 14. The process policy information 1800 allows an application developer to know which placement group to specify for a cluster name when deploying source code to a cluster. The process policy information 1800 stores an entry for each upper-level cluster. The entry of the process policy information 1800 includes fields for a cluster ID 1801, a cluster name 1802, a cluster node ID 1803, and a placement group 1804.

[0076] The cluster ID 1801 stores the cluster ID of the higher-level cluster corresponding to the entry. The cluster name 1802 stores the human-readable name (cluster name) of the higher-level cluster corresponding to the entry. The cluster name may be a name specified by the user for the related service in the annotation of the cluster deployment request information 2200. The cluster node ID 1803 stores the cluster node ID of the higher-level cluster corresponding to the entry. The placement group 1804 stores identification information (placement group name) indicating the placement group (a group of node candidates for deployment destination) specified by the application developer when deploying the source code of the application. The placement group name may be a UUID or a human-readable character string created by the multi-hierarchical cluster policy control device 100.

[0077] Next, the processing operations in the computer system 1 will be described.

[0078] FIG. 12 is a sequence diagram of a cluster deployment process according to an embodiment.

[0079] The cluster deployment process is started, for example, when the user creates cluster deployment request information 2200 for the higher-level cluster to be deployed in the client 700 and issues a cluster deployment instruction. Note that the annotations in the cluster deployment request information 2200 include various requirements corresponding to the cluster to be deployed. In step 5010, the client 700 sends the created cluster deployment request information 2200 to the lower cluster master node 200 of the lower cluster to which the higher-level cluster is to be deployed.

[0080] In step 5020 , upon receiving the cluster deployment request information 2200 , the lower-level cluster master node 200 transmits an acceptance completion notification to the client 700 .

[0081] In step 5030 , the lower-level cluster master node 200 transmits the cluster deployment request information 2200 to the multi-hierarchical cluster policy control apparatus 100 .

[0082] In step 5040, upon receiving the cluster deployment request information 2200, the multi-tier cluster policy control apparatus 100 executes a cluster policy information generation process (see FIG. 13) for generating a cluster deployment manifest 2300 based on the cluster deployment request information 2200. Here, the cluster policy information generation process will be described in detail.

[0083] FIG. 13 is a flowchart of a cluster policy information generation process according to an embodiment.

[0084] In step 7000, the common node specifying unit 110 saves the cluster deployment request information 2200 received in step 5030. Specifically, the common node specifying unit 110 creates a UUID (cluster ID) of the cluster corresponding to this information, associates the cluster ID with the cluster deployment request information 2200, and stores it as a cluster request information table.

[0085] In step 7010, the common node identification unit 110 acquires, from the lower cluster master node 200, the node information 2100 of all or a plurality of nodes that make up the lower cluster.

[0086] In step 7020, the common node identifying unit 110 identifies information about related clusters (related clusters) (having a communication connection relationship) and saves it as related cluster information 1600. Specifically, the common node identifying unit 110 refers to the cluster request information table using the cluster ID issued in step 7000 as a key to identify all names (called cluster names) under the metadata and names of clusters (related clusters) listed in related services (related-services) under the annotations under the metadata. Next, the common node identifying unit 110 identifies all IDs (related cluster IDs) of related clusters whose cluster names in the cluster request information table are the identified related cluster names. Here, if the cluster deploy request information 2200 for the related cluster name does not exist in the cluster request information table, this indicates that the storage process of the cluster deploy request information 2200 corresponding to the related cluster has not yet been performed, and the unit does not proceed to the following steps and waits until the cluster deploy request information 2200 of the related cluster is registered in the cluster request information table.

[0087] The common node identification unit 110 checks whether the cluster ID or related cluster ID exists in the cluster IDs or related cluster IDs of the related cluster information 1600, and if it exists, it grasps the pool ID, and if it does not exist, it creates a UUID as a new pool ID corresponding to these cluster groups (pools). Next, the common node identification unit 110 checks whether the combination of the pool ID, cluster ID, and related cluster ID exists in the related cluster information 1600, and if it does not exist, it saves a new entry of this information in the related cluster information 1600.

[0088] In step 7030, the common node identification unit 110 calculates (identifies) common lower node candidates (common node candidates) to which the cluster and related clusters will be deployed, and saves information about the common node candidates in the common node candidate information 1700. The common node identification unit 110 references the cluster request information table using the related cluster ID identified in step 7020 and the cluster ID identified in step 7000 as keys, and identifies the node preferences under the annotation under the metadata of the cluster deployment request information 2200 corresponding to these. Next, the common node identification unit 110 references the node information 2100 of all nodes, identifies the node information 2100 whose label contains the key / value included in the identified node preference, and identifies the machine ID under the system info in that node information 2100. Next, if a pair of the pool ID and machine ID identified in step 7020 does not exist, the common node identification unit 110 adds them to the common node candidate information 1700 as a new entry.

[0089] In step 7040, the manifest management unit 120 generates a cluster deployment manifest 2300. Specifically, the manifest management unit 120 references the cluster deployment request information 2200 in the cluster request information table using the cluster ID determined in step 7000 as a key. Next, the manifest management unit 120 creates a cluster deployment manifest 2300 by adding affinity to the referenced cluster deployment request information 2200. The affinity includes node affinity. The manifest management unit 120 adds a required condition (requiredDuringSchedulingIgnoredDuringExecution) and a preferred condition (preferredDuringSchedulingIgnoredDuringExecution) to the node affinity, and enters "pool-hetero-stuck-id" as the key and the pool ID determined in step 7020 as the value in the match expressions (matchExpressions) under the node selection terms (nodeSelectorTerms) under the required conditions. Note that if there are other required conditions in the node preferences of the cluster deployment request information 2200, the manifest management unit 120 enters the corresponding key and value under the required conditions. Furthermore, if there are preferred conditions in the node preferences of the cluster deployment request information 2200, the manifest management unit 120 enters the corresponding key and value under the preferred conditions. Furthermore, the manifest management unit 120 adds pool-hetero-stuck-ids to the label of the node information 2100 if it does not exist as a key, and also adds the pool ID identified in step 7020 to the value for the key if it does not exist. According to this step, for example, the cluster deploy manifest 2300-1 shown in Fig. 5 is created based on the cluster deploy request information 2200-1 shown in Fig. 3, and the cluster deploy manifest 2300-2 shown in Fig. 6 is created for the cluster deploy request information 2200-2 shown in Fig. 4.

[0090] Returning to the explanation of FIG. 12, in step 5050 , the multi-hierarchical cluster policy control device 100 transmits the generated cluster deploy manifest 2300 to the lower-level cluster master node 200 .

[0091] In step 5060, the lower cluster master node 200 sends a request for deployment of the upper cluster to the lower cluster worker node 300 based on the received cluster deployment manifest 2300.

[0092] In step 5070, the lower cluster master node 200 transmits information about the deployed upper cluster to the multi-hierarchical cluster policy control device 100. Here, the information about the deployed upper cluster includes the Pod name, the container name, the machine ID of the lower cluster worker node 300 on which the container runs, and the like. Note that the lower cluster master node 200 does not need to actively communicate the information about the deployed upper cluster to the multi-hierarchical cluster policy control device 100. For example, the multi-hierarchical cluster policy control device 100 may constantly monitor the API server of the lower cluster master node 200 (for example, by watching by polling or a gRPC streaming connection), and the multi-hierarchical cluster policy control device 100 may detect the information.

[0093] In step 5080, the multi-hierarchical cluster policy control apparatus 100 executes a process policy information generation process (FIG. 14) for generating process policy information. Details of the process policy information generation process will now be described.

[0094] FIG. 14 is a flowchart of a process policy information generation process according to an embodiment.

[0095] In step 8010, the container location identification unit 130 calls the API of the lower-level cluster master node 200 to obtain the machine ID of the node on which the container is running.

[0096] In step 8020, the container position identifying unit 130 calls the API of the upper cluster master node 400 to obtain the node ID of the upper cluster corresponding to the container.

[0097] In step 8030, the container location identification unit 130 identifies the pod name of the lower cluster corresponding to the node ID of the upper cluster. Here, the node ID of the upper cluster and the pod name of the lower cluster are independent and are not managed in association with each other. There are several methods for identifying the pod name corresponding to the node ID, for example, a method of identifying the pod name using the IP address of a container included in a pod corresponding to the pod name. In this method, the container location identification unit 130 first obtains the IP address of each container from the API of the lower cluster master node 200 and obtains the IP address of the container corresponding to the pod name. Next, the container location identification unit 130 executes source code that obtains the IP address and node ID in the node of the upper cluster, and obtains the node ID and IP address. Next, the container location identification unit 130 identifies the pod name corresponding to the IP address that is the same as the IP address of the node ID of the node of the upper cluster.

[0098] In step 8040, the container location identification unit 130 saves the correspondence between the node ID of the higher-level cluster that has been identified and the pod name in the node mapping information 1400. Specifically, the container location identification unit 130 adds entries to the node mapping information 1400 in which the cluster ID received in step 5070, the node ID that has been identified in step 8020, the machine ID that has been identified in step 8010, and the pod name that has been identified in step 8030 are stored in the fields of the cluster ID, cluster node ID, machine ID, and pod name, respectively.

[0099] In step 8050, the common node identifying unit 110 refers to the related cluster information and identifies the related upper cluster. Specifically, the common node identifying unit 110 refers to the related cluster information 1600 and identifies the related cluster ID corresponding to the cluster ID of the cluster information received in step 5070, and refers to the cluster request information table and identifies the cluster information of the upper cluster of the related cluster ID using the related cluster ID as a key.

[0100] In step 8060, the ID map management unit 150 identifies the containers of the higher-level cluster that operate in the vicinity (predetermined range) of the related higher-level cluster (related cluster) identified in step 8050. Specifically, the ID map management unit 150 calls the API of the lower-level cluster for the related cluster ID identified in step 8050, and acquires node information 2100 of the cluster and the lower-level cluster on which the nodes (containers) of the related cluster operate. Next, the ID map management unit 150 identifies the availability zone (referred to as the availability zone of the cluster) that is set as the availability zone under the label in the node information 2100 of the lower-level cluster on which all nodes of the higher-level cluster operate. Next, the ID map management unit 150 identifies the availability zone (referred to as the availability zone of the related cluster) under the label in the node information of the lower-level cluster on which all containers of the related cluster operate. Next, the ID map management unit 150 identifies nodes (called neighboring nodes) whose availability zones of related clusters are the same as the availability zone of the cluster. Next, the ID map management unit 150 references the node mapping information 1400 and identifies the cluster node IDs using the pod names of the higher-level clusters that operate on the identified neighboring nodes as keys. The ID map management unit 150 performs these processes for all related clusters and holds sets of the cluster IDs of the higher-level clusters, the related cluster names, and the cluster node IDs.

[0101] In step 8070, the ID map management unit 150 issues a UUID or a human-readable and unique character string as the placement group name, and saves it in the process policy information 1800. Specifically, the ID map management unit 150 adds the cluster ID, associated cluster name, cluster node ID, and placement group name retained in step 8060 to the process policy information 1800 as entries.

[0102] Returning to the explanation of FIG. 12, in step 5090, the multi-hierarchical cluster policy control device 100 sets the process policy information 1800 in the upper cluster master node 400 of the upper cluster.

[0103] In step 5100, when a user instructs the client 700 to refer to a policy, the client 700 transmits a policy reference request to the multi-hierarchical cluster policy control device 100. The policy reference request includes, for example, the cluster name (processing cluster identification information) of another higher-level cluster used in the higher-level cluster.

[0104] In step 5110, the multi-hierarchical cluster policy control device 100 refers to the process policy information 1800 using the received other higher-level cluster name, and returns the placement group name linked to the other cluster name to the requesting client 700. This allows the user to know the placement group name corresponding to the higher-level cluster.

[0105] As described above, the cluster deployment process makes it possible to create a cluster deployment manifest 2300 that can appropriately place multiple related upper clusters on the same lower node, and to appropriately place multiple related upper clusters on the same lower node.

[0106] Next, the processing operation of the application deployment process will be described.

[0107] FIG. 15 is a sequence diagram of an application deployment process according to an embodiment.

[0108] The application deployment process begins, for example, when a user creates application deployment request information for the application to be deployed and issues an instruction to deploy the application on the client 700. The application deployment request information includes the source code to be executed, requirements such as the required CPU and memory, and the placement group name. In step 5210, the client 700 sends the application deployment request information to the upper-level cluster master node 400.

[0109] In step 5220, the upper cluster master node 400 sends the source code to the upper cluster worker nodes 500 included in the placement group of the received placement group name that meet the requirements, and requests that they be executed as an application process.

[0110] In step 5230, the multi-hierarchical cluster policy control device 100 receives, as process information, the cluster ID of the upper cluster, the process ID of the application, and the node ID of the upper cluster node on which the application process is running, and stores this information in the process data location information 1500. Note that the upper cluster master node 400 does not have to actively communicate this information to the multi-hierarchical cluster policy control device 100; the multi-hierarchical cluster policy control device 100 may constantly monitor the upper cluster master node 400 (for example, by polling or by watching via a gRPC streaming connection) and the multi-hierarchical cluster policy control device 100 may detect this information.

[0111] This application deployment process allows processes in higher-level clusters to run in the vicinity of related clusters, localizing communication and reducing the amount of communication between distantly located nodes and the communication costs between nodes over long distances.

[0112] Next, the processing operation of the container movement processing will be described.

[0113] FIG. 16 is a sequence diagram of a container movement process according to an embodiment.

[0114] Here, container migration does not actually mean that the container is moving, but rather that the old container is deleted and a new container is created on a different lower cluster worker node 300. When a container is stateful, a volume for the container to store data is generally mounted, and the new container mounts the same volume as the old container. Therefore, from the perspective of the container layer (upper cluster layer), this is equivalent to restarting the container, i.e., the node of the upper cluster. Meanwhile, the new container is created on a node among the lower cluster worker nodes with a low utilization rate of computing resources.

[0115] The container migration process is performed, for example, when the lower cluster master node 200 receives a request from a user to delete an old container, when a failure occurs in the lower cluster worker node 300 on which the old container is running, or when there is a shortage of computing resources in the lower cluster on which the old container is running.

[0116] In step 5410, the lower-level cluster master node 200 deletes the old container from the lower-level cluster worker node 300 and creates a new container on another lower-level cluster worker node 300.

[0117] In step 5420, the multi-hierarchical cluster policy control device 100 acquires information (container movement information) that the operating location of the container has changed from the lower-level cluster master node 2. Note that the lower-level cluster master node 200 does not have to actively communicate the container movement information to the multi-hierarchical cluster policy control device 100; for example, the multi-hierarchical cluster policy control device 100 may constantly monitor the API server of the lower-level cluster master node 200 (for example, by watching by polling or a gRPC streaming connection), and the multi-hierarchical cluster policy control device 100 may detect the container movement information.

[0118] In step 5430, the multi-hierarchical cluster policy control apparatus 100 executes the process policy information generation process (FIG. 14) to generate the process policy information 1800.

[0119] In step 5440 , the multi-hierarchical cluster policy control device 100 transmits the process policy information including the placement group name to the upper-level cluster master node 400 .

[0120] In step 5450, the upper level cluster master node 400 transmits the source code to a node that satisfies the requirements among the upper level cluster worker nodes 500 included in the received process policy information, and requests that it be executed as an application process.

[0121] This container migration process allows processes in a higher-level cluster to run in the vicinity of the associated cluster even when a container is deleted or a new one is created. By localizing communication, the amount of communication between nodes located far away can be reduced, and the communication costs between nodes located far away can be reduced.

[0122] The present invention is not limited to the above-described embodiment, and can be modified appropriately without departing from the spirit of the present invention.

[0123] For example, in the above embodiments, some or all of the processing performed by the processor may be performed by a hardware circuit. Furthermore, the programs in the above embodiments may be installed from a program source. The program source may be a program distribution server or a recording medium (e.g., a portable recording medium). [Explanation of symbols]

[0124] 1...computer system, 100...multi-hierarchical cluster policy control device, 110...common node identification unit, 120...manifest management unit, 130...container location identification unit, 140...process location identification unit, 150...identifier map management unit, 160...policy setting unit, 170...information storage unit, 200...lower cluster master node, 300, 300-1, 300-2...lower cluster worker nodes, 400, 400-1-1, 400-2-1...upper cluster master node, 500, 500-1-1, 500-1-2, 500-2-1, 500-2-2...lower cluster worker nodes.

Claims

1. A multi-tier cluster control device that controls a plurality of processing clusters that operate on a container cluster, The multi-hierarchical cluster control device has a processor, The processor: When deploying the processing cluster, for each of the plurality of processing clusters, deployment request information is received, the deployment request information including requirements for processing cluster nodes constituting the processing cluster and designation of related clusters, which are other processing clusters having a connection relationship with the processing cluster; Identifying one or more container cluster nodes that are nodes of a container cluster that meet the requirements based on the requirements for the processing cluster nodes and the requirements for the processing cluster nodes that make up the associated cluster; transmitting node identification information indicating the identified container cluster node to the container cluster as cluster policy information for determining a container cluster node to deploy the processing cluster in the container cluster to deploy the processing cluster; identifying one or more placement groups, which are groups of processing cluster nodes operating on container cluster nodes within a predetermined range of the container cluster nodes on which the processing cluster nodes of the processing cluster and the processing cluster nodes of the associated cluster operate; The information on the identified placement group is transmitted to the processing cluster as process policy information for determining the processing cluster node on which the process is to be deployed. Multi-level cluster control device.

2. The processor: The node-specific information is incorporated into a manifest that the container cluster follows when deploying the processing cluster, and the manifest is transmitted to the container cluster. The multi-hierarchical cluster control device according to claim 1 .

3. The requirements for the processing cluster nodes include essential conditions that are essential conditions and priority conditions that are not essential but are preferred conditions; The processor identifies one or more container cluster nodes that meet the requirements for the processing cluster node and the processing cluster nodes that constitute the related cluster, giving priority to container cluster nodes that meet the priority conditions. The multi-hierarchical cluster control device according to claim 1 .

4. The processor: when detecting a change in the container cluster node on which the processing cluster node is executed, identifying one or more placement groups, which are groups of processing cluster nodes operating on container cluster nodes within a predetermined range of the container cluster node on which the processing cluster node of the processing cluster node and the processing cluster node of an associated cluster operate; When deploying a process to a processing cluster, information on the identified placement group is transmitted to the processing cluster as information for determining a processing cluster node to which the process is to be deployed. The multi-hierarchical cluster control device according to claim 1 .

5. The container cluster nodes within the predetermined range are container cluster nodes that belong to the same availability zone. The multi-hierarchical cluster control device according to claim 1 .

6. The processor: storing information about the placement group in a storage device; receiving processing cluster identification information indicating a predetermined processing cluster from a predetermined request source, and transmitting to the request source a placement group name to which a processing cluster node of the processing cluster indicated by the processing cluster identification information belongs; The multi-hierarchical cluster control device according to claim 1 .

7. A multi-hierarchical cluster control method by a multi-hierarchical cluster control device that controls a plurality of processing clusters operated on a container cluster, comprising: The multi-hierarchical cluster control device includes: When deploying the processing cluster, for each of the plurality of processing clusters, deployment request information is received, the deployment request information including requirements for processing cluster nodes constituting the processing cluster and designation of related clusters, which are other processing clusters having a connection relationship with the processing cluster; Identifying one or more container cluster nodes that are nodes of a container cluster that meet the requirements based on the requirements for the processing cluster nodes and the requirements for the processing cluster nodes that make up the associated cluster; transmitting node identification information indicating the identified container cluster node to the container cluster in which the processing cluster is to be deployed as information indicating the container cluster node in which the processing cluster is to be deployed; identifying one or more placement groups, which are groups of processing cluster nodes operating on container cluster nodes within a predetermined range of the container cluster nodes on which the processing cluster nodes of the processing cluster and the processing cluster nodes of the associated cluster operate; The information on the identified placement group is transmitted to the processing cluster as information for determining the processing cluster node on which the process is to be deployed. Multi-level cluster control method.

8. A multi-hierarchical cluster control program for causing a computer to execute processing related to control of a plurality of processing clusters operated on a container cluster, The computer When deploying the processing cluster, for each of the plurality of processing clusters, deployment request information is received, the deployment request information including requirements for processing cluster nodes constituting the processing cluster and designation of related clusters, which are other processing clusters having a connection relationship with the processing cluster; Identifying one or more container cluster nodes that are nodes of a container cluster that meet the requirements based on the requirements for the processing cluster nodes and the requirements for the processing cluster nodes that make up the associated cluster; transmitting node identification information indicating the identified container cluster node to the container cluster in which the processing cluster is to be deployed as information indicating the container cluster node in which the processing cluster is to be deployed; identifying one or more placement groups, which are groups of processing cluster nodes operating on container cluster nodes within a predetermined range of the container cluster nodes on which the processing cluster nodes of the processing cluster and the processing cluster nodes of the associated cluster operate; The information on the identified placement group is transmitted to the processing cluster as information for determining the processing cluster node on which the process is to be deployed. to function as Multi-level cluster control program.

Images