Virtualization system recovery device and virtualization system recovery method

Abstract

The present invention includes: a plurality of clusters (12A, 12B) in which containers related to applications (15a, 15b) virtually created on a physical machine by container virtualization software are clustered and arranged; and internal abnormality detection units (17A, 17B) and an external abnormality detection unit (23) that are virtually created outside the clusters 12A, 12B. The external abnormality detection unit (23) is configured to detect, when the internal abnormality detection units (17A, 17B) detect abnormality related to the applications (15a, 15b), a cluster (cluster 12A or 12B) where containers related to the applications (15a, 15b) with abnormality are arranged as being abnormal.

Description

[Technical field]

[0001] The present invention relates to a virtualization system recovery device and a virtualization system recovery method that realize anomaly detection and failure recovery for containers and applications running on containers in a computing platform based on virtual machines or containers. [Background technology]

[0002] The above-mentioned virtual machine is a computer that realizes the same functions as a physical computer through software. A container is a virtualization technology that is created by packaging an application in an environment called a "container" and runs on a container engine. In conventional container-based technologies, anomaly detection and failure recovery of containers and applications running on containers are realized mainly by the Liveness / Readiness Probe function (also called the probe function) of Kubernetes, which will be described later.

[0003] Kubernetes is container virtualization software that creates containers such as Docker and clusters them, and is also open source software. The liveness probe function controls whether or not a container should restart, and the readiness probe function controls whether or not a container should accept a request. A conventional technology of this type is described in Non-Patent Document 1. [Prior art documents] [Non-patent literature]

[0004] [Non-Patent Document 1] "Using Liveness Probes, Readiness Probes, and Startup Probes," kubernetes, [online], [Retrieved June 10, 2021], Internet〈https: / / kubernetes.io / ja / docs / tasks / configure-pod-container / configure-liveness-readiness-startup-probes / 〉 Summary of the Invention [Problem to be solved by the invention]

[0005] Incidentally, in virtualization systems in the field of virtualization technology, not limited to the containers mentioned above, manual recovery work is performed based on an alert issued when a fault in the virtualization system is detected. However, since manual recovery work is performed after an alert is issued, it is difficult to reduce the time from the occurrence of a fault to normalization.

[0006] When detecting and recovering from such a fault using the probe function of CubeSuites that detects anomalies and recovers from the fault, the period for fault monitoring can only be set to a slow period such as one second, which is determined in advance. Therefore, when the fastest possible fault detection and fault recovery are required, there is a problem that the fault detection and fault recovery cannot be performed faster than the default CubeSuites fault detection and recovery function.

[0007] The present invention has been made in consideration of the above circumstances, and its objective is to detect abnormalities and recover from failures that occur in a virtualization system more quickly than the abnormality detection and recovery functions of the container virtualization software. [Means for solving the problem]

[0008] In order to solve the above problems, a virtualization system recovery device of the present invention includes a computational resource cluster that is virtually created on a physical machine by container virtualization software and that clusters and arranges the virtually created containers, a cluster management unit that manages control related to the arrangement and operation of the virtually created and clustered containers, a plurality of clusters each configured to include the computational resource cluster and the cluster management unit, an internal anomaly detection unit that is arranged for each of the plurality of clusters and that is virtually created outside the virtually created computational resource cluster and the cluster management unit and that detects an anomaly in the container, and an external anomaly detection unit that is virtually created outside the plurality of clusters and that detects an anomaly in a cluster in which an abnormal container is arranged as an anomaly when an anomaly in a container is detected by the internal anomaly detection unit. a distribution destination switching unit that is virtually created outside the plurality of clusters and notifies a Domain Name System (DNS) of a communication distribution stop instruction related to an abnormal cluster detected by the external anomaly detection unit, and the DNS that manages a domain name indicating a name of an application related to the container and an IP (Internet Protocol) address for each cluster in association with each other is provided outside a server that constitutes the cluster, and the DNS erases the IP address of the abnormal cluster indicated in the communication distribution stop instruction. It is characterized by: Effect of the Invention

[0009] According to the present invention, it is possible to detect an abnormality and recover from the failure that occurs in the virtualization system more quickly than the abnormality detection and recovery function of the container virtualization software. [Brief description of the drawings]

[0010] [Figure 1] 1 is a block diagram showing a configuration of a virtualization system recovery device according to an embodiment of the present invention. [Diagram 2] FIG. 11 is a block diagram showing a configuration when an endpoint setting unit and a Pod are deployed in a 1:1 configuration by a failure response deployment instruction unit in the virtualization system recovery device of this embodiment. [Diagram 3] FIG. 11 is a block diagram for explaining a first abnormality detection process of a container by a Pod of the virtualization system recovery device of this embodiment. [Figure 4] 11 is a block diagram for explaining a second abnormality detection process by a routing table provided for each worker node of the virtualization system recovery apparatus of the present embodiment. FIG. [Diagram 5]FIG. 11 is a block diagram for explaining a third abnormality detection process by monitoring a daemon of a virtual switch provided for each worker node of the virtualization system recovery apparatus of this embodiment. [Figure 6] FIG. 11 is a block diagram for explaining a fourth abnormality detection process by monitoring a daemon of a container runtime provided for each worker node of the virtualization system recovery device of this embodiment. [Figure 7] FIG. 13 is a block diagram for explaining a fifth abnormality detection process by monitoring each worker node of the virtualization system recovery device of the present embodiment. [Figure 8] FIG. 13 is a block diagram for explaining a sixth abnormality detection process by monitoring a DB externally attached to a cluster of a container system of the virtualization system recovery device of this embodiment. [Figure 9] 11 is a block diagram illustrating a configuration for explaining a process of detecting an anomaly related to the occurrence of a failure in a plurality of clusters by an external anomaly detection unit. FIG. [Figure 10] 1 is a block diagram for explaining an abnormality handling process of the virtualization system recovery device of the present embodiment. FIG. [Figure 11] FIG. 2 is a diagram showing the correspondence between domain names and resolved IP addresses in a DNS record table. [Figure 12] FIG. 13 illustrates how IP addresses of a failed cluster are deleted from a DNS record table. [Figure 13] 11 is a flowchart for explaining an abnormality handling process performed by the virtualization system recovery device of the present embodiment. [Figure 14] FIG. 2 is a block diagram showing a configuration of a virtualization system recovery device according to a first modified example of the embodiment of the present invention. [Figure 15] FIG. 11 is a block diagram showing a configuration of a virtualization system recovery device according to a second modified example of the embodiment of the present invention. [Figure 16] FIG. 2 is a hardware configuration diagram showing an example of a computer that realizes the functions of the virtualization system recovery device according to the present embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. However, in all the drawings in this specification, components having corresponding functions are given the same reference numerals, and the description thereof will be omitted as appropriate. <Configuration of the embodiment> FIG. 1 is a block diagram showing a configuration of a virtualization system recovery device according to an embodiment of the present invention.

[0012] 1 is a virtualization system configured with a plurality of clusters (a first cluster 12A and a second cluster 12B in this example) in which containers are clustered. The first cluster 12A is configured with a cluster manager 14A and a computational resource cluster 15A. The second cluster 12B is configured with a cluster manager 14B and a computational resource cluster 15B.

[0013] The cluster management units 14A and 14B are configured to include a communication distribution unit 14a, a computational resource operation unit 14b, a computational resource management unit 14c, a container configuration reception unit 14d, a container placement destination determination unit 14e, and a container management unit 14f. The computational resource clusters 15A and 15B are configured to include a plurality of applications 15a and 15b.

[0014] The cluster management units 14A and 14B are also referred to as a cluster management unit 14, and the computational resource clusters 15A and 15B are also referred to as a computational resource cluster 15.

[0015] 1 performs anomaly detection and failure recovery for a container in which a failure has occurred in a container system 20. The recovery device 10 includes cluster management units 14A and 14B, computational resource clusters 15A and 15B, internal anomaly detection units 17A and 17B, anomaly recovery response units 18A and 18B, failure response deployment instruction units 19A and 19B, an allocation destination switching unit 21, and an external anomaly detection unit 23.

[0016] The internal anomaly detection units 17A and 17B are also referred to as internal anomaly detection unit 17, the anomaly recovery response units 18A and 18B are also referred to as an anomaly recovery response unit 18, and the failure response deployment instruction units 19A and 19B are also referred to as a failure response deployment instruction unit 19.

[0017] An internal anomaly detection unit 17, an anomaly recovery response unit 18, and a failure response deployment instruction unit 19 are provided inside each of the clusters 12A and 12B. An allocation destination switching unit 21 and an external anomaly detection unit 23 are provided outside each of the clusters 12A and 12B. However, the internal anomaly detection unit 17, the anomaly recovery response unit 18, the failure response deployment instruction unit 19, the allocation destination switching unit 21, and the external anomaly detection unit 23 are provided outside the cluster management unit 14 and the computational resource cluster 15 that are virtually created by the container virtualization software. In addition, the internal anomaly detection unit 17, the anomaly recovery response unit 18, and the failure response deployment instruction unit 19 may be provided outside each of the clusters 12A and 12B, similar to the allocation destination switching unit 21 and the external anomaly detection unit 23.

[0018] Since the first and second clusters 12A, 12B have substantially the same configuration, the functional configuration of the first cluster 12A will be described as a representative.

[0019] The computational resource cluster 15 is configured with a plurality of applications 15a and 15b. In other words, the applications 15a and 15b are pods (see pods 15a and 15b in FIG. 3) as a management unit of a collection of one or a plurality of containers. A pod is the smallest unit of an application that can be executed by CubeNetes (container virtualization software). In other words, containers are created and clustered by the applications 15a and 15b as pods, and the cluster is operated on a container engine. The computational resource cluster 15 is virtually created on a physical machine by container virtualization software, and the virtually created containers are clustered and arranged.

[0020] The cluster management unit 14 manages the control related to the placement and operation of the virtually created and clustered containers. The cluster management unit 14 includes a communication distribution unit 14a, a computational resource operation unit 14b, a computational resource management unit 14c, a container configuration reception unit 14d, a container placement destination determination unit 14e, and a container management unit 14f.

[0021] In the recovery device 10 configured as above, a failure response deployment instruction unit (also referred to as a deployment instruction unit) 19 performs a process of deploying (placing) the endpoint (end point) setting units 14j and 14k and the Pods 15a and 15b shown in Fig. 2 in a 1:1 configuration. The endpoint setting units 14j and 14k are associated with each of the multiple Pods 15a and 15b, and a traffic distribution ratio (%) to each of the Pods 15a and 15b is set to each of the endpoint setting units 14j and 14k, which become the end points of communication data.

[0022] The internal anomaly detection unit 17 shown in FIG. 1 detects an anomaly in one or more containers (pods) 15a and 15b (applications) in the container system 20.

[0023] The anomaly recovery response unit 18 changes the weight value of the deployment instruction unit 19 associated with a Pod (e.g., Pod 15a) in which an anomaly has been detected by the internal anomaly detection unit 17 to 0%, and transmits a change command to the communication sorting unit 14a to separate the abnormal Pod 15a. When recovering the separated Pod 15a, the anomaly recovery response unit 18 transmits a recovery command to the communication sorting unit 14a to gradually increase traffic to the Pod 15a to be restored up to a predetermined traffic value.

[0024] In the cluster management unit 14, the communication distribution unit 14a is a router that distributes and notifies the change command or recovery command from the abnormality recovery response unit 18 to the corresponding units 14b to 14f. The communication distribution unit 14a also distributes traffic to destination endpoint setting units 14j and 14k (described later) based on a weight value (%) indicating a traffic distribution ratio set for each of the endpoint setting units 14j and 14k (described later).

[0025] The container configuration receiving unit (also referred to as a receiving unit) 14d receives configuration information for deploying a container in the computational resource cluster 15 from an external server or the like.

[0026] The container placement destination determination unit (also referred to as placement destination determination unit) 14e determines which container is to be placed on which worker node (computing resource cluster 15) based on the configuration information received by the reception unit 14d.

[0027] The container management unit 14f checks whether the container is operating normally or not.

[0028] The computational resource management unit 14c grasps and manages whether a worker node is operable, the usage amount of computational resources of a server constituting a worker node, the remaining capacity of a CPU (Central Processing Unit), and the like.

[0029] The computational resource operation unit 14b performs an operation to allocate a predetermined amount of computational resources such as a certain amount of CPU to a certain container, in other words, an operation to allocate storage capacity, CPU time, memory capacity available to the container, and the like.

[0030] Next, various types of abnormality detection processing (first to sixth abnormality detection processing) related to the container of the container system 20 by the internal abnormality detection unit 17 of the recovery device 10 will be described with reference to Figs. 3 to 8.

[0031] <First abnormality detection process> 3 is a block diagram for explaining a first abnormality detection process of a container by the Pods (applications) 15a and 15b of the virtualization system recovery device 10 of the present embodiment. Each of the Pods 15a and 15b constitutes one or more containers.

[0032] 3, in the container system 20, a master node 14A, an infrastructure node 14B, and worker nodes 15A and 15B are configured by virtual machines, and each is connected by a virtual switch {OVS (Open vSwitch)} 30. However, the virtual switch may be other virtual switches besides OVS. The master node 14A and the infrastructure node 14B correspond to the cluster management units 14A and 14B (FIG. 1), and the worker nodes 15A and 15B correspond to the computational resource clusters 15A and 15B (FIG. 1).

[0033] Furthermore, the master node 14A and the worker node 15A form a first cluster 12, and the infrastructure node 14B and the worker node 15B form a second cluster 12. It is assumed that these clusters 12 form a container system 20.

[0034] An internal anomaly detection unit 17 is disposed outside the container system 20, similar to the configuration in Fig. 1. Although a total of two internal anomaly detection units 17 are depicted for each of the worker nodes 15A and 15B in Fig. 3, only one may be provided. The master node 14A, the infrastructure node 14B, the worker nodes 15A and 15B, and the internal anomaly detection unit 17 are connected to a counterpart device 24 via a network 22. The counterpart device 24 is a communication device, such as an external server, that transmits request signals and the like to the container system 20.

[0035] The internal anomaly detection unit 17 sends a predetermined command (e.g., "sudo crictl ps") to the Pods 15a and 15b of the worker nodes 15A and 15B by polling indicated by the round-trip arrows Y1 and Y2, and judges whether the Pods 15a and 15b are normal or abnormal based on the response results returned from the Pods 15a and 15b in response to the command. In this actual polling test, the average round-trip time when polling was performed 10 times was 0.06 seconds.

[0036] The internal anomaly detection unit 17 judges whether the container (Pod 15a, 15b) is operating normally by reading a character string indicating normality or abnormality that is written in the command response result returned from the Pods 15a, 15b by polling. For example, the character string "Running" indicates that the container (Pod 15a, 15b) is operating normally, and a character string other than "Running" indicates that the container is operating abnormally. Therefore, the internal anomaly detection unit 17 judges that the container (Pod 15a, 15b) is operating normally when the command response result contains "Running" and judges that the container is operating abnormally when the command response result contains a character string other than "Running."

[0037] <Second abnormality detection process> Next, FIG. 4 is a block diagram for explaining a second abnormality detection process by the routing table 15c provided for each of the worker nodes 15A and 15B of the virtualization system recovery device 10 of this embodiment.

[0038] The routing table (also called table) 15c uses route information indicating the destination container of a packet sent from the opposing device 24 to the Pods 15a and 15b of the worker nodes 15A and 15B via the network 22. If the destination management in this table 15c is incorrect, the packet will not reach the appropriate container. For this reason, the internal anomaly detection unit 17 is configured to detect whether the destination management in table 15c is normal or abnormal.

[0039] However, the routing table 15c is composed of a pair of tables, "iptables" and "nftables." Alternatively, the routing table 15c may be composed of only "iptables" or only "nftables."

[0040] The internal abnormality detection unit 17 sends a predetermined command to each table 15c of the worker nodes 15A, 15B by polling indicated by the reciprocating arrows Y3, Y4, and determines whether the table is normal or abnormal based on the response result returned from each table 15c in response to the command.

[0041] The above-mentioned predetermined command is a pair of "sudo iptables -L│wc-│" and "sudo nft list ruleset". The command "sudo iptables -L│wc-│" is notified to "iptables" in table 15c, and the command "sudo nft list ruleset" is notified to "nftables". Then, each of the tables "iptables" and "nftables" returns a response according to the command to the internal anomaly detection unit 17.

[0042] In actual testing of polling a pair of commands, the average round trip time for 10 polls was 0.03 seconds for the command “sudo iptables -L│wc-│” and 0.08 seconds for the command “sudo nft list ruleset.”

[0043] The internal anomaly detection unit 17 judges anomalies as normal if the command response result returned from each table 15c contains destination route information, and judges anomalies as abnormal if nothing is contained.

[0044] <Third abnormality detection process> 5 is a block diagram for explaining a third abnormality detection process by monitoring a daemon of the virtual switch 30 provided for each of the worker nodes 15A and 15B of the virtualization system recovery device 10 of this embodiment. The daemon of the virtual switch 30 is also called an OVS daemon.

[0045] The daemon is a program that manages the destination of packets in the virtual switch 30. The internal anomaly detection unit 17 monitors the OVS daemon, and detects that a packet is being sent properly if the packet is not being sent, and that an anomaly is being detected if the packet is not being sent.

[0046] The internal abnormality detection unit 17 sends a predetermined command (for example, "ps aux|grep ovs-vswitchd|grep "db.sock"|wc-│") to the virtual switch 30 for each worker node 15A, 15B by polling indicated by the round-trip arrows Y5, Y6, and judges whether the virtual switch 30 is normal or abnormal based on the response result returned from the virtual switch 30 in response to the command.

[0047] In this actual polling test, the average round trip time when polling was performed 10 times was 0.03 seconds.

[0048] The internal anomaly detection unit 17 judges an anomaly as normal if the command response result returned from each virtual switch 30 contains, for example, a “db.sock process” related to the destination, and judges an anomaly as abnormal if it does not contain such a process.

[0049] <Fourth abnormality detection process> Next, Fig. 6 is a block diagram for explaining a fourth abnormality detection process by monitoring a daemon of the container runtime 15d provided for each worker node 15A, 15B of the virtualization system recovery device 10 of the present embodiment. The above daemon is also called a crio daemon and is an example of the container runtime 15d. crio (cri-o) is an open source community-driven container engine used in container-based virtualization technology.

[0050] Since the container runtime 15d is responsible for starting the containers of the Pods 15a and 15b, it is possible to detect whether the containers are running normally by monitoring the container runtime 15d. Therefore, the internal anomaly detection unit 17 is configured to monitor the crio daemon and detect that the container is running normally if it is running, and detect that the container is not running abnormally if it is not running.

[0051] The internal abnormality detection unit 17 sends a predetermined command (e.g., “systemctl│status crio|grep Active”) to the container runtime 15d for each worker node 15A, 15B by polling indicated by the round-trip arrows Y7 and Y8, and determines whether the status is normal or abnormal based on the response result returned from each container runtime 15d in response to the command.

[0052] In this actual polling test, the average round trip time when polling was performed 10 times was 0.03 seconds.

[0053] The internal anomaly detection unit 17 judges an anomaly as normal if the command response result returned from each virtual switch 30 contains "active (running)", which indicates the startup state of the crio daemon, and judges an anomaly as abnormal if it contains anything other than "active (running)".

[0054] <Fifth abnormality detection process> Next, FIG. 7 is a block diagram for explaining a fifth abnormality detection process by monitoring each of the worker nodes 15A and 15B by the virtualization system recovery device 10 of this embodiment.

[0055] However, this is premised on a configuration in which the worker nodes 15A, 15B are created using a virtualization technology (virtual machine) using a physical machine 32. In this configuration, an internal abnormality detection unit 17 exists on the physical machine 32 outside the virtual machine, and this internal abnormality detection unit 17 detects that the container is normal if the virtual machine is running, and detects that the container is abnormal if the virtual machine is not running.

[0056] The internal abnormality detection unit 17 sends a specified command (e.g., "sudo virsh list") to each worker node 15A, 15B by polling indicated by the reciprocating arrows Y9, Y10, and determines whether the node is normal or abnormal based on the response results returned from each worker node 15A, 15B in response to the command.

[0057] In this actual polling test, the average round trip time when polling was performed 10 times was 0.03 seconds.

[0058] The internal anomaly detection unit 17 judges an anomaly as follows: if the command response result returned from each worker node 15A, 15B contains "running", which indicates the startup status of the target worker node 15A, 15B, then the command response result is judged to be normal; if the command response result contains anything other than "running", then the command response result is judged to be abnormal.

[0059] <6th abnormality detection process> Next, FIG. 8 is a block diagram for explaining a sixth abnormality detection process by monitoring DBs (Data Bases) 26a and 26b externally attached to the cluster 12 of the container system 20 of the virtualization system recovery device 10 of this embodiment.

[0060] As external devices to each cluster 12A, 12B (FIG. 1), DBs (also called external DBs) 26a, 26b that store data related to containers are configured to be connected to the worker nodes 15A, 15B via a network 22. In this case, the internal anomaly detection unit 17 is also connected to the worker nodes 15A, 15B via the network 22.

[0061] Here, since there is a configuration in which each cluster 12A, 12B is connected to each other via a network 22, even if the internal anomaly detection unit 17 is connected to the cluster 12 via the network 22 as shown in FIG. 8, it is positioned as the internal anomaly detection unit 17 in each cluster 12A, 12B, as shown in FIG. 1.

[0062] The internal abnormality detection unit 17 transmits a predetermined command to the external DBs 26a, 26b via the network 22 by polling as indicated by the reciprocating arrows Y11, Y12, and judges whether the external DBs 26a, 26b are normal or abnormal based on the response results returned from the external DBs 26a, 26b in response to the command. The command in this case depends on the type of the external DBs 26a, 26b.

[0063] The response results include results related to response / alive monitoring and results related to the number of connections being exceeded. The response / alive monitoring monitors whether the external DBs 26a, 26b are running normally. In other words, the internal anomaly detection unit 17 determines that an anomaly has occurred if the response result indicates that the external DBs 26a, 26b are not running normally.

[0064] The "Connections limit exceeded" indicates that the number of containers connected to the external DBs 26a and 26b exceeds a predetermined threshold. In other words, the internal anomaly detection unit 17 determines that an anomaly has occurred if the response result indicates that the number of connected containers to the external DBs 26a and 26b exceeds the threshold.

[0065] In this actual polling test, the polling round trip time depends on the type of the external DBs 26a and 26b.

[0066] <Multiple cluster anomaly detection 1> Next, a process of detecting an anomaly related to a fault when the fault occurs in a plurality of clusters 12A, 12B by the external anomaly detection unit 23 shown in Fig. 9 will be described. However, it is assumed that the anomaly detection 1 for each of the clusters 12A, 12B is any one of the first to sixth anomaly detections described above.

[0067] 9, the external anomaly detection unit 23 is connected to the internal anomaly detection unit 17A of the first cluster 12A and the internal anomaly detection unit 17B of the second cluster 12B. When the internal anomaly detection units 17A, 17B detect an anomaly related to any one of the first to sixth anomaly detections, the external anomaly detection unit 23 detects the cluster 12A or 12B in which the container related to the anomalous application 15a, 15b is placed as abnormal, as shown by the arrow Y31a or Y31b.

[0068] <Anomaly detection in multiple clusters 2> 9 is connected to a communication distribution unit 14a of a cluster management unit 14A in the first cluster 12A and a communication distribution unit 14a of a cluster management unit 14B in the second cluster 12B. The communication distribution unit 14a is provided at a signal input portion of the cluster management unit 14, and distributes and outputs an input signal to a subsequent stage, and also returns a response when the clusters 12A and 12B are normal in response to confirmation communication for each of the clusters 12A and 12B.

[0069] As indicated by the bidirectional arrows Y33a and Y33b, the external anomaly detection unit 23 performs confirmation communication with the communication distribution unit 14a for each cluster 12A, 12B at regular intervals and detects whether a response is returned normally or not. If no response is returned, it detects an anomaly in the corresponding cluster 12A, 12B.

[0070] In this multiple cluster anomaly detection 2, anomaly detection can be performed for each of the clusters 12A and 12B without going through the internal anomaly detection units 17A and 17B.

[0071] <Anomaly detection in multiple clusters 3> Anomaly detection 3 of multiple clusters is a process in which the external anomaly detection unit 23 performs anomaly detection for each of the clusters 12A and 12B by both of the above-mentioned anomaly detections 1 and 2. In this process, anomaly detection for each of the clusters 12A and 12B can be performed more appropriately.

[0072] 10 is a block diagram for explaining the anomaly handling process of the virtualization system recovery device 10 of this embodiment. The anomaly detection for which the anomaly handling process is performed is any one of the anomaly detections 1 to 3 of the multiple clusters described above.

[0073] The distribution destination switching unit 21 is connected to a DNS (Domain Name System) 25 outside the recovery device 10. The DNS 25 is a server that manages a domain (or a domain name) indicating the name of the application 15a, 15b of each cluster 12A, 12B in association with a resolved IP (Internet Protocol) address corresponding to the address of the communication distribution unit 14a of each cluster 12A, 12B. The DNS 25 converts between a domain and an IP address, and includes a DNS record table 25a.

[0074] 11, the DNS record table (also called a table) 25a stores domain names and resolved IP addresses in association with each other. In this example, in the table 25a, "Svc1.net" as the domain name of the application 15a for each of the clusters 12A and 12B is associated with "the IP address of the first cluster 12A" and "the IP address of the second cluster 12B" as the resolved IP addresses of the communication distribution unit 14a for each of the clusters 12A and 12B.

[0075] This association relationship indicates that the application 15a of "Svc1.net" runs on the first cluster 12A or the second cluster 12B.

[0076] Furthermore, in table 25a, “Svc2.net” as the domain name of application 15b for each cluster 12A, 12B is associated with “the IP address of the first cluster 12A” and “the IP address of the second cluster 12B” as the resolved IP addresses of communication distribution unit 14a for each cluster 12A, 12B.

[0077] This association relationship indicates that the "Svc2.net" application 15b runs on the first cluster 12A or the second cluster 12B.

[0078] When an external server (not shown) queries the DNS 25 having such a table 25a for a resolved IP address, the DNS 25 returns the IP addresses of both clusters 12A and 12B. This enables the external server to transmit data to either of the clusters 12A and 12B.

[0079] 10 detects one of the abnormality detections 1 to 3 in the multiple clusters 12A and 12B (for example, an abnormality in the second cluster 12B). The external anomaly detection unit 23 notifies the allocation destination switching unit 21 of the abnormality detection in the second cluster 12B as shown by arrow Y34.

[0080] The destination switching unit 21 notifies the DNS 25 of an instruction to stop the traffic distribution to the second cluster 12B (traffic distribution stop instruction) as shown by an arrow Y35. In response to the traffic distribution stop instruction, the DNS 25 performs a process of deleting the IP address of the second cluster 12B from the resolved IP addresses associated with both of the domain names "Svc1.net" and "Svc2.net" in the table 25a shown in FIG.

[0081] <Operation of the embodiment> Next, the operation of the abnormality handling process will be described with reference to the flowchart shown in FIG.

[0082] 13, it is assumed that a failure (marked with an x) occurs in the applications 15a and 15b of the second cluster 12B, and this abnormality is detected by the internal abnormality detection unit 17B. In this case, the internal abnormality detection unit 17B notifies the external abnormality detection unit 23 of the abnormality of the second cluster 12B, as shown by the arrow Y31b.

[0083] In step S2, the external anomaly detection unit 23 detects an anomaly in the second cluster 12B from the above notification, and notifies the allocation destination switching unit 21, as shown by an arrow Y34.

[0084] In step S3, the allocation destination switching unit 21 notifies the DNS 25 of an instruction to stop the communication allocation to the second cluster 12B, as indicated by an arrow Y35.

[0085] In step S4, the DNS 25 deletes the IP address of the second cluster 12B from the resolved IP addresses associated with both the domain names "Svc1.net" and "Svc2.net" in the table 25a shown in Fig. 12. As a result, the resolved IP address associated with both the domain names "Svc1.net" and "Svc2.net" in the table 25a becomes only the IP address of the first cluster 12A.

[0086] Therefore, in step S5, when an external server queries the DNS 25 for a resolved IP address, only the IP address of the first cluster 12A is returned from the DNS 25. In other words, the second cluster 12B where the failure has occurred cannot be accessed, and communication to the second cluster 12B is stopped.

[0087] <Effects of the embodiment> The effects of the virtualization system recovery device 10 according to the embodiment of the present invention will be described.

[0088] (1a) The recovery device 10 includes a computational resource cluster 15, a cluster manager 14, a plurality of clusters 12A, 12B, an internal anomaly detector 17, and an external anomaly detector 23. The computational resource cluster 15 is virtually created on a physical machine by container virtualization software, and the virtually created containers are arranged in a cluster. The cluster manager 14 manages control related to the arrangement and operation of the virtually created and clustered containers.

[0089] Each of the clusters 12A, 12B includes a computational resource cluster 15 and a cluster manager 14. The internal anomaly detection unit 17 is disposed in each of the clusters 12A, 12B, and is virtually created outside the virtually created computational resource cluster 15 and cluster manager 14 to detect anomalies in the containers. The external anomaly detection unit 23 is virtually created outside the clusters 12A, 12B, and is configured to detect, when the internal anomaly detection unit 17 detects an anomaly in a container, the cluster in which the abnormal container is disposed as an anomaly.

[0090] According to this configuration, when the internal anomaly detection unit 17 of each cluster 12A, 12B detects an anomaly in a container, the external anomaly detection unit 23 detects the cluster in which the abnormal container is placed as abnormal. The internal anomaly detection unit 17 and the external anomaly detection unit 23 are not involved in the container virtualization software that virtually creates the cluster management unit 14 and the computational resource cluster 15. For this reason, a failure that occurs in each cluster 12A, 12B can be detected earlier than the anomaly detection and recovery function of the container virtualization software. This early anomaly detection allows containers and the like related to the cluster failure to be quickly recovered.

[0091] (2a) The cluster management unit 14 includes a communication distribution unit 14a that is disposed at a signal input portion of the cluster management unit 14, distributes and outputs an input signal to a subsequent stage, and returns a response to a cluster confirmation communication when the cluster is normal. The external anomaly detection unit 23 is configured to perform cluster confirmation communication with the communication distribution unit 14a at a predetermined period, and detect an anomaly in the cluster when no response is returned.

[0092] According to this configuration, an abnormality in each cluster can be detected without going through the internal abnormality detection unit 17 for each of the clusters 12A and 12B.

[0093] (3a) The external anomaly detection unit 23 is configured to detect anomalies by both a process of detecting an abnormality in a cluster in which a container is placed when the internal anomaly detection unit 17 detects an abnormality in the container, and a process of sending confirmation communication of the cluster to the communication distribution unit 14a at a predetermined period and detecting an abnormality in the cluster if no response is returned.

[0094] According to this configuration, anomaly detection for each cluster can be performed more appropriately.

[0095] (4a) A DNS 25 that associates and manages a domain name indicating the name of an application related to a container with an IP address for each cluster is provided outside the servers that configure each cluster 12A, 12B. A distribution destination switching unit that is virtually created outside each cluster 12A, 12B and notifies the DNS 25 of a communication distribution stop instruction related to an abnormal cluster detected by the external anomaly detection unit 23 is provided for each cluster. The DNS 25 is configured to erase the IP address of the abnormal cluster indicated by the communication distribution stop instruction.

[0096] According to this configuration, the IP address of the cluster in which an abnormality has been detected by the external anomaly detection unit 23 is deleted from the IP addresses of the clusters managed by the DNS 25. Therefore, when an external server queries the DNS 25 for the IP address of the cluster, the IP address of the cluster in which the abnormality has occurred cannot be accessed. In other words, communication to the abnormal cluster can be stopped. The external anomaly detection unit 23, the allocation destination switching unit, and the DNS 25 are not involved in the above-mentioned container virtualization software. Therefore, a failure that has occurred in a cluster can be detected earlier than the anomaly detection and recovery function of the container virtualization software, and therefore the containers and the like related to the failure of the cluster in which the abnormality has been detected can be quickly recovered.

[0097] <First Modification of the Embodiment> FIG. 14 is a block diagram showing a configuration of a virtualization system recovery device 10A according to the first modification of the embodiment of the present invention.

[0098] The recovery device 10A of variant example 1 shown in Figure 14 differs from the recovery device 10 (Figure 10) in that the communication distribution stop instruction indicated by arrow Y35 from the distribution destination switching unit 21 is notified not only to DNS 25 but also to the communication distribution units 14a of each cluster 12A, 12B.

[0099] The communication sorting unit 14a stops communication with the first cluster 12A or the second cluster 12B indicated by the notified communication sorting stop instruction. In other words, since communication with each of the clusters 12A and 12B is always performed via the communication sorting unit 14a on the input side, the communication function of the communication sorting unit 14a is stopped in response to the communication sorting stop instruction.

[0100] According to this configuration, a command to stop communication distribution related to an abnormal cluster (e.g., the second cluster 12B) can be sent to the communication distribution unit 14a of the abnormal cluster 12B, and the communication function of the communication distribution unit 14a can be stopped. This stops the communication function of the abnormal cluster 12B, making it impossible to access the abnormal cluster 12B. This makes it possible to omit a query to the DNS 25 of the external server.

[0101] <Modification 2 of the embodiment> FIG. 15 is a block diagram showing a configuration of a virtualization system recovery device 10B according to the second modification of this embodiment.

[0102] The recovery device 10B of variant example 2 shown in Figure 15 differs from the recovery device 10 (Figure 10) in that it has an internal DNS 25A in the first cluster 12A and an internal DNS 25B in the second cluster 12B, and the communication distribution stop instruction indicated by arrow Y35 from the distribution destination switching unit 21 is notified not only to DNS 25 but also to each internal DNS 25A, 25B.

[0103] The internal DNSs 25A and 25B have a DNS record table 25a like the DNS 25, but the difference is that the table 25a is stored in a cache memory. Therefore, in the internal DNSs 25A and 25B, the information in the table 25a is erased at a predetermined time. However, the internal DNSs 25A and 25B can obtain necessary information from the DNS 25 as necessary after the information is erased.

[0104] The internal abnormality detection unit 17A (or the internal abnormality detection unit 17B) performs a process of erasing the IP address of the second cluster 12B from the table 25a shown in FIG. 12 in response to an instruction to stop communication sorting (see arrow Y35) when an abnormality in the second cluster 12B is detected.

[0105] According to this configuration, an external server can make an inquiry to the internal DNS 25A, 25B for each cluster about the IP address of each cluster 12A, 12B, so that the load on the external DNS 25 can be reduced.

[0106] <Hardware configuration> Any one of the virtualization system recovery devices 10, 10A, and 10B according to the above-described embodiments is realized by a computer 100 having a configuration as shown in Fig. 16. The computer 100 has a CPU (Central Processing Unit) 101, a ROM (Read Only Memory) 102, a RAM (Random Access Memory) 103, a HDD (Hard Disk Drive) 104, an input / output I / F (Interface) 105, a communication I / F 106, and a media I / F 107.

[0107] The CPU 101 operates based on a program stored in the ROM 102 or the HDD 104, and controls each functional unit. The ROM 102 stores a boot program executed by the CPU 101 when the computer 100 is started up, programs related to the hardware of the computer 100, and the like.

[0108] The CPU 101 controls an output device 111 such as a printer or a display, and an input device 110 such as a mouse or a keyboard, via the input / output I / F 105. The CPU 101 obtains data from the input device 110 via the input / output I / F 105, or outputs generated data to the output device 111.

[0109] The HDD 104 stores programs executed by the CPU 101 and data used by the programs, etc. The communication I / F 106 receives data from other devices (not shown) via a communication network 112 and outputs the data to the CPU 101, and also transmits data generated by the CPU 101 to other devices via the communication network 112.

[0110] The media I / F 107 reads a program or data stored in the recording medium 113 and outputs it to the CPU 101 via the RAM 103. The CPU 101 loads a program related to a target process from the recording medium 113 onto the RAM 103 via the media I / F 107, and executes the loaded program. The recording medium 113 is an optical recording medium such as a DVD (Digital Versatile Disc) or a PD (Phase change rewritable Disc), a magneto-optical recording medium such as an MO (Magneto Optical disk), a magnetic recording medium, a conductive memory tape medium, a semiconductor memory, or the like.

[0111] For example, when the computer 100 functions as any one of the virtualization system recovery devices 10, 10A, and 10B according to the embodiment, the CPU 101 of the computer 100 executes a program loaded onto the RAM 103 to realize the functions of the virtualization system recovery device 10. Furthermore, the HDD 104 stores data in the RAM 103. The CPU 101 reads and executes a program relating to a target process from the recording medium 113. Additionally, the CPU 101 may read a program relating to a target process from another device via the communication network 112. <Effects> (1) A virtualization system recovery device comprising: a computing resource cluster that is virtually created on a physical machine by container virtualization software and that clusters and arranges the virtually created containers; a cluster management unit that manages control related to the arrangement and operation of the virtually created and clustered containers; a plurality of clusters, each of which is configured to have the computing resource cluster and the cluster management unit; an internal anomaly detection unit that is arranged for each of the plurality of clusters and is virtually created outside the virtually created computing resource cluster and the cluster management unit and that detects an anomaly in the container; and an external anomaly detection unit that is virtually created outside the plurality of clusters and that, when an anomaly in a container is detected by the internal anomaly detection unit, detects as abnormal a cluster in which the abnormal container is arranged.

[0112] According to this configuration, when an anomaly is detected in a container by the internal anomaly detection unit for each of the multiple clusters, the external anomaly detection unit detects the cluster in which the abnormal container is placed as abnormal. The internal anomaly detection unit and the external anomaly detection unit are not involved in the cluster management unit and the container virtualization software that virtually creates the computing resource cluster. For this reason, anomalies that occur in multiple clusters can be detected more quickly than the anomaly detection and recovery function of the container virtualization software. This early anomaly detection allows containers, etc. related to the cluster failure to be quickly recovered.

[0113] (2) The virtualization system recovery device according to (1) above, characterized in that the cluster management unit is provided at a signal input portion of the cluster management unit, distributes and outputs an input signal to a subsequent stage, and includes a communication distribution unit that returns a response when the cluster is normal in response to a confirmation communication of the cluster, and the external anomaly detection unit performs cluster confirmation communication with the communication distribution unit at a predetermined period, and detects an anomaly in the cluster when the response is not returned.

[0114] According to this configuration, an abnormality in each cluster can be detected without going through an internal abnormality detection unit for each of the multiple clusters.

[0115] (3) The external anomaly detection unit is the virtualization system recovery device described in (2) above, characterized in that it detects an anomaly by both a process of detecting an anomaly in a cluster in which a container is placed when the internal anomaly detection unit detects an anomaly in the container, and a process of sending a cluster confirmation communication to the communication distribution unit at a predetermined period and detecting an anomaly in the cluster when no response is returned.

[0116] According to this configuration, anomaly detection for each cluster can be performed more appropriately.

[0117] (4) A virtualization system recovery device according to any one of (1) to (3) above, characterized in that a DNS (Domain Name System) that associates and manages a domain name indicating the name of an application related to the container with an IP (Internet Protocol) address for each of the clusters is provided outside a server constituting the cluster, and a distribution destination switching unit that is virtually created outside the multiple clusters and notifies the DNS of a communication distribution stop instruction for an abnormal cluster detected by the external anomaly detection unit is provided for each of the clusters, and the DNS erases the IP address of the abnormal cluster indicated by the communication distribution stop instruction.

[0118] According to this configuration, the IP address of the cluster in which an abnormality has been detected by the external anomaly detection unit is erased from the IP addresses of the clusters managed by the DNS. Therefore, when an external server queries the DNS for the IP address of the cluster, it becomes impossible to access the IP address of the cluster in which the abnormality has occurred. In other words, communication to the abnormal cluster can be stopped. The external anomaly detection unit, the distribution destination switching unit, and the DNS are not involved in the above-mentioned container virtualization software. Therefore, a failure that has occurred in a cluster can be detected earlier than the anomaly detection and recovery function of the container virtualization software, so that containers and the like related to the failure of the cluster in which the abnormality has been detected can be quickly recovered.

[0119] (5) The virtualization system recovery device described in (4) above is characterized in that the distribution destination switching unit notifies the communication distribution units in the multiple clusters of an instruction to stop communication distribution related to the cluster of the abnormality detected by the external anomaly detection unit, and the communication distribution unit performs processing to stop communication function when the instruction to stop communication distribution related to the cluster of the abnormality detected by the external anomaly detection unit is notified.

[0120] According to this configuration, a command to stop traffic sorting for the abnormal cluster can be sent to the traffic sorting unit of the abnormal cluster, and the communication function of the traffic sorting unit can be stopped. This stops the traffic sorting function from being accessible to the abnormal cluster, and therefore queries to the DNS of the external server can be omitted.

[0121] (6) The virtualization system recovery device described in (4) above is characterized in that, for each of the multiple clusters, an internal DNS is provided which, similar to the DNS, associates and manages a domain name indicating the name of the application related to the container with an IP address for each of the clusters, and notifies the internal DNS of a communication distribution stop instruction from the distribution destination switching unit.

[0122] According to this configuration, an external server can make an inquiry to the internal DNS for each cluster about the IP address of the cluster, thereby reducing the load on the external DNS.

[0123] In addition, the specific configuration can be modified as appropriate without departing from the spirit of the present invention. [Explanation of symbols]

[0124] 10,10A,10B Virtualization system recovery device 12A 1st Cluster (Cluster) 12B 2nd Cluster (Cluster) 14A, 14B Cluster Management Unit 14a Communication distribution section 14b Computing resource operation section 14c Computing Resource Management Department 14d Container configuration reception section 14e Container placement decision unit 14th floor Container Management Department 15A, 15B Computing Resource Cluster 15a,15b Application 17A, 17B Internal abnormality detection section 21 Distribution destination switching unit 23 External anomaly detection unit 25 DNS 25a DNS Record Table 25A,25B Internal DNS

Claims

1. a computing resource cluster that is virtually created on a physical machine by container virtualization software and that clusters and arranges the virtually created containers; a cluster management unit that manages control related to the placement and operation of the virtually created and clustered containers; A plurality of clusters, each of which includes the computational resource cluster and the cluster manager; an internal anomaly detection unit that is disposed for each of the plurality of clusters, that is virtually created outside the virtually created computing resource clusters and the cluster management unit, and that detects an anomaly in the container; an external anomaly detection unit that is virtually created outside the plurality of clusters and detects, when an anomaly is detected in a container by the internal anomaly detection unit, a cluster in which the abnormal container is located as abnormal; a distribution destination switching unit that is virtually created outside the plurality of clusters and notifies a DNS (Domain Name System) of a command to stop distribution of traffic related to an abnormal cluster detected by the external abnormality detection unit; the DNS that associates and manages a domain name indicating a name of an application related to the container with an IP (Internet Protocol) address for each cluster is provided outside a server that constitutes the cluster; The DNS deletes the IP address of the abnormal cluster indicated in the communication distribution stop instruction. A virtualization system recovery device comprising:

2. the cluster management unit includes a communication distribution unit that is disposed at a signal input portion of the cluster management unit, distributes and outputs an input signal to a subsequent stage, and returns a response in response to a confirmation communication of the cluster when the cluster is normal; The external anomaly detection unit performs a cluster confirmation communication to the communication sorting unit at a predetermined period, and detects an anomaly in the cluster when the response is not returned.

2. The virtualization system recovery device according to claim 1.

3. The external anomaly detection unit detects an anomaly by both a process of detecting an anomaly in a cluster in which the container is placed when the internal anomaly detection unit detects an anomaly in the container, and a process of sending a confirmation communication of the cluster to the communication sorting unit at a predetermined period and detecting an anomaly in the cluster when the response is not returned.

3. The virtualization system recovery device according to claim 2.

4. the distribution destination switching unit notifies communication distribution units in the plurality of clusters of a communication distribution stop instruction related to a cluster in which an abnormality has been detected by the external abnormality detection unit, The communication sorting unit performs processing to stop a communication function when a command to stop sorting communication related to a cluster of an abnormality detected by the external abnormality detection unit is notified.

2. The virtualization system recovery device according to claim 1.

5. an internal DNS for managing, for each of the plurality of clusters, a domain name indicating a name of an application related to the container and an IP address for each of the clusters in association with each other, in the same manner as the DNS; The internal DNS is notified of a communication distribution stop instruction from the distribution destination switching unit.

2. The virtualization system recovery device according to claim 1.

6. A method for recovering a virtualization system by a virtualization system recovery device, comprising: Virtualization system recovery device, A plurality of clusters in which containers virtually created by container virtualization software are clustered and arranged on a physical machine; The virtual internal anomaly detection unit and the virtual external anomaly detection unit are provided outside the plurality of clusters, a DNS that associates and manages a domain name indicating a name of an application related to the container with an IP address for each cluster, the DNS being provided outside the server that constitutes the cluster; Virtualization system recovery device, the internal anomaly detection unit detecting an anomaly in a container for each of the plurality of clusters; a step in which the external anomaly detection unit detects, when the internal anomaly detection unit detects an anomaly in a container, a cluster in which the abnormal container is located as an anomaly; notifying the DNS of a command to stop traffic sorting related to the cluster of anomalies detected by the external anomaly detection unit; The DNS executes a step of deleting the IP address of the abnormal cluster indicated in the communication sorting stop instruction. A virtualization system recovery method comprising:

Images