Virtualization system fault isolation device and virtualization system fault isolation method

Abstract

The present invention comprises: a calculation resource cluster (15) virtually created in a physical machine by container virtualization software, the calculation resource cluster (15) clustering and arranging virtually created containers; and a cluster management unit 14 for managing control pertaining to the arrangement and motion of the virtually created and clustered containers. Furthermore, the present invention includes: an abnormality detection unit (17) created on the outside of a virtually created constituent element, the abnormality detection unit (17) detecting an abnormality in the containers; and an abnormality recovery response unit (18) created on said outside, the abnormality recovery response unit (18) transmitting a change command to the cluster management unit (14) upon detection by the abnormality detection unit (17). The cluster management unit (14) is configured such that the allocation rate of an endpoint setting unit (14j) associated with an abnormal container is set to 0% in response to the change command.

Description

[Technical field]

[0001] The present invention relates to a virtualization system fault isolation device and a virtualization system fault isolation method that realize anomaly detection and fault recovery for containers and applications running on containers in a computing platform based on virtual machines or containers. [Background technology]

[0002] The above-mentioned virtual machine is a computer that realizes the same functions as a physical computer through software. A container is a virtualization technology that is created by packaging an application in an environment called a "container" and runs on a container engine. In conventional container-based technologies, anomaly detection and failure recovery of containers and applications running on containers are realized mainly by the Liveness / Readiness Probe function (also called the probe function) of Kubernetes, which will be described later.

[0003] Kubernetes is container virtualization software that creates containers such as Docker and clusters them, and is also open source software. The liveness probe function controls whether or not a container should restart, and the readiness probe function controls whether or not a container should accept a request. A conventional technology of this type is described in Non-Patent Document 1. [Prior art documents] [Non-patent literature]

[0004] [Non-Patent Document 1] "Using Liveness Probes, Readiness Probes, and Startup Probes," kubernetes, [online], [Retrieved June 10, 2021], Internet〈https: / / kubernetes.io / ja / docs / tasks / configure-pod-container / configure-liveness-readiness-startup-probes / 〉 Summary of the Invention [Problem to be solved by the invention]

[0005] Incidentally, in virtualization systems in the field of virtualization technology, not limited to the containers mentioned above, manual recovery work is performed based on an alert issued in response to a failure within the virtualization system. However, since manual recovery work is performed after an alert is issued, it is difficult to reduce the time from the occurrence of a failure to normalization.

[0006] When recovering from such a fault using the probe function that performs fault recovery in Kubernetes, the fault monitoring period can only be set to a slow period such as one second, which is a predefined interval. Therefore, when the fastest possible fault recovery is required, it is not possible to recover faster than the default fault recovery function in Kubernetes.

[0007] The present invention has been made in consideration of the above circumstances, and its objective is to recover from a failure that occurs in a virtualization system faster than the failure recovery function of the container virtualization software. [Means for solving the problem]

[0008] In order to solve the above problems, a virtualization system fault isolation device of the present invention includes a computational resource cluster that is virtually created on a physical machine by container virtualization software and that clusters and arranges the virtually created containers, a cluster management unit that manages control related to the arrangement and operation of the virtually created and clustered containers, a deployment instruction unit that performs a process of associating an endpoint setting unit with a container, the endpoint setting unit being associated with each of a plurality of containers and serving as an end point of communication data in which an allocation ratio of traffic to each container is set, an anomaly detection unit that is created outside the virtually created computational resource cluster and the cluster management unit and that detects anomalies in the containers, and an anomaly recovery response unit that is created outside the virtually created computational resource cluster and the cluster management unit and that transmits a change command to the cluster management unit to set the allocation ratio for an abnormal container detected by the anomaly detection unit to 0%, and the cluster management unit sets the allocation ratio of the endpoint setting unit associated with the abnormal container to 0% in response to the change command. The abnormality recovery response unit transmits a recovery command to the cluster management unit when recovering the abnormal container, for gradually increasing traffic to the container to be recovered up to a predetermined traffic value, and the cluster management unit gradually increases an allocation ratio of an endpoint setting unit associated with the container to be recovered up to the predetermined traffic value in response to the recovery command. It is characterized by: Effect of the Invention

[0009] According to the present invention, a failure that occurs in a virtualization system can be recovered from more quickly than the failure recovery function of the container virtualization software. [Brief description of the drawings]

[0010] [Figure 1] 1 is a block diagram showing a configuration of a virtualization system fault isolation device according to an embodiment of the present invention. [Diagram 2] FIG. 11 is a block diagram for explaining a first abnormality detection process of a container by a Pod of the virtualization system fault isolation device of this embodiment. [Diagram 3] 11 is a block diagram for explaining a second abnormality detection process using a routing table provided for each worker node of the virtualization system fault isolation device of the present embodiment. FIG. [Figure 4]FIG. 13 is a block diagram for explaining a third abnormality detection process by monitoring a daemon of a virtual switch provided for each worker node of the virtualization system fault isolation device of this embodiment. [Diagram 5] FIG. 11 is a block diagram for explaining a fourth abnormality detection process by monitoring a daemon of a container runtime provided for each worker node of the virtualization system fault isolation device of this embodiment. [Figure 6] FIG. 13 is a block diagram for explaining a fifth abnormality detection process by monitoring each worker node of the virtualization system fault isolation device of the present embodiment. [Figure 7] FIG. 13 is a block diagram for explaining a sixth abnormality detection process by monitoring a DB externally attached to a cluster of a container system of the virtualization system fault isolation device of this embodiment. [Figure 8] 13 is a block diagram showing a configuration when an endpoint setting unit and a Pod are deployed in a 1:1 configuration by a failure response deployment instruction unit in the virtualization system fault isolation device of this embodiment. FIG. [Figure 9] 4 is a block diagram for explaining a first anomaly response process of the virtualization system fault isolation device of the present embodiment. FIG. [Figure 10] 10 is a flowchart for explaining the operation of the first anomaly response process. [Figure 11] 11 is a block diagram for explaining a second anomaly response process of the virtualization system fault isolation device of the present embodiment. FIG. [Figure 12] 1 is a hardware configuration diagram showing an example of a computer that realizes the functions of a virtualization system failure isolation device according to the present embodiment. FIG. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. However, in all the drawings in this specification, components having corresponding functions are given the same reference numerals, and the description thereof will be omitted as appropriate. <Configuration of the embodiment> FIG. 1 is a block diagram showing a configuration of a virtualization system fault isolation device according to an embodiment of the present invention.

[0012] A virtualization system fault isolation device (also referred to as a fault isolation device) 10 shown in Fig. 1 stops or deletes a container in which a fault has occurred in a container system 20 described below, thereby isolating the container, and recovering the container after isolation. This fault isolation device 10 is configured to include a cluster management unit 14, a computational resource cluster 15, an anomaly detection unit 17, an anomaly recovery response unit 18, and a fault response deployment instruction unit 19. The cluster management unit 14 and the computational resource cluster 15 configure a cluster 12. The anomaly detection unit 17, the anomaly recovery response unit 18, and the fault response deployment instruction unit 19 are provided outside this cluster 12. The fault response deployment instruction unit 19 constitutes a deployment instruction unit recited in the claims.

[0013] The computing resource cluster 15 is configured with a plurality of applications 15a and 15b. In other words, the applications 15a and 15b are pods that are management units of a collection of one or a plurality of containers. A pod is the smallest unit of an application that can be executed by CubeNetes (container virtualization software). In other words, containers are created and clustered by the applications 15a and 15b as pods, and the cluster is operated on a container engine. The computing resource cluster 15 is virtually created on a physical machine by container virtualization software, and the virtually created containers are clustered and arranged.

[0014] The container system 20 is a virtualization system configured with one or more clusters 12. When there are two clusters 12, each cluster 12 is configured to include a cluster manager 14 and a computational resource cluster 15.

[0015] The cluster management unit 14 manages the control related to the placement and operation of the virtually created and clustered containers. The cluster management unit 14 includes a communication distribution unit 14a, a computational resource operation unit 14b, a computational resource management unit 14c, a container configuration reception unit 14d, a container placement destination determination unit 14e, and a container management unit 14f.

[0016] In the fault isolation device 10 configured as above, the fault response deployment instruction unit (also referred to as the deployment instruction unit) 19 performs a process of deploying (placement) the endpoint (end point) setting units 14j and 14k and the Pods 15a and 15b shown in Fig. 8 as a 1:1 configuration. The endpoint setting units 14j and 14k are associated with each of the multiple Pods 15a and 15b, and a traffic distribution ratio (%) to each of the Pods 15a and 15b is set, and the endpoint setting units 14j and 14k become the end points of communication data. The distribution ratio is referred to as a weight value (%).

[0017] The abnormality detection unit 17 shown in FIG. 1 detects an abnormality in Pods (applications) 15a and 15b, which are one or more containers in a container system 20.

[0018] The anomaly recovery response unit 18 changes the weight value of the deployment instruction unit 19 associated with a Pod (e.g., Pod 15a) in which an anomaly has been detected by the anomaly detection unit 17 to 0%, and transmits a change command to the communication sorting unit 14a to separate the abnormal Pod 15a. When recovering the separated Pod 15a, the anomaly recovery response unit 18 transmits a recovery command to the communication sorting unit 14a to gradually increase traffic to the Pod 15a to be restored up to a predetermined traffic value.

[0019] 1 is a router that distributes and notifies the change command or recovery command from the abnormality recovery response unit 18 to the corresponding units 14b to 14f. The communication distribution unit 14a also distributes traffic to the destination endpoint setting units 14j and 14k (described later) based on a weight value (%) indicating a traffic distribution ratio set for each of the endpoint setting units 14j and 14k described later. The weight value corresponds to the distribution ratio described in the claims.

[0020] The container configuration receiving unit (also referred to as a receiving unit) 14d receives configuration information for deploying a container in the computational resource cluster 15 from an external server or the like.

[0021] The container placement destination determination unit (also referred to as placement destination determination unit) 14e determines which container is to be placed on which worker node (computing resource cluster 15) based on the configuration information received by the reception unit 14d.

[0022] The container management unit 14f checks whether the container is operating normally or not.

[0023] The computational resource management unit 14c grasps and manages whether a worker node is operable, the usage amount of computational resources of a server constituting a worker node, the remaining capacity of a CPU (Central Processing Unit), and the like.

[0024] The computational resource operation unit 14b performs an operation to allocate a predetermined amount of computational resources such as a certain amount of CPU to a certain container, in other words, an operation to allocate storage capacity, CPU time, memory capacity available to the container, and the like.

[0025] Next, various types of abnormality detection processing (first to sixth abnormality detection processing) related to the container of the container system 20 by the abnormality detection unit 17 of the fault isolation device 10 will be described with reference to Figs. 2 to 7.

[0026] <First abnormality detection process> 2 is a block diagram for explaining a first abnormality detection process of a container by the Pods (applications) 15a and 15b of the virtualization system fault isolation device 10 of the present embodiment. Each of the Pods 15a and 15b includes one or more containers.

[0027] 2, in a container system 20, a master node 14J, an infrastructure node 14K, and worker nodes 15J and 15K are configured by virtual machines, and each is connected by a virtual switch {OVS (Open vSwitch)} 30. The master node 14J and the infrastructure node 14K correspond to the cluster management unit 14 (FIG. 1), and the worker nodes 15J and 15K correspond to the computational resource cluster 15 (FIG. 1).

[0028] Furthermore, the master node 14J and the worker node 15J constitute a first cluster 12, and the infrastructure node 14K and the worker node 15K constitute a second cluster 12. It is assumed that a container system 20 is constituted by these clusters 12.

[0029] An abnormality detection unit 17 is arranged outside the container system 20, similar to the configuration in Fig. 1. In Fig. 2, a total of two abnormality detection units 17 are illustrated for each worker node 15J, 15K, but there may be only one. The master node 14J, the infrastructure node 14K, the worker nodes 15J, 15K, and the abnormality detection unit 17 are connected to a counterpart device 24 via a network 22. The counterpart device 24 is a communication device such as an external server that transmits request signals and the like to the container system 20.

[0030] The anomaly detection unit 17 transmits a predetermined command (e.g., "sudo crictl ps") to the Pods 15a and 15b of the worker nodes 15J and 15K by polling indicated by the round-trip arrows Y1 and Y2, and judges whether the Pods 15a and 15b are normal or abnormal based on the response results sent back from the Pods 15a and 15b in response to the command. In this actual polling test, the average round-trip time when polling was performed 10 times was 0.06 seconds.

[0031] The anomaly detection unit 17 judges whether an abnormality exists by reading a character string indicating normality or abnormality described in the command response result returned from the Pods 15a and 15b by polling. For example, the character string "Running" indicates that the operation of the container (Pods 15a and 15b) is normal, and a character string other than "Running" indicates that the operation is abnormal. For this reason, the anomaly detection unit 17 judges that the operation of the container (Pods 15a and 15b) is normal when "Running" is described in the command response result, and judges that the operation is abnormal when a character string other than "Running" is described.

[0032] <Second abnormality detection process> Next, FIG. 3 is a block diagram for explaining a second abnormality detection process by the routing table 15c provided for each of the worker nodes 15J, 15K of the virtualization system fault isolation device 10 of this embodiment.

[0033] The routing table (also called table) 15c uses route information indicating the destination container of a packet sent from the opposing device 24 to the pods 15a and 15b of the worker nodes 15J and 15K via the network 22. If the destination management in this table 15c is incorrect, the packet will not reach the appropriate container. For this reason, the anomaly detection unit 17 is configured to detect whether the destination management in table 15c is normal or abnormal.

[0034] However, the routing table 15c is composed of a pair of tables, "iptables" and "nftables".

[0035] The abnormality detection unit 17 sends a predetermined command to each table 15c of the worker nodes 15J, 15K by polling indicated by the reciprocating arrows Y3, Y4, and determines whether the table is normal or abnormal based on the response result returned from each table 15c in response to the command.

[0036] The above-mentioned predetermined command is a pair of "sudo iptables -L│wc-│" and "sudo nft list ruleset". The command "sudo iptables -L│wc-│" is notified to "iptables" in table 15c, and the command "sudo nft list ruleset" is notified to "nftables". Then, each of the tables "iptables" and "nftables" returns a response according to the command to the anomaly detection unit 17.

[0037] In actual testing of polling a pair of commands, the average round trip time for 10 polls was 0.03 seconds for the command “sudo iptables -L│wc-│” and 0.08 seconds for the command “sudo nft list ruleset.”

[0038] The abnormality detection unit 17 judges an abnormality as follows: if the command response result returned from each table 15c contains destination route information, it is judged as normal; if nothing is contained, it is judged as abnormal.

[0039] <Third abnormality detection process> 4 is a block diagram for explaining a third abnormality detection process by monitoring a daemon of a virtual switch 30 provided for each worker node 15J, 15K of the virtualization system fault isolation device 10 of this embodiment. The daemon of the virtual switch 30 is also called an OVS daemon.

[0040] The daemon is a program that manages the destination of packets in the virtual switch 30. The abnormality detection unit 17 monitors the OVS daemon, and detects that a packet is being transmitted normally if the packet is being transmitted properly, and detects that a packet is not being transmitted abnormally if the packet is not being transmitted properly.

[0041] The abnormality detection unit 17 sends a predetermined command (for example, "ps aux|grep ovs-vswitchd|grep "db.sock"|wc-│") to the virtual switch 30 for each worker node 15J, 15K by polling indicated by the reciprocating arrows Y5, Y6, and judges whether the virtual switch 30 is normal or abnormal based on the response result returned from the virtual switch 30 in response to the command.

[0042] In this actual polling test, the average round trip time when polling was performed 10 times was 0.03 seconds.

[0043] The abnormality detection unit 17 judges an abnormality as normal if the command response result returned from each virtual switch 30 contains, for example, a “db.sock process” related to the destination, and judges an abnormality as abnormal if it does not contain such a process.

[0044] <Fourth abnormality detection process> 5 is a block diagram for explaining a fourth abnormality detection process by monitoring a daemon of the container runtime 15d provided for each worker node 15J, 15K of the virtualization system fault isolation device 10 of the present embodiment. The daemon of the container runtime 15d is also called a crio daemon. crio (cri-o) is an open source community-driven container engine used in container-based virtualization technology.

[0045] Since the container runtime 15d is responsible for starting the containers of the Pods 15a and 15b, it is possible to detect whether the containers are running normally by monitoring the container runtime 15d. Therefore, the anomaly detection unit 17 is configured to monitor the crio daemon and detect that the container is running normally if it is running, and detect that the container is not running as an anomaly.

[0046] The abnormality detection unit 17 sends a predetermined command (e.g., "systemctl status crio | grep Active") to the container runtime 15d for each worker node 15J, 15K by polling indicated by the reciprocating arrows Y7, Y8, and determines whether the status is normal or abnormal based on the response result returned from each container runtime 15d in response to the command.

[0047] In this actual polling test, the average round trip time when polling was performed 10 times was 0.03 seconds.

[0048] The abnormality detection unit 17 judges an abnormality as normal if the command response result returned from each virtual switch 30 contains "active (running)", which indicates the startup state of the crio daemon, and judges an abnormality as abnormal if it contains anything other than "active (running)".

[0049] <Fifth abnormality detection process> Next, FIG. 6 is a block diagram for explaining a fifth abnormality detection process by monitoring each of the worker nodes 15J and 15K by the virtualization system fault isolation device 10 of this embodiment.

[0050] However, this is premised on a configuration in which the worker nodes 15J, 15K are created using a virtualization technology (virtual machine) by the physical machine 32. In this configuration, an abnormality detection unit 17 exists on the physical machine 32 outside the virtual machine, and this abnormality detection unit 17 detects that the container is normal if the virtual machine is running, and detects that the container is abnormal if the virtual machine is not running.

[0051] The abnormality detection unit 17 sends a specified command (e.g., "sudo virsh list") to each worker node 15J, 15K by polling indicated by the reciprocating arrows Y9, Y10, and determines whether the node is normal or abnormal based on the response results returned from each worker node 15J, 15K in response to the command.

[0052] In this actual polling test, the average round trip time when polling was performed 10 times was 0.03 seconds.

[0053] The abnormality detection unit 17 judges an abnormality as follows: if the command response result returned from each worker node 15J, 15K contains "running", which indicates the startup status of the target worker node 15J, 15K, then it is judged to be normal; if it contains anything other than "running", then it is judged to be abnormal.

[0054] <6th abnormality detection process> Next, FIG. 7 is a block diagram for explaining a sixth abnormality detection process by monitoring DBs (Data Bases) 26a, 26b externally attached to the cluster 12 of the container system 20 by the virtualization system failure isolation device 10 of this embodiment.

[0055] As external devices to the cluster 12 (FIG. 1), DBs (also called external DBs) 26a and 26b that store data related to containers are configured to be connected to the worker nodes 15J and 15K via the network 22. In this case, the anomaly detection unit 17 is also connected to the worker nodes 15J and 15K via the network 22.

[0056] Here, since there is a configuration in which multiple clusters 12 are connected to each other via a network 22, even if the anomaly detection unit 17 is connected to the cluster 12 via a network 22 as shown in FIG. 7, it is positioned as the anomaly detection unit 17 within the fault isolation device 10, as shown in FIG. 1.

[0057] The abnormality detection unit 17 transmits a predetermined command to the external DBs 26a, 26b via the network 22 by polling as indicated by the reciprocating arrows Y11, Y12, and judges whether the external DBs 26a, 26b are normal or abnormal based on the response results returned from the external DBs 26a, 26b in response to the command. The command in this case depends on the type of the external DBs 26a, 26b.

[0058] The response results include results related to response / alive monitoring and results related to the number of connections being exceeded. The response / alive monitoring monitors whether the external DBs 26a and 26b are running normally. In other words, the anomaly detection unit 17 determines that an anomaly has occurred if the response result indicates that the external DBs 26a and 26b are not running normally.

[0059] The "Connections limit exceeded" indicates that the number of containers connected to the external DBs 26a and 26b exceeds a predetermined threshold. In other words, the abnormality detection unit 17 determines that an abnormality has occurred if the response result indicates that the number of connected containers to the external DBs 26a and 26b exceeds the threshold.

[0060] In this actual polling test, the polling round trip time depends on the type of the external DBs 26a and 26b.

[0061] Next, the abnormality response process when the above-mentioned first to sixth abnormalities are detected will be described. However, as shown in FIG. 8, it is assumed that the failure response deployment instruction unit 19 deploys (places) the endpoint setting units 14j and 14k and the Pods 15a and 15b in a 1:1 configuration.

[0062] The endpoint setting units 14j and 14k are end points that receive service information related to the communication indicated by the arrow Y20 from the opposing device 24 via the router 14a, and are accessible to the Pods 15a and 15b. In other words, the service information from the opposing device 24 is transmitted from the router 14a to the containers in the Pods 15a and 15b via the endpoint setting units 14j and 14k. In addition, a weight value (%) indicating a traffic allocation ratio is set for each of the endpoint setting units 14j and 14k.

[0063] The router 14a distributes traffic to the destination endpoint setting units 14j and 14k based on the weight values, as indicated by arrows Y16 and Y17. For example, assume that the weight value of the endpoint setting unit 14j is set to 30%, and the weight value of the endpoint setting unit 14k is set to 70%. In this case, 30% of the data transmitted from the router 14a is distributed to the endpoint setting unit 14j in the direction indicated by arrow Y16, and 70% is distributed to the endpoint setting unit 14k in the direction indicated by arrow Y17.

[0064] <First abnormality response process> 9 is a block diagram for explaining the first anomaly response processing of the virtualization system failure isolation device 10 of this embodiment. The anomaly detection that requires the first anomaly response processing is any one of the first to fifth anomaly detections.

[0065] When the abnormality detection unit 17 shown in Figure 9 detects an abnormality in a pod (for example, pod 15a of worker node 15J), the abnormality recovery response unit 18 sends a change command to the router 14a of the master node 14J to set the traffic distribution ratio to 0% for the abnormal pod 15a, as shown by arrow Y21.

[0066] The router 14a performs a process of isolating the abnormal Pod 15a by setting the weight value of the endpoint setting unit 14j associated with the abnormal Pod 15a of the worker node 15J to 0% as indicated by an arrow Y22 (see the cross mark). At this time, the router 14a performs a process of setting the weight value of the endpoint setting unit 14k associated with the Pod 15a of the worker node 15K to 100% as indicated by an arrow Y23, if necessary.

[0067] By setting this weight value, the abnormal Pod 15a is isolated, so that the transmission data from the router 14a of the infrastructure node 14K is not sent in the direction indicated by the arrow Y16, and all of the transmission data (100%) is assigned and sent to the endpoint setting unit 14k in the direction indicated by the arrow Y17.

[0068] Next, a process for recovering the abnormal Pod 15a of the disconnected worker node 15J will be described. In this case, the disconnected abnormal Pod 15a to be recovered is launched into a standby state by the master node 14J after the abnormality is resolved. After that, the abnormality recovery support unit 18 transmits a recovery command to the router 14a of the master node 14J to gradually increase the traffic to the launched Pod 15a to a predetermined traffic value and recover the Pod 15a.

[0069] In response to the restoration change command, the router 14a performs a process of gradually increasing the weight value of the endpoint setting unit 14j associated with the started-up Pod 15a to a predetermined traffic value, as indicated by an arrow Y22. This process sets the weight value to a predetermined value (e.g., 50%), and the restoration of the Pod 15a is completed.

[0070] Next, the operation of the first abnormality response process will be described with reference to the flowcharts shown in FIG. 9 and FIG. However, the endpoint setting units 14j and 14k and the Pods (one or multiple containers) 15a and 15b are deployed in a 1:1 configuration by the failure response deployment instruction unit 19. A prerequisite is that a weight value (%) indicating a predetermined traffic distribution ratio is set to, for example, 50% for each of the endpoint setting units 14j and 14k.

[0071] In step S1 shown in FIG. 10, it is assumed that the abnormality detection unit 17 shown in FIG. 9 detects an abnormality in the Pod 15a (container) of the worker node 15J.

[0072] When this abnormality is detected, in step S2, the abnormality recovery response unit 18 sends a change command (for example, "oc set route-backends Router name Pod name #1=100 Pod name #2=0 *Pod name #1: Pod that continues communication, Pod name #2: Pod that suppresses communication") to the router 14a of the master node 14J to set the traffic distribution ratio to 0%, as shown by arrow Y21.

[0073] In step S3, the router 14a that has received the change command changes the weight value of the endpoint setting unit 14j associated with the abnormal Pod 15a of the worker node 15J from 50% to 0%, as indicated by the arrow Y22. This causes the abnormal Pod 15a to be isolated (see the cross mark). The router 14a also changes the weight value of the other endpoint setting unit 14k associated with the Pod 15a of the worker node 15K from 50% to 100%, as indicated by the arrow Y23.

[0074] With this change, in step S4, all (100%) of the data transmitted from the router 14a of the infrastructure node 14K to the pods 15a of the worker nodes 15J and 15K is transmitted to the normal pods 15a via the endpoint setting unit 14k, as indicated by arrow Y17.

[0075] Thereafter, when the disconnected abnormal Pod 15a is restored, the disconnected abnormal Pod 15a is launched into a standby state after the abnormality is resolved by the master node 14J. Then, in step S5, the abnormality restoration support unit 18 transmits a restoration command to the router 14a of the master node 14J to restore the traffic to the launched Pod 15a by gradually increasing the traffic to a predetermined traffic value, for example, 10%, 30%, 50%.

[0076] In step S6, in response to the recovery command, the router 14a gradually increases the weight value of the endpoint setting unit 14j associated with the Pod 15a to be recovered of the worker node 15J, as shown by arrow Y22, to a predetermined traffic value of 10%, 30%, 50%, and so on, to recover.

[0077] <Second abnormality response process> 11 is a block diagram for explaining the second anomaly response processing of the virtualization system failure isolation device 10 of this embodiment. The anomaly detection that requires the second anomaly response processing is the sixth anomaly detection.

[0078] When the second anomaly response process is performed, the external endpoint setting unit 16 is associated so as to be shared by the Pods 15a and 15b of the worker nodes 15J and 15K. In addition to this configuration, the external endpoint setting unit 16 is configured to be associated with the external DBs 26a and 26b at the endpoint (end point) destinations on a 1:n basis. These association configurations are performed by the deployment instruction unit 19.

[0079] The external endpoint setting unit 16 receives data indicated by the arrow Y31 or Y32 from the Pods 15a, 15b of each worker node 15J, 15K, and distributes and transmits the data to a plurality of external DBs 26a, 26b as indicated by the arrows Y33, Y34. The external endpoint setting unit 16 is also set with a distribution ratio (%) for distributing traffic during transmission, and transmits data to the external DBs 26a, 26b with traffic according to the distribution ratio.

[0080] In the above-described association configuration, when the abnormality detection unit 17 detects an abnormality in an external DB (for example, the external DB 26a), the container management unit 14f deletes the endpoint of the abnormal external DB 26a from the external endpoint setting unit 16. This deletion suppresses communication of a Pod (for example, Pod 15a of the worker node 15J) that communicates with the deleted endpoint.

[0081] Furthermore, when the abnormality detection unit 17 detects an abnormality in the external DB 26a, the abnormality recovery response unit 18 recognizes which external endpoint setting unit 16 has the IP (Internet Protocol) address of the detected external DB 26a. The external DB 26a in which an abnormality is detected is referred to as an abnormal external DB 26a.

[0082] To recognize this, the abnormality recovery response unit 18 makes an inquiry to the container management unit 14f as indicated by the bidirectional arrow Y24, and obtains, from the container management unit 14f, information on the external endpoint setting unit 16 that has the IP address of the abnormal external DB 26a.

[0083] The abnormality recovery response unit 18 transmits to the router 14a of the master node 14J a command for setting to 0% the traffic allocation ratio to the external DB 26a of the IP address set in the acquired external endpoint setting unit 16. The router 14a receives the command and notifies the container management unit 14f.

[0084] The container management unit 14f changes the traffic allocation ratio to the abnormal external DB 26a set in the external endpoint setting unit 16 to 0%. As a result, the abnormal external DB 26a is separated (see the x mark).

[0085] <Hardware configuration> The virtualization system fault isolation device 10 according to the above-described embodiment is realized by a computer 100 having a configuration as shown in Fig. 12. The computer 100 has a CPU (Central Processing Unit) 101, a ROM (Read Only Memory) 102, a RAM (Random Access Memory) 103, a HDD (Hard Disk Drive) 104, an input / output I / F (Interface) 105, a communication I / F 106, and a media I / F 107.

[0086] The CPU 101 operates based on a program stored in the ROM 102 or the HDD 104, and controls each functional unit. The ROM 102 stores a boot program executed by the CPU 101 when the computer 100 is started up, programs related to the hardware of the computer 100, and the like.

[0087] The CPU 101 controls an output device 111 such as a printer or a display, and an input device 110 such as a mouse or a keyboard, via the input / output I / F 105. The CPU 101 obtains data from the input device 110 via the input / output I / F 105, or outputs generated data to the output device 111.

[0088] The HDD 104 stores programs executed by the CPU 101 and data used by the programs, etc. The communication I / F 106 receives data from other devices (not shown) via a communication network 112 and outputs the data to the CPU 101, and also transmits data generated by the CPU 101 to other devices via the communication network 112.

[0089] The media I / F 107 reads a program or data stored in the recording medium 113 and outputs it to the CPU 101 via the RAM 103. The CPU 101 loads a program related to a target process from the recording medium 113 onto the RAM 103 via the media I / F 107, and executes the loaded program. The recording medium 113 is an optical recording medium such as a DVD (Digital Versatile Disc) or a PD (Phase change rewritable Disc), a magneto-optical recording medium such as an MO (Magneto Optical disk), a magnetic recording medium, a conductive memory tape medium, a semiconductor memory, or the like.

[0090] For example, when the computer 100 functions as the virtualization system fault isolation device 10 according to the embodiment, the CPU 101 of the computer 100 executes a program loaded onto the RAM 103 to realize the functions of the virtualization system fault isolation device 10. Furthermore, the HDD 104 stores data in the RAM 103. The CPU 101 reads and executes a program relating to a target process from the recording medium 113. Additionally, the CPU 101 may read a program relating to a target process from another device via the communication network 112.

[0091] <Effects of the embodiment> The effects of the virtualization system fault isolation device 10 according to the embodiment of the present invention will be described.

[0092] (1a) The fault isolation device 10 includes a computational resource cluster 15 that is virtually created on a physical machine by container virtualization software and that clusters and arranges the virtually created containers, and a cluster management unit 14 that manages control related to the arrangement and operation of the virtually created and clustered containers.

[0093] The fault isolation device 10 also includes a deployment instruction unit 19 that performs a process of placing endpoint setting units 14j, 14k, which are associated with each of multiple containers and serve as the end points of communication data in which the traffic distribution ratio to each container is set, in association with the container, and an anomaly detection unit 17 that is created outside the virtually created computing resource cluster 15 and the cluster management unit 14 and detects anomalies in the container.

[0094] Furthermore, the fault isolation device 10 includes an abnormality recovery response unit 18 that is created externally and transmits to the cluster management unit 14 a change command for setting the allocation ratio for the abnormal container detected by the abnormality detection unit 17 to 0%. The cluster management unit 14 is configured to set the allocation ratio of the endpoint setting unit (e.g., the endpoint setting unit 14j) associated with the abnormal container to 0% in response to the change command.

[0095] According to this configuration, the traffic of communication of the abnormal container via the endpoint setting units 14j and 14k with a distribution ratio of 0% is 0. Therefore, the abnormal container can be separated from the normal container. Since the abnormality detection unit 17 and the abnormality recovery response unit 18 are not involved in the container virtualization software, recovery is possible faster than recovery by the container failure recovery function of the container virtualization software. To further explain the reason for this faster recovery, the above-mentioned failure recovery function can only set the cycle for monitoring failures to a predetermined cycle, but the present invention can detect a container failure and stop the abnormal container regardless of the monitoring cycle. Therefore, recovery is possible faster than recovery by the above-mentioned failure recovery function.

[0096] (2a) When recovering an abnormal container, the abnormality recovery response unit 18 transmits a recovery command to the cluster management unit 14 to gradually increase the traffic to the container to be recovered up to a predetermined traffic value. The cluster management unit 14 is configured to gradually increase the distribution ratio of the endpoint setting units 14j, 14k associated with the container to be recovered up to the predetermined traffic value in response to the recovery command.

[0097] According to this configuration, when recovering an abnormal container, the traffic distribution ratio of the endpoint setting units 14j and 14k associated with the abnormal container is gradually increased to a predetermined traffic value. This reduces the risk of a sudden increase in traffic during container recovery, which can cause a failure. In addition, the abnormal container is recovered by sending a recovery command from the abnormality recovery unit 18, which is not involved in the container virtualization software, so that recovery can be faster than recovery by a container failure recovery function of the container virtualization software.

[0098] (3a) The fault isolation device 10 includes a computational resource cluster 15 that is virtually created on a physical machine using container virtualization software and that clusters and arranges the virtually created containers, and a cluster management unit 14 that manages control related to the arrangement and operation of the virtually created and clustered containers.

[0099] The fault isolation device 10 also includes a plurality of external DBs 26a, 26b that are connected to the outside of the computational resource cluster 15 via a network and store data related to the containers, and an external endpoint setting unit 16 that corresponds to the plurality of containers of the computational resource cluster 15 and also corresponds to the plurality of external DBs 26a, 26b, and in which a traffic distribution ratio is set when distributing and transmitting data from the containers to the plurality of external DBs 26a, 26b.

[0100] The failure isolation device 10 also includes a deployment instruction unit 19 that performs processing for placing, in association with each container, endpoint setting units 14j, 14k that are associated with each of a plurality of containers and serve as the end points of communication data in which the distribution ratio of traffic to each container is set, and an anomaly detection unit 17 that is created outside the virtually created computing resource cluster 15 and the cluster management unit 14 and detects anomalies in the external DBs 26a, 26b.The failure isolation device 10 also includes an anomaly recovery response unit 18 that is created outside the above and transmits a change command to the cluster management unit 14 to change the distribution ratio to the anomaly DB 16a detected by the anomaly detection unit 17 to 0%.

[0101] When the abnormality detection unit 17 detects an abnormality in an external DB (for example, the external DB 26a), the abnormality recovery response unit 18 acquires information on the external endpoint setting unit 16 having the IP address of the detected abnormal external DB 26a from the cluster management unit 14, and transmits a command to the cluster management unit 14 to change the distribution ratio set in the external endpoint setting unit 16 of the acquired information to 0%. The cluster management unit 14 is configured to change the traffic distribution ratio to the abnormal external DB 26a to 0% in response to the command.

[0102] According to this configuration, the amount of communication traffic to the abnormal external DB 26a outside the computational resource cluster 15 via the external endpoint setting unit 16 with a distribution ratio of 0% is 0. Therefore, the abnormal external DBs 26a and 26b outside the computational resource cluster 15 can be separated.

[0103] <Effects> (1) A virtualization system fault isolation device comprising: a computing resource cluster that is virtually created on a physical machine by container virtualization software and that clusters and arranges the virtually created containers; a cluster management unit that manages control related to the arrangement and operation of the virtually created and clustered containers; a deployment instruction unit that performs a process of placing, in association with a container, an endpoint setting unit that is associated with each of a plurality of containers and serves as an end point of communication data in which an allocation ratio of traffic to each container is set; an anomaly detection unit that is created outside the virtually created computing resource cluster and the cluster management unit and that detects anomalies in the containers; and an anomaly recovery response unit that is created outside the virtually created computing resource cluster and the cluster management unit and that sends a change command to the cluster management unit to set the allocation ratio for an abnormal container detected by the anomaly detection unit to 0%, wherein the cluster management unit sets the allocation ratio of the endpoint setting unit associated with the abnormal container to 0% in response to the change command.

[0104] According to this configuration, the communication traffic of the abnormal container via the endpoint setting unit with a distribution ratio of 0% is 0. Therefore, the abnormal container can be separated from the normal container. Since the abnormality detection unit and the abnormality recovery response unit are not involved in the container virtualization software, recovery is possible faster than recovery by the container failure recovery function of the container virtualization software. To further explain the reason for this faster recovery, the above-mentioned failure recovery function can only set the cycle for monitoring failures to a predetermined cycle, but the present invention can detect a container failure and stop the abnormal container regardless of the monitoring cycle. Therefore, recovery is possible faster than recovery by the above-mentioned failure recovery function.

[0105] (2) The virtualization system fault isolation device described in (1) above is characterized in that, when recovering the abnormal container, the abnormality recovery response unit sends a recovery command to the cluster management unit to gradually increase traffic to the container to be recovered to a predetermined traffic value, and the cluster management unit, in response to the recovery command, gradually increases the allocation ratio of the endpoint setting unit corresponding to the container to be recovered to the predetermined traffic value.

[0106] According to this configuration, when recovering an abnormal container, the traffic distribution ratio of the endpoint setting unit associated with the abnormal container is gradually increased to a predetermined traffic value. This reduces the risk of a sudden increase in traffic during container recovery, which can cause a failure. In addition, since the abnormal container is recovered by sending a recovery command from an abnormality recovery unit that is not involved in the container virtualization software, recovery can be faster than recovery by a container failure recovery function of the container virtualization software.

[0107] (3) A computing resource cluster that is virtually created on a physical machine by container virtualization software and that clusters and arranges the virtually created containers; a cluster management unit that manages control related to the arrangement and operation of the virtually created and clustered containers; a plurality of DBs (Data Bases) that are connected to the outside of the computing resource cluster via a network and that store data related to the containers; an external endpoint setting unit that is associated with the plurality of containers of the computing resource cluster and the plurality of DBs and that sets a traffic distribution ratio when data from the containers is distributed and transmitted to the plurality of DBs; an anomaly detection unit that is created outside the virtually created computing resource cluster and the cluster management unit and that detects an anomaly in the DB; and an abnormality recovery response unit that is created outside the virtually created computing resource cluster and the cluster management unit and that transmits a change command to the cluster management unit to change the distribution ratio to the abnormal DB detected by the abnormality detection unit to 0%, wherein the abnormality recovery response unit, when an abnormality in the DB is detected by the abnormality detection unit, and acquiring information about an external endpoint setting unit having a (NAT) (Network Address Translation) (NAT) address from the cluster management unit, transmitting a command to the cluster management unit to set the distribution ratio set in the external endpoint setting unit of the acquired information to 0%, and the cluster management unit changing the traffic distribution ratio to the abnormal DB to 0% in response to the command.

[0108] According to this configuration, the amount of communication traffic to the abnormal DB outside the computational resource cluster via the external endpoint setting unit with a distribution ratio of 0% is 0. Therefore, the abnormal DB outside the computational resource cluster can be isolated.

[0109] In addition, the specific configuration can be modified as appropriate without departing from the spirit of the present invention. [Explanation of symbols]

[0110] 10 Virtualization System Fault Isolation Device 14 Cluster Management Unit 14a Communication distribution section 14b Computing resource operation section 14c Computing Resource Management Department 14d Container configuration reception section 14e Container placement decision unit 14th floor Container Management Department 14j, 14k Endpoint setting section 15 Computing Resource Cluster 15a,15b Application 16 External endpoint setting section 17 Anomaly detection section 18. Abnormality Recovery Department 19 Deployment Instructions for Failure Response (Deployment Instructions) 26a,26b External DB (DB)

Claims

1. a computing resource cluster that is virtually created on a physical machine by container virtualization software and that clusters and arranges the virtually created containers; a cluster management unit that manages control related to the placement and operation of the virtually created and clustered containers; a deployment instruction unit that performs a process of placing an endpoint setting unit, which is associated with each of a plurality of containers and serves as an end point of communication data in which a traffic distribution ratio to each container is set, in association with the container; an anomaly detection unit that is created outside the virtually created computing resource cluster and the cluster management unit and detects an anomaly in the container; an abnormality recovery response unit that is created externally and transmits to the cluster management unit a change command for changing the distribution ratio of the abnormal container detected by the abnormality detection unit to 0%; Equipped with The cluster management unit sets the allocation ratio of the endpoint setting unit associated with the abnormal container to 0% in response to the change command, the abnormality recovery response unit transmits a recovery command to the cluster management unit when recovering the abnormal container, for gradually increasing traffic to the container to be recovered up to a predetermined traffic value; The cluster management unit gradually increases an allocation ratio of an endpoint setting unit associated with a container to be restored up to the predetermined traffic value in response to the restoration command. A virtualization system fault isolation device comprising:

2. a computing resource cluster that is virtually created on a physical machine by container virtualization software and that clusters and arranges the virtually created containers; a cluster management unit that manages control related to the placement and operation of the virtually created and clustered containers; a plurality of DBs (Data Bases) connected to an external portion of the computing resource cluster via a network and configured to store data related to the container; an external endpoint setting unit that is associated with a plurality of containers of the computational resource cluster and is associated with the plurality of DBs, and in which a traffic distribution ratio is set when data from the container is distributed to the plurality of DBs and transmitted; an anomaly detection unit that is created outside the virtually created computing resource cluster and the cluster management unit and detects an anomaly in the DB; an abnormality recovery response unit that is created externally and transmits to the cluster management unit a change command for changing the distribution ratio to the abnormal DB detected by the abnormality detection unit to 0%; Equipped with when the anomaly detection unit detects an anomaly in the DB, the anomaly recovery response unit acquires from the cluster management unit information on an external endpoint setting unit having an IP (Internet Protocol) address of the detected anomaly DB, and transmits a command to the cluster management unit to set an allocation ratio set in the external endpoint setting unit of the acquired information to 0%, The cluster management unit changes the traffic allocation ratio to the abnormal DB to 0% in response to the command. A virtualization system fault isolation device comprising:

3. A method for isolating a virtualization system fault by a virtualization system fault isolation device, comprising: The virtualization system fault isolation device comprises: A step of clustering and arranging the virtually created containers on a computing resource cluster that is virtually created on a physical machine by container virtualization software; creating a cluster management unit that manages control related to placement and operation of the clustered containers; an endpoint setting unit that is associated with each of a plurality of containers and serves as an end point of communication data in which a traffic distribution ratio to each container is set, the endpoint setting unit being associated with the container; detecting an anomaly in the container outside the virtually created computing resource cluster and a cluster manager; a step of transmitting a change command to the cluster management unit, the change command being generated externally and for changing the allocation ratio of the detected abnormal container to 0%; the cluster management unit sets an allocation ratio of the endpoint setting unit associated with the abnormal container to 0% in response to the change command; sending a recovery command to the cluster management unit to gradually increase traffic to the container to be restored up to a predetermined traffic value when the abnormal container is restored; gradually increasing the distribution ratio of the endpoint setting unit associated with the container to be restored to the predetermined traffic value in response to the restoration command; A method for isolating a virtualization system fault, comprising:

4. A method for isolating a virtualization system fault by a virtualization system fault isolation device, comprising: The virtualization system fault isolation device comprises: a computing resource cluster that is virtually created on a physical machine by container virtualization software and that clusters and arranges the virtually created containers; a cluster management unit that manages control related to the placement and operation of the virtually created and clustered containers; a plurality of DBs (Data Bases) connected to an external portion of the computing resource cluster via a network and configured to store data related to the container; an external endpoint setting unit that is associated with a plurality of containers of the computational resource cluster and is associated with the plurality of DBs, and in which a traffic distribution ratio is set when data from the container is distributed to the plurality of DBs and transmitted; an anomaly detection unit that is created outside the virtually created computing resource cluster and the cluster management unit and detects an anomaly in the DB; an abnormality recovery response unit that is created externally and transmits to the cluster management unit a change command for changing the distribution ratio to the abnormal DB detected by the abnormality detection unit to 0%; Equipped with the anomaly recovery response unit, when an anomaly in the DB is detected by the anomaly detection unit, acquires information of an external endpoint setting unit having an IP address of the detected anomaly DB from the cluster management unit, and transmits a command to the cluster management unit to set the distribution ratio set in the external endpoint setting unit of the acquired information to 0%; the cluster management unit changes the traffic allocation ratio to the abnormal DB to 0% in response to the command; A method for isolating a virtualization system fault, comprising:

Images