Method and system for automatic recovery of virtual machine based on SAN storage and prevention of split-brain

Abstract

The application discloses a method and system for automatically recovering virtual machine and preventing brain split based on SAN storage, wherein the method comprises the following steps: creating a shared storage volume on a SAN storage device, and simultaneously mapping the shared storage volume to a source host and a target host; deploying and running an HA agent on the source host and the target host, and using the HA agent to perform heartbeat detection of a storage surface; when detecting that the heartbeat of the source host is abnormal, mounting the disk of a virtual machine on the source host to the target host; detecting whether the source host has failed through the storage surface on the target host; if the source host has not failed, terminating the reconstruction of the virtual machine; and if the source host has failed, performing the reconstruction of the virtual machine. The application detects and judges whether the source host has failed through the storage surface, and effectively avoids the loss of virtual machine service caused by the failure of a management network.

Description

Technical Field

[0001] This invention relates to the field of shared storage technology, and in particular to a method and system for automatic virtual machine recovery and split-brain prevention based on SAN storage. Background Technology

[0002] OpenStack is an open-source project designed to provide software for building and managing public and private clouds. When a physical host fails, it will affect the services of virtual machines running on the host. In this case, a mechanism is needed to detect and recover the virtual machines running on the host when the physical host fails. The OpenStack community provides a virtual machine HA (HA is an abbreviation for Highly Available, which is short for dual-machine cluster system) solution.

[0003] In the OpenStack community solution, if an abnormal heartbeat is detected on the source host, it is considered to have failed. The fault isolation process involves directly powering down the source host. After powering down, a new, healthy host is selected, and the virtual machines are rebuilt and services restored on the new host. However, this solution simply and crudely powers down the supposedly faulty host during the recovery process. In some scenarios, only the management network is faulty, and the virtual machine's services are not affected. Furthermore, there is no mechanism to prevent split-brain scenarios during the virtual machine recovery and rebuilding process.

[0004] In the community solution, the network plane used for heartbeat detection is the cloud platform's management network. An abnormal heartbeat in the management network is interpreted as a host failure, thus causing damage to virtual machine workloads and services. This is unreasonable. If only the management network fails, while the physical host, service network, and storage network are all functioning normally, the virtual machine can run normally, and the services on the virtual machine are unaffected. The community solution's process of directly shutting down the host and rebuilding the virtual machine actually causes damage to the virtual machine's services. Summary of the Invention

[0005] The purpose of this invention is to provide a method and system for automatic recovery of virtual machines based on SAN storage to prevent split-brain, aiming to solve the problem of virtual machine service losses caused by management network failures being mistakenly identified as host failures.

[0006] To solve the above-mentioned technical problems, the present invention is achieved through the following technical solution: providing a method for automatic recovery of virtual machines based on SAN storage to prevent split-brain, comprising:

[0007] Create a shared storage volume on the SAN storage device and map the shared storage volume to both the source host and the target host.

[0008] Deploy and run HA agents on the source host and the target host for storage plane heartbeat detection;

[0009] When an abnormal heartbeat is detected on the source host, the disk of the virtual machine on the source host is mounted to the target host;

[0010] On the target host, the source host is checked for faults via the storage plane.

[0011] If the source host is not faulty, the virtual machine rebuild will be terminated.

[0012] If the source host fails, the virtual machine will be rebuilt.

[0013] Furthermore, the step of detecting whether the source host has failed via the storage plane on the target host includes:

[0014] On the target host, a KEY is registered on its own shared storage volume via SCSI commands based on the HA agent;

[0015] The target host detects whether there is a key written by another host on its own shared storage volume. If a key written by another host is detected, the source host is determined to be not faulty; if no key written by another host is detected, the source host is determined to be faulty.

[0016] Furthermore, the step of determining that the source host is faulty if no key written by another host is detected includes:

[0017] The target host checks at a first predetermined time interval whether any other host has written a KEY;

[0018] If no key is detected written by other hosts, return to continue waiting for the first predetermined time, and continue to check if any key has been written by other hosts after the time has elapsed;

[0019] If no key written by another host is detected for a predetermined number of consecutive attempts, the source host is deemed to be faulty.

[0020] Furthermore, the method for automatic recovery of virtual machines based on SAN storage to prevent split-brain also includes:

[0021] The HA agent running on the source host checks every second predetermined time whether the shared storage volume on the local host has been written to a key on another host;

[0022] If so, the HA running on the source host writes a KEY to the shared storage volume via SCSI commands and enters a waiting state.

[0023] Furthermore, the step of terminating the virtual machine reconstruction if the source host does not fail includes:

[0024] Unmount the shared storage volume from the target host to intercept the virtual machine auto-recovery process.

[0025] Furthermore, the step of terminating the virtual machine reconstruction if the source host does not fail also includes:

[0026] After the source host waits for a third predetermined time, the KEY written to the shared storage volume by this host will be cleared.

[0027] Furthermore, if the source host fails, the virtual machine is rebuilt, including:

[0028] Automatic recovery and reconstruction of the virtual machine is performed on the target host, and the KEY written to the shared storage volume on the target host is cleared.

[0029] This invention also provides a system for automatic virtual machine recovery and split-brain prevention based on SAN storage, comprising: a SAN storage device, a source host, and a target host;

[0030] The SAN storage device is used to create a shared storage volume, which is simultaneously mapped to both the source host and the target host.

[0031] The source host and the target host are used to deploy and run the HA agent, which is used to perform heartbeat detection on the storage plane, and when an abnormal heartbeat is detected on the source host, the disk of the virtual machine on the source host is mounted to the target host.

[0032] The target host is also used to detect whether the source host has failed through the storage plane. If the source host has not failed, the virtual machine reconstruction is terminated; if the source host has failed, the virtual machine is reconstructed.

[0033] Furthermore, the target host is also used to register a KEY on its own shared storage volume via SCSI commands and based on the HA agent; and to detect whether there is a KEY written by another host on its own shared storage volume. If a KEY written by another host is detected, the source host is determined to be not faulty; if no KEY written by another host is detected, the source host is determined to be faulty.

[0034] Furthermore, the source host is used to detect at a second predetermined time interval whether a KEY of another host has been written to the shared storage volume on the source host; if so, the HA running on the source host writes a KEY to the shared storage volume via SCSI commands and enters a waiting state.

[0035] This invention provides a method and system for automatic virtual machine recovery and split-brain prevention based on SAN storage. The method includes: creating a shared storage volume on a SAN storage device and mapping the shared storage volume to both a source host and a target host; deploying and running an HA agent on both the source and target hosts to perform storage plane heartbeat detection; when an abnormal heartbeat is detected on the source host, mounting the virtual machine's disk from the source host to the target host; detecting whether the source host has failed on the target host through storage plane detection; if the source host has not failed, terminating the virtual machine rebuild; if the source host has failed, rebuilding the virtual machine. This invention effectively avoids losses to virtual machine services due to management network failures by determining whether the source host has failed through storage plane detection. Attached Figure Description

[0036] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0037] Figure 1 A flowchart illustrating a method for automatic virtual machine recovery to prevent split-brain based on SAN storage, provided as an embodiment of the present invention;

[0038] Figure 2 A schematic diagram of the workflow of the source host provided in an embodiment of the present invention;

[0039] Figure 3 A schematic diagram of the workflow of the target host provided in an embodiment of the present invention;

[0040] Figure 4 This is a schematic diagram of a system for automatic virtual machine recovery and anti-split-brain mechanism based on SAN storage, provided as an embodiment of the present invention. Detailed Implementation

[0041] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0042] It should be understood that, when used in this specification and the appended claims, the terms "comprising" and "including" indicate the presence of the described features, integrals, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or collections thereof.

[0043] It should also be understood that the terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the invention. As used in this specification and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise.

[0044] It should also be further understood that the term "and / or" as used in this specification and the appended claims refers to any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0045] Please see Figure 1 , Figure 1 This invention provides a flowchart illustrating a method for automatic virtual machine recovery and split-brain prevention based on SAN storage, which may include steps S101 to S106:

[0046] S101. Create a shared storage volume on the SAN storage device and map the shared storage volume to both the source host and the target host.

[0047] In the OpenStack community solution, the network plane used for heartbeat detection is the cloud platform's management network. An abnormal heartbeat in the management network is considered a host failure, which leads to the virtual machine's load and service being damaged. This embodiment of the invention provides a device based on SAN storage. A shared storage volume is created on the SAN storage device, which can enable communication between the source host and the target host, providing a detection platform for step S104.

[0048] S102. Deploy and run the HA agent on the source and target hosts to perform heartbeat detection on the storage plane.

[0049] In OpenStack, High Availability (HA) is a common and effective way to ensure business continuity. HA agents are deployed and run on both the source and target hosts. When an anomaly occurs in the ongoing business on the source host, the HA agent sends an anomaly signal to the target host through the SAN storage device, and the target host immediately detects the abnormal heartbeat of the source host.

[0050] S103. When an abnormal heartbeat is detected on the source host, mount the disk of the virtual machine on the source host to the target host.

[0051] Before actually rebuilding the virtual machine, mount the virtual machine's disk to the target host. This ensures the continuity of virtual machine services regardless of whether the source host fails.

[0052] S104. On the target host, detect whether the source host has failed through the storage plane. If yes, proceed to step S105; otherwise, proceed to step S106.

[0053] Before the virtual machine is rebuilt after the disk is mounted, commands are executed by the HA agent deployed on the target host to detect whether the host is faulty through the storage plane, thus preventing the storage plane split-brain problem.

[0054] In one embodiment, such as Figure 3 As shown, step S104 includes steps S301 to S304:

[0055] S301. The target host registers a KEY on its own shared storage volume via SCSI commands and based on the HA agent. In a more specific embodiment, the value of the KEY registered by the target host is the hostname of the target host, which can be distinguished from the KEY written to the shared storage volume by the source host to avoid misjudgment in subsequent detection results.

[0056] S302. The target host checks at a first predetermined time interval whether there is a key written by another host on its own shared storage volume. If a key written by another host is detected, proceed to step S303. If no key written by another host is detected, proceed to step S304.

[0057] In a specific embodiment, the first predetermined time is set to 10 seconds to avoid the source host from mistakenly judging the source host as faulty because the time is too short and the source host has already cleared the KEY in the shared storage volume.

[0058] S303: The source host is determined to be not faulty;

[0059] If the target host detects a key written by another host, that is, a key written by the source host, it proves that the source host is still using the shared storage volume created on the SAN device normally, and the source host has not failed.

[0060] S304, Determine that the source host is faulty;

[0061] If the target host does not detect any keys written by other hosts, it indicates that the source host is not using the shared storage volume created on the SAN device properly, and the source host has failed.

[0062] In one embodiment, step S304 specifically includes:

[0063] The target host checks at a first predetermined time interval whether any other host has written a KEY;

[0064] If no key is detected written by other hosts, return to continue waiting for the first predetermined time, and continue to check if any key has been written by other hosts after the time has elapsed;

[0065] If no key written by another host is detected for a predetermined number of consecutive attempts, the source host is deemed to be faulty.

[0066] In the above embodiment, the target host will continuously check for keys written by other hosts multiple times. If no keys written by other hosts are detected in multiple consecutive checks, the source host is determined to be faulty. Conversely, if a key written by another host is found in any check, it proves that the source host is still using the shared storage volume normally. This method improves the fault tolerance of the detection, avoids inaccurate results from a single check, and prevents detection anomalies.

[0067] For example, the target host can check every 10 seconds (a total of 3 checks, i.e., the predetermined number of checks) to see if any keys have been written by other hosts. If two keys are found in one check, and at least one of the keys is not the hostname of this host, it proves that the source host is still using the shared storage volume normally. If keys written by other hosts are detected in 3 consecutive checks, the source host is determined to be faulty.

[0068] Furthermore, the method for automatic recovery of virtual machines based on SAN storage to prevent split-brain also includes steps S201 to S203:

[0069] S201. The HA agent running on the source host checks every second predetermined time whether the shared storage volume on the local host has been written to a key of another host.

[0070] Specifically, the second predetermined time can be designed independently. The value of the second predetermined time is mutually restricted with the first predetermined time and the number of predetermined times. In a more specific embodiment, the second predetermined time is set to 10 seconds. The value of 10 seconds avoids the situation where the KEY written by the target host cannot be discovered due to the excessively long interval.

[0071] S202. If so, the HA running on the source host writes a KEY to the shared storage volume via SCSI commands and enters a waiting state.

[0072] In one embodiment, the KEY written by the source host is the hostname of the source host, which is used to distinguish it from the KEY written by other hosts; in the waiting state, the HA agent stops the command to detect whether the shared storage volume on the local host has been written to a KEY of another host, so as to avoid duplicate KEY sending and misjudgment of the target host.

[0073] S203. After the source host has waited for the third predetermined time, the KEY written to the shared storage volume by this host is cleared.

[0074] Specifically, the value of the third predetermined time must ensure that the target host has already detected the KEY written by the source host. This prevents the KEY written to the shared storage volume from being deleted before it is detected by the source host, which could lead to a misjudgment of the source host as faulty and thus avoid loss of virtual machine services. Specifically, the value of the first predetermined time is related to the second and third predetermined times mentioned above. It must be ensured that the sum of the first and second predetermined times is less than the third predetermined time to prevent the target host from being unable to detect the KEY written from the source host, thus avoiding a misjudgment of the source host as faulty. In a more specific embodiment, the third predetermined time is set to 30 seconds. This value of 30 seconds not only prevents the target host from being unable to detect the KEY written by the source host due to insufficient time, but also effectively reduces the loss of virtual machine services caused by fault detection time.

[0075] Please continue to refer to Figure 3 , Figure 3 The schematic diagram of the target host's workflow provided in the embodiments of the present invention also includes S305 to S307:

[0076] S305, Terminate the rebuilding of the virtual machine;

[0077] If the target host detects that the source host is not faulty, it will unload the shared storage volume from the target host. The virtual machine automatic recovery process is intercepted, reducing the losses caused by the transfer of virtual machine services due to management network failure and avoiding the occurrence of split-brain.

[0078] S306. Rebuild the virtual machine;

[0079] The target host detected a failure in the source host and quickly rebuilt the virtual machine, ensuring the continuity of virtual machine services and avoiding losses caused by service interruption due to source host failure.

[0080] S307. Clean up the key written to the shared storage volume by the target host.

[0081] After the target host detects and determines the source host, the key written to the shared storage volume by the target host is cleared to avoid affecting the normal operation of the source host and the next heartbeat anomaly detection of the source host.

[0082] S105, Terminate the rebuilding of the virtual machine;

[0083] If the target host detects that the source host is not faulty, it will unload the shared storage volume from the target host. The virtual machine automatic recovery process is intercepted, reducing the losses caused by the transfer of virtual machine services due to management network failure and avoiding the occurrence of split-brain.

[0084] S106. Rebuild the virtual machine.

[0085] The target host detected a failure in the source host and quickly rebuilt the virtual machine, ensuring the continuity of virtual machine services and avoiding losses caused by service interruption due to source host failure.

[0086] Please see Figure 4 , Figure 4 This is a schematic diagram of a system for automatic virtual machine recovery and anti-split-brain mechanism based on SAN storage, provided for an embodiment of the present invention. The system includes a SAN storage device 401, a source host 402, and a target host 403.

[0087] The SAN storage device 401 is used to create a shared storage volume and simultaneously map the shared storage volume to both the source host and the target host, enabling communication between the source host and the target host and providing a detection platform for the target host to detect whether the source host is faulty.

[0088] The source host 402 is used to detect at a second predetermined time interval whether a KEY of another host has been written to the shared storage volume on the source host; if so, the HA running on the source host writes a KEY to the shared storage volume through SCSI commands and enters a waiting state.

[0089] The target host 403 is used to register a key on its own shared storage volume via SCSI commands and based on the HA agent; and to detect whether there are keys written by other hosts on its own shared storage volume. If a key written by another host is detected, the source host is determined to be functioning correctly; if no key written by another host is detected, the source host is determined to be faulty. Compared to the original handling solution in the OpenStack community, detecting whether the source host is faulty effectively avoids the loss of virtual machine services due to management network failures.

[0090] In one embodiment, the target host 403 is further configured to register a KEY on its own shared storage volume via SCSI commands and based on the HA agent; and to detect whether there is a KEY written by another host on its own shared storage volume. If a KEY written by another host is detected, the source host is determined to be not faulty; if no KEY written by another host is detected, the source host is determined to be faulty.

[0091] In one embodiment, the source host 402 is configured to detect at second predetermined intervals whether a key of another host has been written to the shared storage volume on the source host; if so, the HA running on the source host writes a key to the shared storage volume via SCSI commands and enters a waiting state.

[0092] The specific technical details of the above-mentioned system embodiment for automatic virtual machine recovery and anti-split-brain system based on SAN storage have been described in detail in the previous method embodiments, and therefore will not be repeated here.

[0093] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and these modifications or substitutions should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for automatic virtual machine recovery and split-brain prevention based on SAN storage, characterized in that, include: Create a shared storage volume on the SAN storage device and map the shared storage volume to both the source host and the target host. Deploy and run HA agents on the source host and the target host for storage plane heartbeat detection; When an abnormal heartbeat is detected on the source host, the disk of the virtual machine on the source host is mounted to the target host; On the target host, the source host is checked for faults via the storage plane. If the source host is not faulty, the virtual machine rebuild will be terminated. If the source host fails, the virtual machine will be rebuilt. The HA agent running on the source host checks every second predetermined time whether a key of another host has been written to the shared storage volume on the local host; if so, the HA agent running on the source host writes a key to the shared storage volume via SCSI commands and enters a waiting state. In the waiting state, the HA agent stops checking whether a key of another host has been written to the shared storage volume on the local host.

2. The method for automatic virtual machine recovery and brain-prevention based on SAN storage according to claim 1, characterized in that, The step of detecting whether the source host has failed on the target host through the storage plane includes: registering a KEY on the target host's own shared storage volume via SCSI commands based on the HA agent; the target host detecting whether there is a KEY written by another host on its own shared storage volume; if a KEY written by another host is detected, it is determined that the source host has not failed; if no KEY written by another host is detected, it is determined that the source host has failed.

3. The method for automatic virtual machine recovery and brain-split prevention based on SAN storage according to claim 2, characterized in that, The step of determining that the source host is faulty if no key written by another host is detected includes: The target host checks at a first predetermined time interval whether any other host has written a KEY; If no key is detected written by other hosts, return to continue waiting for the first predetermined time, and continue to check if any key has been written by other hosts after the time has elapsed; If no key written by another host is detected for a predetermined number of consecutive attempts, the source host is deemed to be faulty.

4. The method for automatic virtual machine recovery and brain-split prevention based on SAN storage according to claim 1, characterized in that, If the source host does not fail, then terminating the virtual machine reconstruction includes: Unmount the shared storage volume from the target host to intercept the virtual machine auto-recovery process.

5. The method for automatic virtual machine recovery and brain-split prevention based on SAN storage according to claim 1, characterized in that, The clause stating that if the source host does not fail, the virtual machine reconstruction will be terminated also includes: After the source host waits for a third predetermined time, the KEY written to the shared storage volume by this host will be cleared.

6. The method for automatic virtual machine recovery and brain-prevention based on SAN storage according to claim 1, characterized in that, If the source host fails, the virtual machine will be rebuilt, including: Automatic recovery and reconstruction of the virtual machine is performed on the target host, and the KEY written to the shared storage volume on the target host is cleared.

7. A system for automatic virtual machine recovery and split-brain prevention based on SAN storage, characterized in that, include: Source host, target host, and SAN storage device; The SAN storage device is used to create a shared storage volume, which is simultaneously mapped to both the source host and the target host. The source host and the target host are used to deploy and run the HA agent, which is used to perform heartbeat detection on the storage plane, and when an abnormal heartbeat is detected on the source host, the disk of the virtual machine on the source host is mounted to the target host. The target host is also used to detect whether the source host has failed through the storage plane. If the source host has not failed, the reconstruction of the virtual machine is terminated. If the source host fails, the virtual machine will be rebuilt. The source host is used to detect at a second predetermined time interval whether a key of another host has been written to the shared storage volume on the source host; if so, the HA running on the source host writes a key to the shared storage volume via SCSI command and enters a waiting state. In the waiting state, the HA agent stops detecting whether a key of another host has been written to the shared storage volume on the source host.

8. The SAN-based virtual machine automatic recovery anti-split-brain system according to claim 7, characterized in that, The target host is also used to register a key on its own shared storage volume via SCSI commands and based on the HA agent; and to detect whether there is a key written by another host on its own shared storage volume. If a key written by another host is detected, it is determined that the source host is not faulty. If no key is detected written by other hosts, the source host is deemed to be faulty.

Images