A method, apparatus, device, and medium for detecting storage link failures

Abstract

The present application discloses a method, apparatus, device and medium for detecting storage link failures, including: creating a monitoring virtual machine on each host node; mounting volumes corresponding to different storage backends to the corresponding monitoring virtual machines; obtaining health status detection data, where the health status detection data is data obtained by the monitoring virtual machine performing a health status detection on its own disk; and determining the data link status between the monitoring virtual machine and the storage backend based on the health status detection data. It can timely detect data link failures between virtual machines and backend storage, and has universality, thereby improving the user experience.

Description

Technical Field

[0001] This application relates to the field of storage technology, and particularly relates to a method, apparatus, device, and medium for detecting storage link failures. Background Art

[0002] With the rapid development and maturity of cloud computing technology, open-source cloud computing platforms represented by OpenStack have been widely used. Cloud computing is inseparable from storage, and different types of storage can be used in cloud platforms, such as virtualized storage, distributed storage, centralized storage, etc. When a failure occurs in the storage data link, disk read / write failures will occur inside the virtual machine, resulting in abnormal operation or even crashing of the operating system inside the virtual machine, and normal services cannot be provided to users.

[0003] Currently, the detection schemes for storage backend failures in cloud platforms, such as etcd, pacemaker, consul, etc., are all based on the detection of the connectivity between the host and the storage network interface of the IP network. For the actual business data link of the storage, that is, the data link between the virtual machine and the backend storage, it is not directly detected, and these methods have limitations and can only support the detection of specific types of storage backends. There is no generally applicable unified data link failure detection scheme for different types of storage backends. Summary of the Invention

[0004] In view of this, the purpose of this application is to provide a method, apparatus, device, and medium for detecting storage link failures, which can timely detect the data link failures between the virtual machine and the backend storage and have universality, thereby improving the user experience. The specific scheme is as follows:

[0005] In a first aspect, this application discloses a method for detecting storage link failures, including:

[0006] Create a monitoring virtual machine on each host node;

[0007] Mount the volumes corresponding to different storage backends to the corresponding monitoring virtual machines;

[0008] Obtain health status detection data; wherein, the health status detection data is the data obtained by the monitoring virtual machine performing a health status detection on its own disk;

[0009] Determine the data link status between the monitoring virtual machine and the storage backend based on the health status detection data.

[0010] Optionally, the creating a monitoring virtual machine on each host node includes:

[0011] Based on the monitoring server, and through the Nova service, create a monitoring virtual machine on each host node.

[0012] Optionally, mounting the volumes corresponding to different storage backends to the corresponding monitoring virtual machines includes:

[0013] Based on the monitoring server and through the Cinder service, mounting the volumes corresponding to different storage backends to the corresponding monitoring virtual machines.

[0014] Optionally, obtaining the health status detection data includes:

[0015] Obtaining the health status detection data reported by the host agent in the host node;

[0016] Wherein, the health status detection data is the data obtained by the monitoring virtual machine calling its own virtual machine agent under the trigger of the host agent to perform a health status detection on the disk of the monitoring virtual machine itself.

[0017] Optionally, the host agent periodically triggers the monitoring virtual machine to call its own virtual machine agent through QGA to perform a health status detection on the disk of the monitoring virtual machine itself.

[0018] Optionally, it further includes:

[0019] Pre - placing the virtual machine agent and the QGA in the operating system image of the monitoring virtual machine.

[0020] Optionally, after determining the data link status between the monitoring virtual machine and the storage backend based on the health status detection data, it further includes:

[0021] If the data link fails, reporting the corresponding fault information.

[0022] In a second aspect, the present application discloses a storage link fault detection device, including:

[0023] A monitoring virtual machine creation module, configured to create a monitoring virtual machine on each host node;

[0024] A backend storage volume mounting module, configured to mount the volumes corresponding to different storage backends to the corresponding monitoring virtual machines;

[0025] A health detection data acquisition module, configured to acquire health status detection data; wherein, the health status detection data is the data obtained by the monitoring virtual machine performing a health status detection on its own disk;

[0026] A data link status determination module, configured to determine the data link status between the monitoring virtual machine and the storage backend based on the health status detection data.

[0027] In a third aspect, the present application discloses an electronic device, including:

[0028] a memory for storing a computer program;

[0029] a processor for executing the computer program to implement the foregoing storage link fault detection method.

[0030] In a fourth aspect, the present application discloses a computer-readable storage medium for storing a computer program, and when the computer program is executed by a processor, the foregoing storage link fault detection method is implemented.

[0031] It can be seen that in the present application, a monitoring virtual machine is first created on each host node, and then the volumes corresponding to different storage backends are mounted to the corresponding monitoring virtual machines to obtain health status detection data; wherein, the health status detection data is the data obtained by the monitoring virtual machine performing a health status detection on its own disk, and then the data link status between the monitoring virtual machine and the storage backend is determined based on the health status detection data. That is to say, in the embodiments of the present application, a monitoring virtual machine can be created on a host node, and then the volumes corresponding to different storage backends are mounted to the corresponding monitoring virtual machines. In this way, without intruding into the user business virtual machine, by simulating the running environment of the user business with the monitoring virtual machine, the data link fault between the virtual machine and the backend storage can be detected in a timely manner, and it has universality, thereby improving the user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, the drawings in the following description are only the embodiments of the present application. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained according to the provided drawings.

[0033] Figure 1 It is a flowchart of a storage link fault detection method provided by the present application;

[0034] Figure 2 It is a schematic diagram of a storage link fault detection solution provided by the present application;

[0035] Figure 3 It is a schematic structural diagram of a storage link fault detection device provided by the present application;

[0036] Figure 4 It is a structural diagram of an electronic device provided by the present application. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0037] Next, the technical solutions in the embodiments of the present application will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments in the present application without creative efforts belong to the scope of protection of the present application.

[0038] Currently, the detection solutions for cloud platform storage backend failures, such as etcd, pacemaker, consul, etc., are all based on the detection of the connectivity between the host and the storage network interface of the IP network. For the actual business data link of the storage, that is, the data link between the virtual machine and the backend storage, there is no direct detection, and these methods have limitations and can only support the detection of specific types of storage backends. There is no generally applicable unified data link failure detection solution for different types of storage backends. For this reason, the present application provides a storage link failure detection solution, which can timely detect the data link failure between the virtual machine and the backend storage and has universality, thereby improving the user experience.

[0039] See Figure 1 As shown, the embodiments of the present application disclose a storage link failure detection method, including:

[0040] Step S11: Create a monitoring virtual machine on each host node.

[0041] In a specific implementation manner, a monitoring virtual machine can be created on each host node based on the monitoring server and through the Nova service.

[0042] OpenStack is an open-source cloud computing management platform project, which is a combination of a series of software open-source projects. The Nova service is a computing service module of the OpenStack cloud platform, responsible for managing and maintaining the computing resources in the cloud computing environment and the management of the entire life cycle of virtual machines in the cloud environment. When receiving a virtual machine creation request, it schedules the virtual machine to a suitable host according to the scheduling policy. In the OpenStack cloud platform, the Cinder service supports multiple backend storage types through the driver architecture, provides management for the entire life cycle of the volume from creation to deletion, and interacts with the Nova computing service to provide a stable data block storage service for the running virtual machines. From the perspective of the virtual machine, each mounted volume is a disk.

[0043] Step S12: Mount the volumes corresponding to different storage backends to the corresponding monitoring virtual machines.

[0044] In a specific implementation manner, the volumes corresponding to different storage backends can be mounted to the corresponding monitoring virtual machines based on the monitoring server and through the Cinder service.

[0045] Among them, the Cinder service is the block storage service of the OpenStack cloud platform, providing stable data block storage services for running virtual machines. It provides a unified interface for different types of backend storage. Storage device providers only need to implement their drivers in Cinder to integrate with OpenStack. The Cinder service integrates various types of backend storage and provides volume services for virtual machines through the API interface.

[0046] Moreover, this application can combine with user services, refer to the mounting situation of different storage backends on the virtual machines of user services, and mount the volumes corresponding to different storage backends to the corresponding monitoring virtual machines, so as to simulate the running environment of user services. That is, when a storage backend is mounted to a virtual machine of user service, the volume of this storage backend is mounted to the monitoring virtual machine corresponding to this virtual machine of user service.

[0047] In addition, the embodiments of this application can maintain the corresponding relationship between the virtual machine disks and the storage backends. In a specific implementation manner, record the mounting relationship between volumes and virtual machines, and the corresponding relationship between volumes and storage backends.

[0048] Step S13: Obtain health status detection data; among them, the health status detection data is the data obtained by the monitoring virtual machine for performing a health status detection on its own disk.

[0049] In a specific implementation manner, the embodiments of this application can obtain the health status detection data reported by the host agent in the host node; among them, the health status detection data is the data obtained by the monitoring virtual machine for performing a health status detection on its own disk by calling its own virtual machine agent under the trigger of the host agent.

[0050] Moreover, the host agent regularly triggers the monitoring virtual machine to call its own virtual machine agent through QGA (i.e., Qemu-guest-agent) to perform a health status detection on the disk of the monitoring virtual machine itself.

[0051] Correspondingly, the embodiments of this application can pre-place the virtual machine agent and the QGA in the operating system image of the monitoring virtual machine.

[0052] Among them, QGA: A general application program running inside the virtual machine, which realizes a way of interaction between the host and the virtual machine by adding a virtio-serial on the virtual machine.

[0053] In a specific implementation, the host agent performs regular inspections. After monitoring the creation and running of the virtual machine, the host agent interacts with the monitoring virtual machine through the virtio serial channel. The host agent periodically triggers the QGA in the monitoring virtual machine to call the virtual machine agent to perform disk health status detection through the virtio serial channel of the monitoring virtual machine, and obtains the health status detection data.

[0054] Step S14: Determine the data link status between the monitoring virtual machine and the storage backend based on the health status detection data.

[0055] It can be understood that the data link status between the monitoring virtual machine and the storage backend also represents the data link status between the host node and the storage backend.

[0056] In a specific implementation, if a fault occurs in the data link, the corresponding fault information is reported. Specifically, a storage fault alarm can be reported to the platform monitoring and alarm center, or the high-availability recovery process can be triggered. In high-availability computing, if there is a data link fault in the data link status between the monitoring virtual machine and the storage backend, all virtual machines on the host node where the monitoring virtual machine is located may be affected. In order to continue to provide services to users, the virtual machines on the faulty host need to be evacuated to healthy hosts in a timely manner.

[0057] That is to say, this solution is implemented by monitoring three components: the monitoring server, the host agent, and the virtual machine agent. The monitoring server creates a monitoring virtual machine on each host node by interacting with the Nova service, mounts the volumes corresponding to different storage backends to the monitoring virtual machine by interacting with the Cinder service, and maintains the corresponding relationship between the monitoring virtual machine disk and the storage backend. The host agent periodically triggers the virtual machine agent to perform disk health detection through virtio-serial and feeds back the detection results to the monitoring server. The virtual machine agent is pre-installed in the operating system image of the monitoring virtual machine and includes a disk fault detection function module. When called, it detects the health status of the disks mounted inside the monitoring virtual machine and returns the detection results. The monitoring server analyzes the health status of the host node and the storage backend based on the collected disk health detection data. If a fault occurs, a storage fault alarm is reported to the platform monitoring and alarm center, or the high-availability recovery process is triggered.

[0058] It can be seen that in the embodiments of the present application, a monitoring virtual machine is first created on each host node, and then the volumes corresponding to different storage backends are mounted to the corresponding monitoring virtual machines to obtain health status detection data; wherein, the health status detection data is the data obtained by the monitoring virtual machine performing a health status detection on its own disk, and then the data link status between the monitoring virtual machine and the storage backend is determined based on the health status detection data. That is to say, in the embodiments of the present application, a monitoring virtual machine can be created on the host node, and then the volumes corresponding to different storage backends are mounted to the corresponding monitoring virtual machines. In this way, without invading the user business virtual machine, by means of the monitoring virtual machine simulating the running environment of the user business, the data link failure between the virtual machine and the backend storage can be detected in time, and it has generality, thereby improving the user experience.

[0059] For example, as shown in Figure 2 the embodiments of the present application provide a schematic diagram of a specific storage link failure detection solution. It includes three components: watch server (i.e., the monitoring server), host watch (i.e., the host agent), and guest watch (i.e., the virtual machine agent).

[0060] Among them, the monitoring server runs on the control node, the host agent runs on the computing node, and the virtual machine agent runs inside the monitoring virtual machine. The computing node is the host node. Figure 2 An example including 2 computing nodes: computing node 1 and computing node 2 is given in. The monitoring server creates a monitoring virtual machine on each host node by interacting with the Nova computing service. The virtual machine agent and QGA are pre-installed into the operating system image of the monitoring virtual machine and automatically run with the creation and startup of the monitoring virtual machine. The monitoring server mounts the volumes corresponding to different storage backends to the monitoring virtual machine by interacting with the Cinder service. The volumes are presented as local disks inside the monitoring virtual machine, such as vda, vdb... Figure 2 An example including storage 1 and storage 2 is given in. The host agent performs regular inspections. After the monitoring virtual machine is created and running, the host agent interacts with the monitoring virtual machine through the virtio serial channel. The host agent triggers the monitoring virtual machine QGA to call the virtual machine agent to perform disk health status detection through the virtio serial channel of the monitoring virtual machine at regular intervals. The virtual machine agent performs disk health status detection on the local disk of the monitoring virtual machine and returns the result to the host agent. The host agent reports the disk health status detection result of the monitoring virtual machine on this node to the monitoring server. The monitoring server aggregates, analyzes, and diagnoses the disk health status detection results reported by each host agent.

[0061] It should be noted that when the cloud platform is connected to multiple storage backends simultaneously, Cinder will define different volume pools (i.e., storage pools) for the storage backends. At this time, if it is necessary to monitor the health status of all storage backends simultaneously, only need to allocate a volume from the storage pools of different storage backends and mount it to the monitoring virtual machine. For example, if there are two storage backends, Storage 1 and Storage 2, in the cloud platform. After the monitoring virtual machine is created, started, and the volume mounting is completed, the internal disk vda of the monitoring virtual machine uses the volume corresponding to the backend storage 1, and the disk vdb uses the volume corresponding to the backend storage 2. When a disk failure is detected inside the monitoring virtual machine, through the host proxy, the detection result is reported to the monitoring server for summary analysis. Without invading the user's business virtual machine, by simulating the operating environment of the user's business with the monitoring virtual machine, the storage data link failure can be detected as early as possible, warned in time, and processed in time to prevent the failure from escalating and causing damage to the user's business, and it is not limited by the type of storage backend.

[0062] See Figure 3 As shown, an embodiment of the present application discloses a storage link failure detection device, including:

[0063] A monitoring virtual machine creation module 11, configured to create a monitoring virtual machine on each host node;

[0064] A backend storage volume mounting module 12, configured to mount the volumes corresponding to different storage backends to the corresponding monitoring virtual machines;

[0065] A health detection data acquisition module 13, configured to acquire health status detection data; wherein, the health status detection data is the data obtained by the monitoring virtual machine performing a health status detection on its own disk;

[0066] A data link state determination module 14, configured to determine the data link state between the monitoring virtual machine and the storage backend based on the health status detection data.

[0067] It can be seen that in the embodiment of the present application, a monitoring virtual machine is first created on each host node, and then the volumes corresponding to different storage backends are mounted to the corresponding monitoring virtual machines to acquire health status detection data; wherein, the health status detection data is the data obtained by the monitoring virtual machine performing a health status detection on its own disk, and then the data link state between the monitoring virtual machine and the storage backend is determined based on the health status detection data. That is to say, in the embodiment of the present application, a monitoring virtual machine can be created on the host node, and then the volumes corresponding to different storage backends are mounted to the corresponding monitoring virtual machines. In this way, without invading the user's business virtual machine, by simulating the operating environment of the user's business with the monitoring virtual machine, the data link failure between the virtual machine and the backend storage can be detected in time, and it has universality, thereby improving the user experience.

[0068] Among them, the monitoring virtual machine creation module 11 is specifically configured to create a monitoring virtual machine on each host node based on the monitoring server and through the Nova service.

[0069] The backend storage volume mounting module 12 is specifically configured to mount the volumes corresponding to different storage backends to the corresponding monitoring virtual machines based on the monitoring server and through the Cinder service.

[0070] The health detection data acquisition module 13 is specifically configured to acquire the health status detection data reported by the host agent in the host node;

[0071] Among them, the health status detection data is the data obtained by the monitoring virtual machine calling its own virtual machine agent under the trigger of the host agent to perform a health status detection on the disk of the monitoring virtual machine itself.

[0072] Among them, the host agent periodically triggers the monitoring virtual machine to call its own virtual machine agent through QGA to perform a health status detection on the disk of the monitoring virtual machine itself.

[0073] Furthermore, the device further includes a presetting module for presetting the virtual machine agent and the QGA in the operating system image of the monitoring virtual machine.

[0074] Furthermore, the device further includes a fault reporting module for reporting corresponding fault information if a fault occurs in the data link.

[0075] See Figure 4 As shown, an electronic device 20 disclosed in an embodiment of the present application includes a processor 21 and a memory 22; among them, the memory 22 is used to store a computer program; the processor 21 is used to execute the computer program, which is the storage link fault detection method disclosed in the foregoing embodiment.

[0076] For the specific process of the above storage link fault detection method, reference can be made to the corresponding content disclosed in the foregoing embodiment, and details are not described herein again.

[0077] Moreover, as a carrier for resource storage, the memory 22 can be a read-only memory, a random access memory, a disk, or an optical disc, etc., and the storage method can be short-term storage or permanent storage.

[0078] In addition, the electronic device 20 further includes a power supply 23, a communication interface 24, an input / output interface 25, and a communication bus 26. Among them, the power supply 23 is used to provide operating voltage for each hardware device on the electronic device 20. The communication interface 24 can create a data transmission channel between the electronic device 20 and external devices, and the communication protocol it follows can be any communication protocol applicable to the technical solution of this application, and no specific limitation is imposed here. The input / output interface 25 is used to obtain external input data or output data to the outside, and the specific interface type can be selected according to specific application needs, and no specific limitation is imposed here.

[0079] Furthermore, an embodiment of the present application also discloses a computer-readable storage medium for storing a computer program. When the computer program is executed by a processor, it implements the storage link fault detection method disclosed in the foregoing embodiment.

[0080] For the specific process of the above storage link fault detection method, reference can be made to the corresponding content disclosed in the foregoing embodiment, and details will not be repeated here.

[0081] In this specification, each embodiment is described in a progressive manner. The key point of each embodiment is to illustrate the differences from other embodiments. The same or similar parts among the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the description of the method part.

[0082] The steps of the method or algorithm described in connection with the embodiments disclosed in this document can be directly implemented by hardware, a software module executed by a processor, or a combination of both. The software module can be placed in a random access memory (RAM), memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, register, hard disk, removable disk, CD-ROM, or any other form of storage medium well-known in the technical field.

[0083] The above has introduced in detail a storage link fault detection method, device, equipment, and medium provided by this application. Specific examples are used in this document to elaborate on the principle and implementation manner of this application. The description of the above embodiments is only used to help understand the method and its core idea of this application. At the same time, for those of ordinary skill in the art, according to the idea of this application, there will be changes in the specific implementation manner and application scope. In summary, the content of this specification should not be construed as a limitation to this application.

Claims

1. A method for detecting storage link failures, characterized in that, Including: Create a monitoring virtual machine on each host node; Mount the volumes corresponding to different storage backends to the corresponding monitoring virtual machines; Obtain health status detection data; wherein, the health status detection data is data obtained by the monitoring virtual machine performing a health status detection on its own disk; Determine the data link status between the monitoring virtual machine and the storage backend based on the health status detection data; The obtaining of the health status detection data includes: Obtain the health status detection data reported by the host agent in the host node; Wherein, the health status detection data is data obtained by the monitoring virtual machine invoking its own virtual machine agent under the trigger of the host agent to perform a health status detection on the disk of the monitoring virtual machine itself; The host agent periodically triggers the monitoring virtual machine to invoke its own virtual machine agent through QGA to perform a health status detection on the disk of the monitoring virtual machine itself through the virtio serial channel; Also including: Pre-place the virtual machine agent and the QGA in the operating system image of the monitoring virtual machine; The mounting of the volumes corresponding to different storage backends to the corresponding monitoring virtual machines includes: According to the mounting situation of different storage backends on the user service virtual machine, mount the volumes corresponding to different storage backends to the corresponding monitoring virtual machines, so as to realize the simulation of the running environment of the user service.

2. The storage link fault detection method according to claim 1, characterized in that The creating of a monitoring virtual machine on each host node includes: Based on the monitoring server, and through the Nova service, create a monitoring virtual machine on each host node.

3. The storage link fault detection method according to claim 1, characterized in that The mounting of the volumes corresponding to different storage backends to the corresponding monitoring virtual machines includes: Based on the monitoring server, and through the Cinder service, mount the volumes corresponding to different storage backends to the corresponding monitoring virtual machines.

4. The storage link fault detection method according to any one of claims 1 to 3, characterized in that, After determining the data link status between the monitoring virtual machine and the storage backend based on the health status detection data, further including: If a fault occurs in the data link, report the corresponding fault information.

5. A storage link fault detection device, characterized in that, Including: A monitoring virtual machine creation module, used to create a monitoring virtual machine on each host node; A backend storage volume mounting module, used to mount the volumes corresponding to different storage backends to the corresponding monitoring virtual machines; A health detection data acquisition module, used to obtain health status detection data; wherein, the health status detection data is data obtained by the monitoring virtual machine performing a health status detection on its own disk; A data link status determination module, used to determine the data link status between the monitoring virtual machine and the storage backend based on the health status detection data; The health detection data acquisition module is specifically used to obtain the health status detection data reported by the host agent in the host node; Wherein, the health status detection data is data obtained by the monitoring virtual machine invoking its own virtual machine agent under the trigger of the host agent to perform a health status detection on the disk of the monitoring virtual machine itself; Among them, the host agent periodically triggers the monitored virtual machine to call its own virtual machine agent through QGA via the virtio serial channel to detect the health status of the disk of the monitored virtual machine itself; It further includes: A preset module for presetting the virtual machine agent and the QGA in the operating system image of the monitored virtual machine; The backend storage volume mounting module is specifically used to mount the volumes corresponding to different storage backends to the corresponding monitored virtual machines according to the mounting conditions of different storage backends in the user service virtual machine, so as to realize the simulation of the operating environment of the user service.

6. An electronic device, characterized in that, It includes: A memory for storing a computer program; A processor for executing the computer program to implement the storage link fault detection method according to any one of claims 1 to 4.

7. A computer-readable storage medium, characterized in that, For storing a computer program, when the computer program is executed by a processor, it implements the storage link fault detection method according to any one of claims 1 to 4.

Images