Cloud platform virtual machine operating system anomaly detection and recovery method, equipment and medium

Abstract

The invention discloses a cloud platform virtual machine operating system anomaly detection and recovery method and device and a medium, and the method comprises the steps: creating and monitoring a socket file, transmitting a descriptor of the socket file to a qemu process when a virtual machine is created, and when the qemu process obtains an IO error returned by a storage rear end, returning the IO error to a virtual machine operating system layer, the IO error is written into the socket file monitored by the Libvirt; the Libvirt component integrates the monitored IO errors and reports the IO errors to a computing service of a cloud platform, and the computing service of the cloud platform obtains an abnormal event of a virtual machine operating system through an interrupt function registered in Libvirt in advance; the cloud platform computing service performs state detection on an operating system of the virtual machine, detects networks of all virtual machines on a host machine at regular time, and reports abnormal information to a monitoring alarm component; and the monitoring alarm component displays the abnormal information and recovers the virtual machine. The working stability and reliability of the virtual machine in the cloud platform are improved.

Description

Technical Field

[0001] The present invention relates to virtual machine operation and maintenance technology, and in particular to a cloud platform virtual machine operating system abnormality detection and recovery method, equipment and medium. Background Art

[0002] As the scope and scale of private cloud deployments increase dramatically, customers are paying more and more attention to the security of cloud business operations. How to promptly detect and resolve abnormal conditions that may occur in virtual machines on the cloud platform has become an operation and maintenance problem.

[0003] When the application inside the virtual machine sends IO read / write requests to the storage backend through the virtual file system inside the operating system, the virtual file system will select the corresponding driver according to the type of storage backend used. Because the virtio protocol is widely used in virtual machines, the virtio-blk driver is used as an example here. The virtio-blk front end of the operating system inside the virtual machine sends IO read / write requests to the virtio-queue queue. The virtio-blk back end will take out the IO read / write requests sent by the virtio-blk front end driver from the queue, and send or read data to the corresponding storage backend in the actual virtualization layer (qemu layer). If the local file system is connected, the data will be transmitted through the file system on the host; if it is Ceph or other storage, the data will be transmitted through the network layer. When transmitting data in the virtualization layer, IO timeout or IO Error may occur, causing the virtual machine file system to become read-only.

[0004] Here we take the common Ceph storage or centralized storage as an example. Since the corresponding storage network devices need to be added in the cloud platform, when the network controller restarts or crashes, or when there is a network delay or even interruption in the network link, the virtio-blk storage backend in the virtual machine layer cannot obtain the corresponding data within the set time. At the same time, the virtualization layer (qemu process) will return an IO error to the operating system layer of the virtual machine, indicating that the IO request of the operating system layer of the virtual machine has failed. Due to the virtual file system mechanism in the operating system, the file system with IO error will become read-only for protection, which may make the entire operating system unusable.

[0005] The Chinese patent application with publication number CN106201752A proposes a method and system for collecting multipath IO error events in a Linux system. The IO error detection process is at the storage link layer. The difference between the present invention application and the patent application is that the object of IO error detection in the present application is the virtual machine operating system. The detection method is to obtain specific error information through the communication mechanism between libvirt and the virtual machine when the request error occurs during the IO request issued by the virtual machine. The IO error detection process is at the virtualization layer where the virtual machine runs.

[0006] The Chinese patent application with publication number CN105975358A proposes a fault-tolerant method and system based on SCSI devices, which aims to optimize the operating system layer to complete the processing of IO errors, and the work done is still at the storage link layer. Summary of the invention

[0007] In view of the deficiencies in the prior art, the present invention provides a cloud platform virtual machine operating system anomaly detection and recovery method, device and medium, which can detect abnormal virtual machines at an early stage through the computing services in the cloud platform, and automatically restore the virtual machine business according to the pre-established strategy in the monitoring and alarm service, thereby avoiding immeasurable losses caused by business suspension.

[0008] To achieve the above object, the present invention adopts the following technical solutions: A method for detecting and recovering abnormality of a virtual machine operating system on a cloud platform comprises the following steps: Libvirt creates a socket file and listens to it. When creating a virtual machine, it passes the descriptor of the socket file to the qemu process for reading and writing the socket file. When the qemu process obtains the IO error returned by the storage backend, it returns the IO error to the virtual machine operating system layer and writes the IO error to the socket file monitored by Libvirt. The Libvirt component integrates the monitored IO errors and reports them to the computing service of the cloud platform through registration. The computing service of the cloud platform obtains the abnormal events of the virtual machine operating system through the interrupt function registered in Libvirt in advance. The cloud platform computing service detects the status of the virtual machine's operating system and periodically detects the network of all virtual machines on the host machine, and reports abnormal information to the monitoring and alarm component; The monitoring alarm component displays abnormal information and recovers the virtual machine according to the pre-set abnormal strategy.

[0009] To optimize the above technical solutions, the specific measures taken also include: Furthermore, the step of writing the IO error into the socket file monitored by Libvirt is as follows: The time, reason, location, and size of the IO error are written as parameters to the socket file.

[0010] Furthermore, the cloud platform computing service performs status detection on the operating system of the virtual machine, and periodically detects the network of all virtual machines on the host machine, and reports abnormal information to the monitoring alarm component. Specifically: The cloud platform computing service determines whether the current virtual machine has any abnormality by writing specific instructions to the QGA driver device in the virtual machine, including accessing the system disk, data disk and specific file reading and writing in the virtual machine operating system; it periodically accesses the specific network IP through QGA for all virtual machines managed on the host machine to determine whether the operating system network of all virtual machines on the host machine is normal; if the computing service returns normally through the instruction of accessing the virtual machine operating system through QGA, the reported abnormal event will be ignored; if the file cannot be accessed, the specific disk cannot be read or written, or the operating system does not respond, the erroneous virtual machine information, including the name of the virtual machine, global identifier, error cause and read and write block size, will be reported to the monitoring and alarm component of the cloud platform through the message call mechanism in the cloud platform.

[0011] Furthermore, the recovery of the virtual machine according to the pre-set abnormal strategy is specifically as follows: If a problem occurs in the virtual machine network, the monitoring alarm component calls the virtual machine's evacuation interface to dispatch the virtual machine to other normal host nodes; or it is set as an alarm event to display the problematic virtual machine in the monitoring alarm component; If an IO error occurs in the virtual machine operating system, the virtual machine will be scheduled to other normal host nodes, or the virtual machine will be restarted, or set as an alarm event.

[0012] The present invention also proposes an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the cloud platform virtual machine operating system anomaly detection and recovery method as described above is implemented.

[0013] The present invention also proposes a computer-readable storage medium storing a computer program, wherein the computer program enables a computer to execute the cloud platform virtual machine operating system anomaly detection and recovery method as described above.

[0014] The beneficial effect of the present invention is that the present invention can monitor and alarm all virtual machine operating systems in the cloud platform, and recover according to the corresponding strategy according to the specific error type. The present invention creatively proposes an event loop mechanism that passes error information such as IO error in the qemu process to libvirt through a socket file, and reports it to the computing service in the cloud platform. The computing service will confirm the state of the virtual machine operating system through the qga driver device in the virtual machine, and determine whether the current abnormal event will affect the normal operation of the virtual machine operating system. If the virtual machine operating system is detected to be abnormal, the computing service will use the message call mechanism in the cloud platform to report the information of the virtual machine, such as the name, global identifier, node, and error type, to the monitoring and alarm platform. The monitoring and alarm platform will display it, and at the same time, it will recover the virtual machine in the abnormal state according to the pre-set recovery strategy. The invention can quickly identify the virtual machine whose operating system is in an abnormal state, and can effectively recover it. At the same time, the alarm information obtained is displayed, which greatly saves the workload of the operation and maintenance personnel in maintaining the cloud platform, and improves the stability and reliability of the virtual machine work in the cloud platform. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 It is a schematic diagram of the cloud platform virtual machine operating system anomaly detection and recovery method proposed by the present invention. DETAILED DESCRIPTION

[0016] The present invention will now be described in further detail with reference to the accompanying drawings.

[0017] Embodiment 1

[0018] The present invention proposes a method for detecting and recovering abnormalities in a virtual machine operating system on a cloud platform. The principle diagram of the method is shown in FIG. Figure 1 As shown, the following steps are included: First, the virtual machines in the cloud platform are managed and operated by libvirt. When libvirt starts the virtual machine, a socket file will be created at the same time, and the file descriptor of the socket will be passed to the qemu process to be created. After that, libvirt will monitor the event of the socket. When the qemu process obtains the IOerror returned by the storage backend, it will return the IO error to the virtual machine operating system layer. The present invention performs function optimization in the qemu process, and also sends the error information of the IO error to the socket file descriptor monitored by libvirt.

[0019] The present invention writes the key information such as the time, reason, location, size, etc. of the IO error generated by the virtual machine qemu process into the socket file as key parameters. At the same time, with the help of the event loop mechanism of the libvirt management software layer, the obtained IO error information is integrated and reported to the cloud platform. The computing service in the cloud platform obtains the corresponding IO error abnormal event through the interrupt function pre-registered in libvirt.

[0020] After the computing service in the cloud platform obtains the abnormal event of the virtual machine operating system, it will identify and process the abnormal information of the virtual machine, and determine whether the current virtual machine has an abnormal situation by writing specific instructions to the QGA driver device in the virtual machine, such as accessing the system disk, data disk and specific file reading and writing in the virtual machine operating system. At the same time, the computing service will periodically access the specific network IP of all virtual machines managed on the host through QGA to determine whether the operating system network of all virtual machines on the host is normal.

[0021] If the instruction of the computing service to access the virtual machine operating system through qga returns normally, the abnormal event reported this time will be ignored; for example, through qga, normal reading and writing can be performed on each file system in the operating system, indicating that the operating system is not abnormal at this time, and the abnormal event will be ignored. If the file system cannot be accessed or a specific disk cannot be read or written, or even the operating system does not respond, the erroneous virtual machine information, including the name of the virtual machine, global identifier, cause of error, read and write block size, etc., will be reported to the monitoring and alarm component of the cloud platform through the message call mechanism in the cloud platform. At the same time, the mechanism in the computing service that periodically determines whether the virtual machine network service is normal will also periodically report information to the monitoring and alarm component.

[0022] The operation and maintenance personnel will set corresponding event processing rules in the monitoring and alarm component, such as restarting the virtual machine, evacuating the virtual machine, and displaying alarm information. When the service monitoring component receives the virtual machine operating system abnormality event reported by the nova-compute service, it will process the virtual machine according to the above rules and display the operation logs such as the cause of the virtual machine operation abnormality, the time when the abnormality occurred, and the subsequent processing results on the interface to facilitate the operation and maintenance personnel to investigate and manage.

[0023] The present invention also proposes a monitoring and alarm strategy for abnormal operation of virtual machines. If the virtual machine has a network problem, it may be that the host node where the virtual machine is located has a fault. The evacuation interface of the virtual machine can be called through the monitoring and alarm component to dispatch the virtual machine to other normal host nodes; it can also be set as an alarm event to display the problematic virtual machine in the monitoring and alarm component. If the virtual machine operating system has other abnormal events such as IO error, it can be set to evacuate, restart the virtual machine or alarm, and the operation and maintenance personnel can adopt corresponding strategies for possible abnormalities of the virtual machine according to actual needs.

[0024] Embodiment 2

[0025] The present invention proposes an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the cloud platform virtual machine operating system anomaly detection and recovery method as described in Example 1 is implemented.

[0026] Embodiment 3

[0027] The present invention provides a computer-readable storage medium storing a computer program, wherein the computer program enables a computer to execute the cloud platform virtual machine operating system anomaly detection and recovery method as described in Example 1.

[0028] In the embodiments disclosed in the present application, the computer storage medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or apparatus. The computer storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or equipment, or any suitable combination of the foregoing. More specific examples of computer storage media may include electrical connections based on one or more lines, portable computer disks, hard disks, random access memories (RAM), read-only memories (ROM), erasable programmable read-only memories (EPROM or flash memory), optical fibers, portable compact disk read-only memories (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0029] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed in this application can be implemented in electronic hardware or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0030] The above are only preferred embodiments of the present invention, and the protection scope of the present invention is not limited to the above embodiments. All technical solutions under the concept of the present invention belong to the protection scope of the present invention. It should be pointed out that for ordinary technicians in this technical field, some improvements and modifications without departing from the principle of the present invention should be regarded as the protection scope of the present invention.

Claims

1. A method for detecting and recovering abnormalities in a virtual machine operating system on a cloud platform, characterized in that: The following steps are involved: Libvirt creates a socket file and listens to it. When creating a virtual machine, it passes the descriptor of the socket file to the qemu process for reading and writing the socket file. When the qemu process obtains the IO error returned by the storage backend, it returns the IO error to the virtual machine operating system layer and writes the IO error to the socket file monitored by Libvirt. The Libvirt component integrates the monitored IO errors and reports them to the computing service of the cloud platform through registration. The computing service of the cloud platform obtains the abnormal events of the virtual machine operating system through the interrupt function registered in Libvirt in advance. The cloud platform computing service detects the status of the virtual machine's operating system and periodically detects the network of all virtual machines on the host machine, and reports abnormal information to the monitoring and alarm component; The monitoring alarm component displays abnormal information and recovers the virtual machine according to the pre-set abnormal strategy.

2. The cloud platform virtual machine operating system abnormality detection and recovery method according to claim 1, characterized in that: The specific steps of writing the IO error to the socket file monitored by Libvirt are as follows: The time, reason, location, and size of the IO error are written as parameters to the socket file.

3. The cloud platform virtual machine operating system abnormality detection and recovery method according to claim 1, characterized in that: The cloud platform computing service performs status detection on the operating system of the virtual machine, and periodically detects the network of all virtual machines on the host machine, and reports abnormal information to the monitoring alarm component. Specifically: The cloud platform computing service determines whether the current virtual machine has any abnormality by writing specific instructions to the QGA driver device in the virtual machine, including accessing the system disk, data disk and specific file reading and writing in the virtual machine operating system; it periodically accesses the specific network IP through QGA for all virtual machines managed on the host machine to determine whether the operating system network of all virtual machines on the host machine is normal; if the computing service returns normally through the instruction of accessing the virtual machine operating system through QGA, the reported abnormal event will be ignored; if the file cannot be accessed, the specific disk cannot be read or written, or the operating system does not respond, the erroneous virtual machine information, including the name of the virtual machine, global identifier, error cause and read and write block size, will be reported to the monitoring and alarm component of the cloud platform through the message call mechanism in the cloud platform.

4. The cloud platform virtual machine operating system abnormality detection and recovery method according to claim 1, characterized in that: The recovery of the virtual machine according to the pre-set abnormal strategy is specifically as follows: If a problem occurs in the virtual machine network, the monitoring alarm component calls the virtual machine's evacuation interface to dispatch the virtual machine to other normal host nodes; or it is set as an alarm event to display the problematic virtual machine in the monitoring alarm component; If an IO error occurs in the virtual machine operating system, the virtual machine will be scheduled to other normal host nodes, or the virtual machine will be restarted, or set as an alarm event.

5. An electronic device, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the cloud platform virtual machine operating system anomaly detection and recovery method as described in any one of claims 1 to 4 is implemented.

6. A computer-readable storage medium, characterized in that: A computer program is stored, and the computer program enables a computer to execute the cloud platform virtual machine operating system anomaly detection and recovery method as described in any one of claims 1 to 4.

Images