Distributed storage hard disk fault processing method and device

Abstract

The present invention discloses a distributed storage hard disk fault processing method and device, which can be used in the financial field or other technical fields. The distributed storage hard disk fault processing method includes: performing storage hard disk abnormality detection on the target storage pool; if the detection result is that the storage hard disk is abnormal, sending an abnormality notification to all servers of applications that deploy application virtual machines in the target storage pool, so that the servers of each application isolate the application virtual machines deployed in the target storage pool according to the abnormality notification. The present invention effectively solves the problem of affecting the normal operation of the application due to the abnormality of the storage hard disk, which helps to improve the service quality of the application.

Description

Technical Field

[0001] The present invention relates to the field of distributed storage technology, and in particular to a distributed storage hard disk failure processing method and device. Background Art

[0002] Distributed storage is a data storage technology that builds dispersed storage resources into a virtual storage resource pool through the network and stores data in multiple independent devices. It has high-performance large concurrent read and write, high-availability fault automatic isolation, dynamic expansion, and automated and intelligent operation and maintenance management. A distributed storage cluster is generally composed of multiple storage pools. The storage pools are independent of each other and do not affect each other. A storage pool contains multiple storage hard disks, and the data is ultimately written on the storage hard disks. Applications read and write data to distributed storage clusters by deploying application virtual machines on multiple storage pools to achieve normal operation of applications. If a storage pool fails or has performance problems, it will inevitably affect the normal operation of applications. When each storage hard disk in the storage pool is abnormal or fails, it will affect the performance of the entire storage pool, and then affect the normal operation of applications deployed in the storage pool. However, how to solve the problem of application failure due to abnormal storage hard disks, the existing technology does not have a good solution. Summary of the invention

[0003] In order to solve the technical problems in the above-mentioned background technology, the present invention proposes a distributed storage hard disk failure processing method and device.

[0004] In order to achieve the above object, according to one aspect of the present invention, a method for handling a distributed storage hard disk failure is provided, the method comprising:

[0005] Perform storage hard disk anomaly detection on the target storage pool;

[0006] If the detection result is that the storage hard disk is abnormal, an abnormal notification is sent to all servers of applications that deploy application virtual machines in the target storage pool, so that the servers of each application isolate the application virtual machines deployed in the target storage pool according to the abnormal notification.

[0007] Optionally, the distributed storage hard disk failure processing method further includes:

[0008] After the server of each application isolates the application virtual machine of the application deployed in the target storage pool, determine the faulty hard disk in the target storage pool, wherein if the storage hard disk in the target storage pool detects M abnormalities within a preset time, then determine that the storage hard disk is a faulty hard disk;

[0009] A hard disk failure notification is sent to a management node of a distributed storage cluster according to the failed hard disk, so that the management node isolates the failed hard disk in the target storage pool according to the hard disk failure notification.

[0010] Optionally, the distributed storage hard disk failure processing method further includes:

[0011] After isolating the faulty hard disk, a troubleshooting notification is sent to all application servers that deploy application virtual machines in the target storage pool, so that the servers of each application release the isolation of the application virtual machines deployed in the target storage pool according to the troubleshooting notification.

[0012] Optionally, the performing storage hard disk anomaly detection on the target storage pool specifically includes:

[0013] Performing storage hard disk anomaly detection on the target storage pool according to a preset detection frequency;

[0014] If the detection result is that the storage hard disk is abnormal, sending an abnormality notification to all servers of applications that deploy application virtual machines in the target storage pool specifically includes:

[0015] If the detection results for N consecutive times are that the storage hard disk is abnormal, an abnormality notification is sent to all servers of the application that deploys the application virtual machine in the target storage pool.

[0016] Optionally, the performing storage hard disk anomaly detection on the target storage pool includes:

[0017] Simultaneously sending an anomaly detection request to multiple storage hard disks in the target storage pool, and obtaining a response result of each storage hard disk to the anomaly detection request;

[0018] Determine whether each storage hard disk is abnormal according to the response result;

[0019] If at least one storage hard disk is abnormal, the detection result is determined to be that the storage hard disk is abnormal.

[0020] Optionally, determining whether each storage hard disk is abnormal according to the response result includes:

[0021] If the response result is no response, an error is returned, or the response delay is greater than a preset threshold, it is determined that the storage hard disk is abnormal.

[0022] Optionally, the sending anomaly detection requests to multiple storage hard disks in the target storage pool at the same time, and obtaining a response result of each storage hard disk to the anomaly detection request, specifically includes:

[0023] Sending an abnormality detection request to a data read / write module of the target storage pool, so that the data read / write module simultaneously sends the abnormality detection request to multiple storage hard disks in the target storage pool;

[0024] A response result of each storage hard disk to the abnormality detection request is obtained from the data reading and writing module.

[0025] Optionally, the abnormality detection request includes: a write IO request or a read IO request.

[0026] In order to achieve the above object, according to another aspect of the present invention, a distributed storage hard disk fault processing device is provided, the device comprising:

[0027] A hard disk anomaly detection unit, used to detect storage hard disk anomalies in a target storage pool;

[0028] The exception notification sending unit is used to send an exception notification to all servers of applications that deploy application virtual machines in the target storage pool if the detection result is a storage hard disk exception, so that the server of each application isolates the application virtual machine deployed in the target storage pool according to the exception notification.

[0029] In order to achieve the above-mentioned purpose, according to another aspect of the present invention, a computer device is also provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps in the above-mentioned distributed storage hard disk failure handling method when executing the computer program.

[0030] In order to achieve the above-mentioned purpose, according to another aspect of the present invention, a computer-readable storage medium is further provided, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed in a computer processor, the steps in the above-mentioned distributed storage hard disk failure handling method are implemented.

[0031] The beneficial effects of the present invention are as follows: the embodiment of the present invention performs storage hard disk anomaly detection on the target storage pool, and when the detection result is a storage hard disk anomaly, sends an exception notification to all servers of applications that deploy application virtual machines in the target storage pool, so that the servers of each application isolate the application virtual machines deployed in the target storage pool according to the exception notification, thereby effectively solving the problem of affecting the normal operation of the application due to the anomaly of the storage hard disk, and helping to improve the service quality of the application. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the prior art descriptions. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. In the drawings:

[0033] Figure 1 This is a first flow chart of a method for handling a distributed storage hard disk failure according to an embodiment of the present invention;

[0034] Figure 2 is a second flow chart of the method for handling a distributed storage hard disk failure according to an embodiment of the present invention;

[0035] Figure 3 is a first flow chart of performing storage hard disk abnormality detection according to an embodiment of the present invention;

[0036] Figure 4 is a second flow chart of performing storage hard disk abnormality detection according to an embodiment of the present invention;

[0037] Figure 5 It is a structural block diagram of a distributed storage hard disk fault processing device according to an embodiment of the present invention;

[0038] Figure 6 It is a schematic diagram of a computer device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0039] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0040] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0041] It should be noted that the terms "including" and "having" and any variations thereof in the specification and claims of the present invention and the above-mentioned drawings are intended to cover non-exclusive inclusions. For example, a process, method, system, product or apparatus comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or apparatus.

[0042] It should be noted that, in the absence of conflict, the embodiments of the present invention and the features in the embodiments can be combined with each other. The present invention will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0043] It should be noted that the distributed storage hard disk failure processing method and device of the present invention can be used in the financial field, and can also be used in other technical fields.

[0044] Figure 1 is a first flow chart of a method for handling a distributed storage hard disk failure according to an embodiment of the present invention. Figure 1 As shown, the distributed storage hard disk failure processing method of this embodiment includes steps S101 to S102.

[0045] Step S101: Perform storage hard disk anomaly detection on the target storage pool.

[0046] In an embodiment of the present invention, a distributed storage cluster is generally composed of multiple storage pools, which are independent of each other and will not affect each other. A storage pool is composed of multiple storage servers, and the failure of each server will affect the performance of the storage pool (in fact, a server failure is equivalent to the failure of all storage hard disks on each server). A storage server has multiple storage hard disks, and the failure of each storage hard disk will also affect the performance of the storage pool, and data will eventually be written on the storage hard disks.

[0047] The present invention sets a corresponding hard disk anomaly detection device for each storage pool in a distributed storage cluster. The hard disk anomaly detection device is used to perform anomaly detection on each storage hard disk in the corresponding storage pool. The implementation subject of this step is the hard disk anomaly detection device corresponding to the target storage pool. In one embodiment of the present invention, the hard disk anomaly detection device can be a detection virtual machine set in the storage pool. Specifically, when each storage pool of the storage cluster is enabled, the present invention automatically creates a detection virtual machine in the storage pool and mounts a volume device.

[0048] Step S102, if the detection result is that the storage hard disk is abnormal, an abnormal notification is sent to all application servers that deploy application virtual machines in the target storage pool, so that the server of each application isolates the application virtual machine deployed in the target storage pool according to the abnormal notification.

[0049] In the present invention, applications are usually deployed in a high-availability manner by deploying application virtual machines on multiple storage pools. When one of the storage pools fails, the application virtual machines deployed on the storage pool may not be able to read and write data normally, causing the application virtual machines to fail to operate normally and fail to provide services to users. Therefore, the present invention notifies all applications that deploy application virtual machines in the storage pool that may fail to operate to isolate the application virtual machines in the storage pool, while the application virtual machines deployed on other storage pools can still operate normally and provide services to users. Therefore, the present invention can effectively solve the problem of affecting the normal operation of the application due to abnormalities in the storage hard disk, which helps to improve the service quality of the application and the user experience.

[0050] In the embodiment of the present invention, the implementation subject of this step is a hard disk anomaly detection device (detecting virtual machines).

[0051] In an embodiment of the present invention, when the distributed storage cluster is built and enabled, the present invention automatically creates a scheduling virtual machine outside the distributed storage cluster, and the scheduling virtual machine saves the corresponding relationship between the storage pool and the application virtual machine. In one embodiment of the present invention, this step is specifically as follows: if the detection result is that the storage hard disk is abnormal, the hard disk abnormality detection device (detection virtual machine) sends a notification to the scheduling virtual machine, and the notification includes the identifier of the target storage pool; the scheduling virtual machine sends an abnormality notification to all servers of the application that deploys the application virtual machine in the target storage pool according to the identifier of the target storage pool and the corresponding relationship between the stored storage pool and the application virtual machine, so that the server of each application isolates the application virtual machine deployed in the target storage pool according to the abnormality notification.

[0052] Figure 2 is a second flow chart of the method for handling a distributed storage hard disk failure according to an embodiment of the present invention, and the implementation subject may be the above-mentioned hard disk anomaly detection device (detecting a virtual machine), such as Figure 2 As shown, the distributed storage hard disk failure processing method of the embodiment of the present invention also includes step S201 and step S202.

[0053] Step S201, after the server of each application isolates the application virtual machine deployed in the target storage pool, determines the faulty hard disk in the target storage pool, wherein if the storage hard disk in the target storage pool detects M abnormalities within a preset time, the storage hard disk is determined to be a faulty hard disk.

[0054] After the present invention applies virtual machine isolation to the storage pool where the fault occurs, it is also necessary to determine the faulty hard disk in the storage pool to eliminate the fault. In an embodiment of the present invention, if a storage hard disk detects M abnormalities within a preset time, it is determined to be a faulty hard disk. In an optional embodiment of the present invention, the preset time ranges from 30 seconds to 2 minutes, and M is an integer greater than or equal to 3.

[0055] Step S202: Send a hard disk failure notification to a management node of a distributed storage cluster according to the failed hard disk, so that the management node isolates the failed hard disk in the target storage pool according to the hard disk failure notification.

[0056] In an embodiment of the present invention, after finding the faulty hard disk in the storage pool, the present invention sends a hard disk failure notification to the management node, and the hard disk failure notification includes a storage pool identifier and a hard disk identifier. The management node isolates the faulty hard disk in the target storage pool based on the hard disk failure notification.

[0057] In one embodiment of the present invention, after the above step S202, the distributed storage hard disk failure processing method of the present invention embodiment further includes:

[0058] After isolating the faulty hard disk, a troubleshooting notification is sent to all application servers that deploy application virtual machines in the target storage pool, so that the servers of each application release the isolation of the application virtual machines deployed in the target storage pool according to the troubleshooting notification.

[0059] In an embodiment of the present invention, after the faulty hard disk in the target storage pool is eliminated, the target storage pool returns to normal. At this time, the isolation of the application virtual machines deployed in the target storage pool in the above step S102 can be released, and the application virtual machines deployed in the target storage pool can now work normally.

[0060] It can be seen that the present invention isolates the application virtual machine when a hard disk failure occurs in the storage pool, and releases the isolation of the application virtual machine after the hard disk failure is eliminated, which can effectively solve the problem of affecting the normal operation of the application due to the abnormality of the storage hard disk, help to improve the service quality of the application, and help to improve the user experience.

[0061] In one embodiment of the present invention, the above step S101 of performing storage hard disk anomaly detection on the target storage pool specifically includes: performing storage hard disk anomaly detection on the target storage pool according to a preset detection frequency.

[0062] In an optional embodiment of the present invention, the preset detection frequency may be once per second.

[0063] In one embodiment of the present invention, if the detection result of the above step S102 is that the storage hard disk is abnormal, an abnormal notification is sent to all servers of applications that deploy application virtual machines in the target storage pool, specifically including:

[0064] If the detection results for N consecutive times are that the storage hard disk is abnormal, an abnormality notification is sent to all servers of the application that deploys the application virtual machine in the target storage pool.

[0065] The present invention is to improve detection. In an optional embodiment of the present invention, N is an integer greater than or equal to 2, and preferably N is equal to 3.

[0066] Figure 3 FIG. 1 is a first flow chart of an abnormality detection of a storage hard disk according to an embodiment of the present invention. Figure 3 As shown, the above step S101 of performing storage hard disk abnormality detection on the target storage pool specifically includes steps S301 to S303.

[0067] Step S301, simultaneously sending an abnormality detection request to multiple storage hard disks in the target storage pool, and obtaining a response result of each storage hard disk to the abnormality detection request.

[0068] In one embodiment of the present invention, the abnormality detection request includes: a write IO request or a read IO request. Since the write IO request is faster, the preferred abnormality detection request is the write IO request.

[0069] In an embodiment of the present invention, the hard disk anomaly detection device (detection virtual machine) performs storage hard disk anomaly detection on the target storage pool according to a preset detection frequency, and sends anomaly detection requests to multiple storage hard disks in the target storage pool at the same time each time the storage hard disk anomaly detection is performed. Specifically, the present invention can randomly select a preset number of storage hard disks from all storage hard disks in the target storage pool to send anomaly detection requests.

[0070] Step S302: Determine whether each storage hard disk is abnormal according to the response result.

[0071] In the embodiment of the present invention, if the response result is no response, an error is returned, or the response delay is greater than a preset threshold, it is determined that the storage hard disk is abnormal.

[0072] Step S303: If at least one storage hard disk is abnormal, determine that the detection result is that the storage hard disk is abnormal.

[0073] In the embodiment of the present invention, each time the storage hard disk abnormality detection is performed, if at least one storage hard disk is abnormal, the detection result of this time is determined to be that the storage hard disk is abnormal.

[0074] Figure 4is a second flow chart of performing storage hard disk abnormality detection according to an embodiment of the present invention, such as Figure 4 As shown, the above step S301 describes sending an anomaly detection request to multiple storage hard disks in the target storage pool at the same time, and obtaining a response result of each storage hard disk to the anomaly detection request, specifically including step S401 and step S402.

[0075] Step S401: Send an abnormality detection request to a data read / write module of the target storage pool, so that the data read / write module simultaneously sends the abnormality detection request to multiple storage hard disks in the target storage pool.

[0076] In the embodiment of the present invention, the hard disk anomaly detection device (detection virtual machine) performs storage hard disk anomaly detection on the target storage pool according to a preset detection frequency, and sends an anomaly detection request to the data read-write module of the target storage pool each time the storage hard disk anomaly detection is performed. Then, the data read-write module selects multiple storage hard disks from all storage hard disks of the target storage pool according to a preset rule to send an anomaly detection request.

[0077] Step S402: Obtain, from the data reading and writing module, a response result of each storage hard disk to the abnormality detection request.

[0078] In the embodiment of the present invention, each storage hard disk feeds back a response result of the abnormality detection request to the data read-write module, and the hard disk abnormality detection device (detection virtual machine) obtains the response result of each storage hard disk from the data read-write module.

[0079] It can be seen from the above embodiments that the present invention provides a method for handling distributed storage hard disk failures. By creating a detection virtual machine in the storage pool to perform IO availability detection, the IO responses and results of each storage hard disk sent to the storage server are analyzed and detected. When an abnormal result occurs, the application virtual machine in the current storage pool is notified to perform high-availability switching, and then the IO detection of the faulty disk is continuously performed. After the faulty hard disk is determined, it is isolated, and the distributed storage service capability is quickly restored. The advantages of the present invention are as follows:

[0080] 1. Automatically create a detection virtual machine to detect IO availability, which can reduce the average detection time of failures and minimize the impact on business;

[0081] 2. Automatically schedule and switch the storage upper-layer application virtual machine to ensure that the business is not affected;

[0082] 3. The combination of automatic switching and timing detection can prevent false isolation caused by short coding detection cycle and detect faults more accurately.

[0083] It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and that, although a logical order is shown in the flowcharts, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0084] Based on the same inventive concept, an embodiment of the present invention also provides a distributed storage hard disk fault processing device, which can be used to implement the distributed storage hard disk fault processing method described in the above embodiment, as described in the following embodiment. Since the principle of solving the problem by the distributed storage hard disk fault processing device is similar to that of the distributed storage hard disk fault processing method, the embodiment of the distributed storage hard disk fault processing device can refer to the embodiment of the distributed storage hard disk fault processing method, and the repeated parts will not be repeated. As used below, the term "unit" or "module" can be a combination of software and / or hardware that implements the predetermined function. Although the device described in the following embodiments is preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.

[0085] Figure 5 is a structural block diagram of a distributed storage hard disk fault processing device according to an embodiment of the present invention. Figure 5 As shown, the distributed storage hard disk fault processing device according to the embodiment of the present invention includes:

[0086] The hard disk anomaly detection unit 1 is used to detect storage hard disk anomalies in the target storage pool;

[0087] The exception notification sending unit 2 is used to send an exception notification to all servers of applications that deploy application virtual machines in the target storage pool if the detection result is a storage hard disk exception, so that the server of each application isolates the application virtual machine deployed in the target storage pool according to the exception notification.

[0088] In one embodiment of the present invention, the distributed storage hard disk fault processing device of the present invention further includes:

[0089] a faulty hard disk determining unit, configured to determine a faulty hard disk in the target storage pool after the server of each application isolates the application virtual machine of the application deployed in the target storage pool, wherein if a storage hard disk in the target storage pool detects M abnormalities within a preset time, the storage hard disk is determined to be a faulty hard disk;

[0090] The hard disk failure notification sending unit is used to send a hard disk failure notification to the management node of the distributed storage cluster according to the failed hard disk, so that the management node isolates the failed hard disk in the target storage pool according to the hard disk failure notification.

[0091] In one embodiment of the present invention, the distributed storage hard disk fault processing device of the present invention further includes:

[0092] A troubleshooting notification sending unit is used to send a troubleshooting notification to all servers of applications that deploy application virtual machines in the target storage pool after isolating the faulty hard disk, so that the servers of each application can release the isolation of the application virtual machines deployed in the target storage pool according to the troubleshooting notification.

[0093] In one embodiment of the present invention, the hard disk anomaly detection unit 1 performs storage hard disk anomaly detection on the target storage pool according to a preset detection frequency; the anomaly notification sending unit 2 is specifically used to send an anomaly notification to all servers of applications that deploy application virtual machines in the target storage pool if N consecutive detection results are storage hard disk anomalies.

[0094] In one embodiment of the present invention, the hard disk abnormality detection unit 1 includes:

[0095] A detection request sending module, used to simultaneously send an anomaly detection request to multiple storage hard disks in the target storage pool, and obtain a response result of each storage hard disk to the anomaly detection request;

[0096] A judgment module, used to determine whether each storage hard disk is abnormal according to the response result;

[0097] The determination module is used to determine that the detection result is that the storage hard disk is abnormal if at least one storage hard disk is abnormal.

[0098] In one embodiment of the present invention, the judgment module is specifically used to determine that the storage hard disk is abnormal if the response result is no response, an error is returned, or the response delay is greater than a preset threshold.

[0099] In one embodiment of the present invention, the detection request sending module is specifically used to send an exception detection request to the data read and write module of the target storage pool, so that the data read and write module simultaneously sends the exception detection request to multiple storage hard disks in the target storage pool; and obtain the response result of each storage hard disk to the exception detection request from the data read and write module.

[0100] In one embodiment of the present invention, the abnormality detection request includes: a write IO request or a read IO request.

[0101] In order to achieve the above object, according to another aspect of the present application, a computer device is also provided. Figure 6As shown, the computer device includes a memory, a processor, a communication interface and a communication bus. A computer program that can be executed on the processor is stored in the memory. When the processor executes the computer program, the steps in the above-mentioned embodiment method are implemented.

[0102] The processor may be a central processing unit (CPU). The processor may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, or a combination of the above chips.

[0103] The memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs, non-transitory computer executable programs and units, such as the corresponding program units in the above method embodiments of the present invention. The processor executes various functional applications of the processor and works data processing by running the non-transitory software programs, instructions and modules stored in the memory, that is, implementing the method in the above method embodiments.

[0104] The memory may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created by the processor, etc. In addition, the memory may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some embodiments, the memory may optionally include a memory remotely arranged relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0105] The one or more units are stored in the memory, and when executed by the processor, the method in the above embodiment is performed.

[0106] The specific details of the above-mentioned computer device can be understood by referring to the corresponding related descriptions and effects in the above-mentioned embodiments, and will not be repeated here.

[0107] In order to achieve the above purpose, according to another aspect of the present application, a computer-readable storage medium is also provided, wherein the computer-readable storage medium stores a computer program, and the computer program implements the steps in the above-mentioned distributed storage hard disk fault handling method when executed in a computer processor. Those skilled in the art can understand that the implementation of all or part of the process in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a computer program, and the program can be stored in a computer-readable storage medium, and the program can include the process of the embodiment of each method as described above when executed. Among them, the storage medium can be a disk, an optical disk, a read-only memory (ROM), a random access memory (RAM), a flash memory (Flash Memory), a hard disk (Hard Disk Drive, abbreviated as: HDD) or a solid-state drive (SSD), etc.; the storage medium can also include a combination of the above-mentioned types of memory.

[0108] Obviously, those skilled in the art should understand that the above modules or steps of the present invention can be implemented by a general computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, and optionally, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. Thus, the present invention is not limited to any specific combination of hardware and software.

[0109] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for handling a distributed storage hard disk failure, characterized in that: include: A hard disk anomaly detection device corresponding to the target storage pool in the distributed storage cluster is used to perform storage hard disk anomaly detection on the target storage pool, wherein the distributed storage cluster is composed of multiple storage pools, one storage pool is composed of multiple storage servers, one storage server has multiple storage hard disks, and each storage pool in the distributed storage cluster is provided with a corresponding hard disk anomaly detection device; If the detection result is that the storage hard disk is abnormal, an abnormal notification is sent to all servers of the application that deploys the application virtual machine in the target storage pool, so that the server of each application isolates the application virtual machine deployed in the target storage pool according to the abnormal notification. The application achieves high-availability deployment by deploying the application virtual machine on multiple storage pools. When the distributed storage cluster is built and enabled, a scheduling virtual machine is automatically created outside the distributed storage cluster. After the server of each application isolates the application virtual machine of the application deployed in the target storage pool, determine the faulty hard disk in the target storage pool, wherein if the storage hard disk in the target storage pool detects M abnormalities within a preset time, then determine that the storage hard disk is a faulty hard disk; Sending a hard disk failure notification to a management node of a distributed storage cluster according to the failed hard disk, so that the management node isolates the failed hard disk in the target storage pool according to the hard disk failure notification; If the detection result is that the storage hard disk is abnormal, an abnormal notification is sent to all servers of applications that deploy application virtual machines in the target storage pool, so that the servers of each application isolate the application virtual machines deployed in the target storage pool according to the abnormal notification, including: If the detection result is a storage hard disk abnormality, a notification is sent to the scheduling virtual machine through the hard disk abnormality detection device corresponding to the target storage pool, and the notification includes the identifier of the target storage pool; the scheduling virtual machine sends an abnormality notification to all servers of applications that deploy application virtual machines in the target storage pool based on the identifier of the target storage pool and the correspondence between the stored storage pool and the application virtual machines, so that the servers of each application isolate the application virtual machines deployed in the target storage pool according to the abnormality notification.

2. The distributed storage hard disk failure processing method according to claim 1 is characterized in that: Also includes: After isolating the faulty hard disk, a troubleshooting notification is sent to all application servers that deploy application virtual machines in the target storage pool, so that the servers of each application release the isolation of the application virtual machines deployed in the target storage pool according to the troubleshooting notification.

3. The distributed storage hard disk failure processing method according to claim 1 is characterized in that: The performing storage hard disk abnormality detection on the target storage pool specifically includes: Performing storage hard disk anomaly detection on the target storage pool according to a preset detection frequency; If the detection result is that the storage hard disk is abnormal, sending an abnormality notification to all servers of applications that deploy application virtual machines in the target storage pool specifically includes: If the detection results for N consecutive times are that the storage hard disk is abnormal, an abnormality notification is sent to all servers of the application that deploys the application virtual machine in the target storage pool.

4. The distributed storage hard disk failure processing method according to claim 1, characterized in that: The performing storage hard disk abnormality detection on the target storage pool includes: Simultaneously sending an anomaly detection request to multiple storage hard disks in the target storage pool, and obtaining a response result of each storage hard disk to the anomaly detection request; Determine whether each storage hard disk is abnormal according to the response result; If at least one storage hard disk is abnormal, the detection result is determined to be that the storage hard disk is abnormal.

5. The method for handling a distributed storage hard disk failure according to claim 4, characterized in that: The determining whether each storage hard disk is abnormal according to the response result includes: If the response result is no response, an error is returned, or the response delay is greater than a preset threshold, it is determined that the storage hard disk is abnormal.

6. The distributed storage hard disk failure processing method according to claim 4 is characterized in that: The simultaneously sending an anomaly detection request to multiple storage hard disks in the target storage pool, and obtaining a response result of each storage hard disk to the anomaly detection request, specifically includes: Sending an abnormality detection request to a data read / write module of the target storage pool, so that the data read / write module simultaneously sends the abnormality detection request to multiple storage hard disks in the target storage pool; A response result of each storage hard disk to the abnormality detection request is obtained from the data reading and writing module.

7. The distributed storage hard disk failure processing method according to claim 4 is characterized in that: The abnormality detection request includes: a write IO request or a read IO request.

8. A distributed storage hard disk fault processing device, characterized in that: include: A hard disk anomaly detection unit, used to perform storage hard disk anomaly detection on the target storage pool through a hard disk anomaly detection device corresponding to the target storage pool in the distributed storage cluster, wherein the distributed storage cluster is composed of multiple storage pools, one storage pool is composed of multiple storage servers, one storage server has multiple storage hard disks, and each storage pool in the distributed storage cluster is provided with a corresponding hard disk anomaly detection device; an abnormality notification sending unit, for sending an abnormality notification to all servers of applications that deploy application virtual machines in the target storage pool if the detection result is that the storage hard disk is abnormal, so that the servers of each application isolate the application virtual machines deployed in the target storage pool according to the abnormality notification, wherein the application achieves high-availability deployment by deploying application virtual machines on multiple storage pools, and automatically creates a scheduling virtual machine outside the distributed storage cluster when the distributed storage cluster is built and enabled; a faulty hard disk determining unit, configured to determine a faulty hard disk in the target storage pool after the server of each application isolates the application virtual machine of the application deployed in the target storage pool, wherein if a storage hard disk in the target storage pool detects M abnormalities within a preset time, the storage hard disk is determined to be a faulty hard disk; A hard disk failure notification sending unit, used to send a hard disk failure notification to a management node of a distributed storage cluster according to the failed hard disk, so that the management node isolates the failed hard disk in the target storage pool according to the hard disk failure notification; If the detection result is that the storage hard disk is abnormal, an abnormal notification is sent to all servers of applications that deploy application virtual machines in the target storage pool, so that the servers of each application isolate the application virtual machines deployed in the target storage pool according to the abnormal notification, including: If the detection result is a storage hard disk abnormality, a notification is sent to the scheduling virtual machine through the hard disk abnormality detection device corresponding to the target storage pool, and the notification includes the identifier of the target storage pool; the scheduling virtual machine sends an abnormality notification to all servers of applications that deploy application virtual machines in the target storage pool based on the identifier of the target storage pool and the correspondence between the stored storage pool and the application virtual machines, so that the servers of each application isolate the application virtual machines deployed in the target storage pool according to the abnormality notification.

9. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the method according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed in a computer processor, the computer program implements the method according to any one of claims 1 to 7.

Images