Disk fault diffusion processing method and device and medium
By combining the health detection of faulty disks in distributed storage systems and disk expansion functions, OSD service downtime and data loss caused by disk failure spread are solved, and the stable operation of the storage system and data security are achieved.
Patent Information
- Application Number
- CN202411197002.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-29
- Publication Date
- 2024-11-22
AI Technical Summary
In distributed storage systems, disk failure diffusion problems lead to problems with multiple disks at the same time. In severe cases, it may cause the OSD service downtime of the entire node and even cause the loss of customer data. It is difficult for the existing technology to effectively solve such failure diffusion problems. .
By regularly performing health detection on the fault disk detected by the slow disk detection function, we will determine whether it is a real fault disk or a normal disk caused by the failure diffusion. If it is a fault diffusion disk, the disk expansion function of the storage system will be re-replaced. It is added to the disk topology and restarts the OSD service.
It effectively solves the OSD service downtime and data loss caused by failure spread, ensures the stability and data security of the storage system, and reduces operation and maintenance costs.
Smart Images

Figure CN119003233A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of distributed storage systems, and in particular to a disk failure diffusion processing method, device and medium. Background Art
[0002] Disks are the main storage media of computers and can be divided into two types: hard disks and floppy disks. As for hard disks, the two most widely used types are solid-state drives (SSDs) and mechanical hard disks (HDDs). Among them, HDD devices are prone to bad blocks, head damage and other faults due to their physical structure and working principle after vibration or long-term use. This will cause their read and write performance to drop significantly or become completely incapable of reading and writing, affecting the normal operation of their corresponding object storage daemon (OSD) services.
[0003] To address this issue, disk manufacturers have developed a slow disk detection function in related technologies. By detecting whether there is an abnormality in the input / output (IO) of a disk, it is determined whether the disk has read and write abnormalities, thereby promptly stopping the OSD service corresponding to the failed hard disk and removing it from the cluster.
[0004] However, the above solution is mainly aimed at the problem of single disk failure, which can effectively solve the problem of abnormal reading and writing of a single disk affecting the normal service of the storage system. However, due to the working mechanism of the board connected to the disk, the failure of a single disk may cause the board to restart or the IO of the failed disk to be blocked for a long time, thereby affecting the IO operation of other normal disks in the queue. At this time, these disks may also be mistakenly kicked out of the storage system, forming a fault spread. Fault spread will not only cause problems with multiple disks at the same time, but in severe cases it will also cause the OSD service of the entire node to crash, and even cause the loss of customer data.
[0005] Therefore, technicians in this field are in urgent need of a disk fault diffusion processing method to solve the fault diffusion problem that may occur during the current slow disk detection. Summary of the invention
[0006] The purpose of the present invention is to provide a disk fault diffusion processing method, device and medium, which are used to solve the fault diffusion problem that may occur when performing slow disk detection.
[0007] In order to solve the above technical problems, the present invention provides a disk failure diffusion processing method, comprising:
[0008] Scanning the faulty slow disk in the storage system according to a preset cycle; wherein the faulty slow disk is a disk marked as a faulty disk by a slow disk detection function and the object storage daemon service is stopped;
[0009] Performing a health check on the faulty slow disk, and marking the faulty slow disk that passes the health check as a fault diffusion disk;
[0010] The fault diffusion disk is added back to the disk topology through the disk expansion function of the storage system, and the object storage daemon service of the fault diffusion disk is started.
[0011] In a possible embodiment, performing a health check on the faulty slow disk and marking the faulty slow disk that passes the health check as a fault diffusion disk includes:
[0012] For any of the faulty slow disks, read the self-test information and hardware fault log of the current faulty slow disk; wherein the self-test information is information output by the self-monitoring, analysis and reporting technology command line tool;
[0013] Determine whether the current faulty slow disk is a faulty disk according to the self-check information and the hardware fault log;
[0014] If not, the current faulty slow disk is marked as the fault diffusion disk.
[0015] In a possible embodiment, before marking the current faulty slow disk as the fault diffusion disk, the method further includes:
[0016] Obtaining a slow disk detection timestamp of the current faulty slow disk; wherein the slow disk detection timestamp is a timestamp generated based on the slow disk detection moment when the slow disk detection function detects the faulty slow disk;
[0017] According to the slow disk detection timestamp of the current faulty slow disk, determining whether the current faulty slow disk satisfies the requirement that the slow disk detection function marks the faulty disk as a faulty disk because the number of times the faulty disk occurs exceeds a preset time period;
[0018] If not, the current faulty slow disk is marked as the fault diffusion disk.
[0019] In a possible embodiment, after determining whether the current faulty slow disk is a faulty disk according to the self-check information and the hardware fault log, the method further includes:
[0020] If yes, mark the current faulty slow disk as the faulty bad disk;
[0021] After determining, according to the slow disk detection timestamp of the current faulty slow disk, whether the current faulty slow disk satisfies the condition that the slow disk is marked as a faulty disk by the slow disk detection function for more than a preset number of times within a preset time period, the method further includes:
[0022] If the conditions are met, the current faulty slow disk is marked as the faulty bad disk;
[0023] The method further comprises:
[0024] The failed disk is removed from the disk topology, and the object storage daemon service corresponding to the failed disk is stopped.
[0025] In a possible embodiment, it further includes:
[0026] Create a log file as a slow disk checkout log;
[0027] The log information items in the slow disk detection log include: a faulty slow disk mark, a slow disk detection timestamp, and a faulty bad disk mark;
[0028] The faulty slow disk mark and the faulty bad disk mark include two states: a set state and an initial state; when the faulty slow disk mark and the faulty bad disk mark are both in the initial state, the corresponding disk is a normal disk; when the faulty slow disk mark is in the set state and the faulty bad disk mark is in the initial state, the corresponding disk is the faulty slow disk; when the faulty bad disk mark is in the set state, the corresponding disk is the faulty bad disk;
[0029] When the slow disk detection function detects a certain disk as the faulty slow disk, it also sets the faulty slow disk mark corresponding to the disk in the slow disk detection log; and writes the correspondingly generated slow disk detection timestamp into the slow disk detection log;
[0030] Then the step of obtaining the slow disk detection timestamp of the currently faulty slow disk includes:
[0031] Reading all the slow disk detection timestamps corresponding to the currently faulty slow disk from the slow disk detection log;
[0032] After marking the current faulty slow disk as the faulty bad disk, the method further includes:
[0033] Setting the faulty disk mark corresponding to the currently faulty slow disk in the slow disk detection log;
[0034] After marking the current faulty slow disk as the fault diffusion disk, the method further includes:
[0035] The faulty slow disk mark corresponding to the current faulty slow disk in the slow disk detection log is initialized.
[0036] In a possible embodiment, it further includes:
[0037] Processing concurrent access requests to the slow disk checkout log by a log writing module;
[0038] The log writing module is a functional module pre-written by a script tool, and the log writing module is deployed with a file lock mechanism.
[0039] In a possible embodiment, after marking the current faulty slow disk as the faulty bad disk, the method further includes:
[0040] Execute an alarm action and return fault information; wherein the fault information includes disk information of the faulty disk that triggers the alarm action this time;
[0041] After receiving the fault log export instruction, the fault log is exported; wherein the fault log includes the disk information of all the faulty disks generated from the last time the fault log was exported to the current time and the time when the faulty disks were detected.
[0042] In order to solve the above technical problems, the present invention further provides a disk fault diffusion processing device, comprising:
[0043] A slow disk scanning module is used to scan a faulty slow disk in the storage system according to a preset period; wherein the faulty slow disk is a disk marked as a faulty disk by a slow disk detection function and the object storage daemon service is stopped;
[0044] A health detection module, used to perform health detection on the faulty slow disk, and mark the faulty slow disk that passes the health detection as a fault diffusion disk;
[0045] The fault recovery module is used to add the fault diffusion disk back to the disk topology through the disk expansion function of the storage system, and start the object storage daemon service of the fault diffusion disk.
[0046] In order to solve the above technical problems, the present invention further provides a disk fault diffusion processing device, comprising:
[0047] Memory for storing computer programs;
[0048] The processor is used to implement the steps of the disk failure diffusion processing method as described above when executing the computer program.
[0049] To solve the above technical problem, the present invention further provides a non-volatile storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the disk failure diffusion processing method described above are implemented.
[0050] The present invention provides a method for processing disk fault diffusion, which periodically performs health checks on the faulty disk (i.e., the faulty slow disk) detected by the slow disk detection function to determine whether it is a truly abnormal faulty disk or a normal disk (i.e., the faulty diffusion disk) caused by the spread of faults from other faulty disks connected to the board. If a normal disk is removed from the disk topology of the storage system due to fault diffusion, the disk is re-added to the disk topology through the disk expansion function of the storage system, and the OSD service of the disk is restarted to restore its normal operation to provide storage functions. This method can effectively solve the fault diffusion problem that occurs when the traditional slow disk detection function detects a single disk fault, and can restore the disk that was in normal status but had its OSD service terminated and removed from the storage system due to fault diffusion to work, thereby avoiding serious problems such as OSD service downtime and customer data loss caused by fault diffusion.
[0051] The disk failure diffusion processing device and non-volatile storage medium provided by the present invention correspond to the above method and have the same effect as above. BRIEF DESCRIPTION OF THE DRAWINGS
[0052] In order to more clearly illustrate the embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0053] FIG1 is a flow chart of a disk failure diffusion processing method provided by an embodiment of the present invention;
[0054] FIG2 is a flow chart of a disk health detection method provided by an embodiment of the present invention;
[0055] FIG3 is a structural diagram of a disk failure diffusion processing device provided by an embodiment of the present invention;
[0056] FIG. 4 is a structural diagram of another disk failure diffusion processing device provided by an embodiment of the present invention. DETAILED DESCRIPTION
[0057] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0058] The core of the present invention is to provide a disk fault diffusion processing method, device and medium.
[0059] In order to enable those skilled in the art to better understand the solution of the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings and specific implementation methods.
[0060] At present, in distributed storage systems, solid-state drives (SSDs) and hard disk drives (HDDs) are usually used as data storage devices. Since solid-state drives and hard disk drives have their own advantages in performance and cost, in the current storage environment, solid-state drives are generally used as cache disks and hard disk drives are used as data disks to achieve a balance between low cost and high performance.
[0061] However, due to their physical structure and working principle, mechanical hard disk devices are prone to bad blocks, head damage and other faults after vibration or long-term use, which can cause their read and write performance to drop significantly or become completely inoperable, thus affecting the normal operation of their corresponding object storage daemon (OSD) services. To address this, the slow disk detection function came into being. The slow disk detection function determines whether a disk is faulty by detecting whether the input / output (IO) of a single disk is abnormal, and promptly stops the OSD service corresponding to the faulty disk and removes it from the cluster.
[0062] The above solution can effectively solve the problem of single disk failure in most cases and ensure the normal service of the storage system. However, due to the working mechanism of the board connected to the disk, the failure of a single disk may cause the board to restart or the IO of the failed disk to be blocked for a long time, thereby affecting the IO operations of other normal disks in the queue, causing these normal disks to be mistakenly kicked out of the storage system, forming a fault spread. Fault spread will not only cause problems on multiple disks at the same time, but in severe cases it will also cause the OSD service of the entire node to crash, and even cause the loss of customer data. In this case, the process of system recovery and data reconstruction is time-consuming and labor-intensive. Suppliers usually need to invest a lot of operation and maintenance resources to restore data and screen disks that were mistakenly kicked out, resulting in a significant increase in operation and maintenance costs. In addition, frequent failures and recovery operations may also lead to a decline in the overall performance of the system, further affecting user experience and system reliability.
[0063] In order to solve the above-mentioned series of problems caused by fault diffusion, the present invention provides a disk fault diffusion processing method, as shown in FIG1 , comprising:
[0064] S11: Scanning the faulty slow disks in the storage system according to a preset period.
[0065] The faulty slow disk is a disk that is marked as a faulty disk by the slow disk detection function and for which the object storage daemon service is stopped.
[0066] S12: Perform a health check on the faulty slow disk, and mark the faulty slow disk that passes the health check as a fault diffusion disk.
[0067] S13: Use the disk expansion function of the storage system to add the fault diffusion disk back to the disk topology and start the object storage daemon service of the fault diffusion disk.
[0068] As for the specific implementation of step S11, it can be known from the above that there is currently a slow disk detection function that can determine whether the disk function is abnormal by detecting whether the disk IO is abnormal. And in order to eliminate the impact of the faulty disk on the storage system, the slow disk detection function will remove the detected faulty disk from the disk topology (map) and stop the corresponding OSD service. Based on this, the faulty slow disk in step S11 can be obtained by scanning all the disks in the storage system that are marked as faulty disks by the slow disk detection function and the OSD service is stopped by the slow disk detection function. It should be noted that the faulty slow disk scanned in step S11 includes a faulty disk, and may also include a normal disk that is erroneously detected due to the spread of the fault. It is not completely equivalent to a "faulty disk" and needs to be further processed through subsequent steps. So to reflect this point, the disk scanned in this step is called a faulty slow disk.
[0069] For step S12, a health check is performed on all the faulty slow disks detected in step S11 to determine whether there are normal disks that are misdetected due to the spread of the fault. Specifically, there are currently a variety of disk health status detection or fault detection schemes, such as a hardware detection scheme that analyzes the hardware parameters of the disk, or a software detection scheme that performs a comprehensive analysis of the disk information to determine the health status of the disk, and this embodiment does not limit this. By selecting an appropriate disk health status detection or fault detection scheme according to actual needs, it can be determined whether the faulty slow disk is indeed a faulty disk. However, it should be noted that the health check performed in step S12 should not include the slow disk detection function. The slow disk detection function cannot identify the false detection problem caused by the fault diffusion problem, but the information output by the slow disk detection function can be further analyzed to achieve a health check based on software detection.
[0070] In addition, for the faulty slow disk that passes the health check in step S12, that is, the normal disk that is misdetected by the slow disk detection function due to the fault diffusion problem, subsequent function recovery is required to solve the problem caused by the fault diffusion. In order to distinguish this normal disk that is misdetected by the slow disk detection function from other disks, this method calls this type of disk a fault diffusion disk, which is a normal disk that is misdetected due to the fault diffusion problem.
[0071] Step S13 is a step of restoring the function of the fault diffusion disk detected in step S12. As can be seen from the above, after the slow disk detection function detects the faulty disk, in order to prevent the faulty disk from affecting the storage function of the storage system, the OSD service corresponding to the faulty disk will be stopped and the faulty disk will be removed from the disk topology. In view of this, this step performs functional restoration on the faulty diffusion disk that is misdetected due to fault diffusion. Specifically, functional restoration includes two steps, one is to add the faulty diffusion disk back to the disk topology, and the other is to restore the OSD service corresponding to the faulty diffusion disk. The first step can be implemented through the add disk command or process in the original disk expansion function of the storage system, and this embodiment will not be repeated here. The second step can be implemented through the OSD CLI (command line) command, and the OSD service of the faulty diffusion disk can be pulled up again to realize the restoration of the OSD service corresponding to the faulty diffusion disk, which will also be not repeated in this embodiment.
[0072] It should be noted that it is not difficult to see that this method is a fault diffusion processing solution based on the existing slow disk detection function, which is aimed at the fault diffusion problem that may occur when the slow disk detection function implements disk fault detection. The above steps of this method and the original slow disk detection function are two independent processes, so when implemented through processes, they are also implemented through independent processes. As for the process implementation of this method, the purpose can be achieved by writing a script. According to the architecture of the currently commonly used storage system, it can be specifically written in Python (a computer programming language) script.
[0073] It should also be noted that in the above step S11, it is explained that the present method is to periodically process the fault diffusion problem. This is adapted to the fact that the slow disk detection function is a continuous process during the operation of the storage system, and a faulty disk may be detected at any time. Therefore, in order to enable the present method to automatically cover the fault diffusion problem processing in the entire operation cycle of the storage system, the present method periodically scans the faulty slow disk and performs subsequent steps to achieve fault diffusion processing. The implementation of the periodic triggering of the present method can be achieved through the existing scheduled task mechanism in the current storage system, for example, by crontab (a program instruction that can be used to timed trigger scripts, that is, a Python script that can be timed to trigger the execution of the above method).
[0074] It is not difficult to see from the above that the disk fault diffusion processing method provided by the present invention performs a health check on all disks detected as faults by the slow disk detection function, and determines whether there are normal disks in the faulty slow disks according to the health check results. If yes, it means that this part of the faulty slow disks is a faulty diffusion disk that is mistakenly detected by the fault diffusion problem. For the fault diffusion disk, this method adds the fault diffusion disk back to the disk topology through the disk adding process of the storage system disk expansion function, and restarts its corresponding OSD service to restore its storage function, thereby effectively solving the problems caused by the fault diffusion. Based on this, the method can timely stop the data security risks caused by the fault diffusion and ensure the security of customer data. It can also timely screen out non-faulty disks and continue to add them to the storage system service, reduce the scope of the fault, and ensure that the system can continue to provide services. In addition, based on the method to implement the processing of disks that are misdetected due to fault diffusion, the frequency and workload of operation and maintenance personnel to handle faulty disks on site can be reduced, thereby effectively reducing operation and maintenance costs.
[0075] On the other hand, in the above embodiment, there is no restriction on the health detection method used in step S12, and any other solution with disk health status monitoring or fault detection other than the slow disk detection function can be used according to actual needs. Based on this, this embodiment provides a specific solution for health detection, as shown in FIG2, step S12 specifically includes:
[0076] S121: For any faulty slow disk, read the self-test information and hardware fault log of the current faulty slow disk.
[0077] The self-check information is smartctl information, and the smartctl information is information output by the SelfMonitoring Analysis and Reporting Technology (SMART) command line tool (smartctl).
[0078] S122: Determine whether the current slow disk is a faulty disk based on the self-check information and the hardware fault log. If not, go to step S123.
[0079] S123: Mark the current faulty slow disk as a fault diffusion disk.
[0080] Specifically, the disk health status detection scheme provided in this embodiment is equivalent to the hardware detection scheme mentioned in the above embodiment. By reading the smartctl information and hardware fault log of the disk, which can directly reflect the health status of the disk, it is possible to determine whether the disk is faulty. Specifically, both the smartctl information and the hardware fault log contain information that can reflect whether the disk has failed. If the disk fails, the smartctl information and the hardware fault log will contain corresponding fault codes, based on which it is possible to analyze whether the disk has failed and what kind of failure has occurred on the disk. Since the smartctl tool and the hardware fault log are both existing hardware fault self-check mechanisms, they will not be described in detail in this embodiment.
[0081] Furthermore, based on the hardware detection solution for health detection of a faulty slow disk provided in the above embodiment, this embodiment further provides a software detection solution, as shown in FIG. 2 , before the above step S123, the following steps are further included:
[0082] S124: Obtain the slow disk detection timestamp of the current faulty slow disk.
[0083] The slow disk detection timestamp is a timestamp generated based on the slow disk detection moment when the slow disk detection function detects a faulty slow disk.
[0084] S125: Based on the slow disk detection timestamp of the current faulty slow disk, determine whether the current faulty slow disk satisfies the requirement of being marked as a faulty disk by the slow disk detection function within a preset time length and exceeding a preset number of times; if not, go to step S123.
[0085] It should be noted that the specific values of the preset duration and the preset number of times mentioned in the above steps are not limited in this embodiment and can be freely set according to actual needs. However, in a possible implementation scenario, the preset duration can be one hour and the preset number of times can be three times.
[0086] Specifically, this embodiment uses the disk fault detection of the slow disk detection function to implement further software detection based on hardware detection. When the hardware detects that the disk is normal, if the software detection solution provided by this embodiment finds that the disk has been issued a fault report by the slow disk detection function for more than a preset number of times within a preset time, then the disk is also considered abnormal and the function recovery of the subsequent step S13 is not performed on it.
[0087] In summary, the software detection scheme for health detection of faulty slow disks provided in this embodiment is based on the hardware detection scheme provided in the above embodiment. When the hardware detection scheme determines that the faulty slow disk is a normal disk, further judgment is performed. If the number of failures reported by the slow disk detection function of the disk exceeds the preset number within the preset time length, then even if the hardware detection of the disk is normal, it is considered not to be a normal disk and cannot be determined as a fault diffusion disk. In other words, a disk determined to be abnormal through hardware detection can be directly identified as abnormal without the need for subsequent software detection. A disk determined to be normal through hardware detection still needs to be tested by software, and only when the software detection is also normal can the disk be considered to be a normal disk, that is, a fault diffusion disk that needs to be functionally restored in step S13.
[0088] Based on this, this embodiment is a software detection solution provided on the basis of the disk health hardware detection solution provided in the above embodiment. It can identify abnormal disks that cannot be identified by some hardware detection solutions, thereby further improving the accuracy of disk status detection, further ensuring that the fault diffusion disk recovered by this method is indeed a normal disk, and improving the reliability of this method.
[0089] Further, according to the disk health hardware detection scheme and software detection scheme provided in the above embodiments, it can be known that the above two schemes have two branches, corresponding to the branches of passing and failing the detection of the faulty slow disk being detected. The above embodiments mainly illustrate the scheme from the aspect of how to determine the fault diffusion disk. Corresponding to the branch of passing the detection, this embodiment improves the hardware detection scheme and software detection scheme provided above, and also provides a possible implementation plan for the branch of failing the detection. As shown in FIG2, after the above step S122, if the yes branch is entered, the method further includes:
[0090] S126: Mark the current faulty slow disk as a faulty bad disk.
[0091] In addition, after step S125, if the current faulty slow disk meets the condition of being marked as a faulty disk by the slow disk detection function within a preset time and for more than a preset number of times, that is, entering the yes branch, it also goes to step S126.
[0092] After step S126, the method further includes:
[0093] S127: Remove the faulty disk from the disk topology, and stop the object storage daemon service corresponding to the faulty disk.
[0094] That is, based on the implementation scheme provided in this embodiment and the hardware detection scheme and software detection scheme provided in the above embodiments, there is a complete disk health detection scheme, as shown in Figure 2: for the fault diffusion disk scanned in step S11, first pass the hardware detection corresponding to step S121 and step S122; if the hardware detection fails, the faulty slow disk is indeed a faulty disk, enters the branch corresponding to step S127, and is marked as a faulty bad disk; if the hardware detection passes, further pass the software detection corresponding to step S124 and step S125; if the software detection fails, then even if the hardware detection passes, it is considered that the faulty slow disk is indeed a faulty disk, enters the branch corresponding to step S127, and is marked as a faulty bad disk; if the software detection also passes, it is determined that the faulty slow disk is indeed a normal disk, enters the branch corresponding to step S123, and is marked as a faulty diffusion disk.
[0095] It should be further explained that the removal of the faulty disk corresponding to step S127 is considered to be "permanently" removed in the disk fault detection process of the present method flow, so as to avoid the impact of the faulty disk on the OSD services of other normal disks in the storage system. However, this will result in a smaller capacity of the storage system. Based on this, this embodiment also provides a possible implementation plan, and the method further includes:
[0096] S21: Use the disk expansion function of the storage system to add the repaired faulty disk back to the disk topology, and start the object storage daemon service of the repaired faulty disk.
[0097] That is, the faulty disk that is determined and "permanently" removed from the disk topology by the above method in this embodiment can be added back to the disk topology through the solution of this embodiment after the operation and maintenance personnel perform troubleshooting or replace the disk to restore normal operation, so as to avoid the reduction of storage capacity due to the removal of the faulty disk.
[0098] In summary, the above embodiments respectively provide a hardware detection solution, a software detection solution, and a complete disk health status detection solution for determining a fault diffusion disk, and provide different implementation plans for determining a fault diffusion disk in step S12 in a progressive manner. Moreover, the solution provided by each embodiment is more complete and reliable than the previous embodiment. In practical applications, a suitable solution can be selected as a specific implementation plan for detecting a fault diffusion disk according to actual detection needs, and this embodiment does not limit this.
[0099] On the other hand, in the above embodiment, different status marks for disks such as faulty slow disks and faulty broken disks are mentioned. In addition, in the above software detection scheme, it is also necessary to obtain the information of the slow disk detection timestamp. Therefore, this embodiment provides a specific management scheme for the faulty slow disk mark, the faulty broken disk mark, and the slow disk detection timestamp. This method also includes:
[0100] S22: Create a log file as a slow disk checkout log.
[0101] Among them, the log information items in the slow disk detection log include: faulty slow disk mark, slow disk detection timestamp and faulty bad disk mark. The faulty slow disk mark and the faulty bad disk mark include two states: set state and initial state; when the faulty slow disk mark and the faulty bad disk mark are both in the initial state, the corresponding disk is a normal disk; when the faulty slow disk mark is in the set state and the faulty bad disk mark is in the initial state, the corresponding disk is a faulty slow disk; when the faulty bad disk mark is in the set state, the corresponding disk is a faulty bad disk.
[0102] Correspondingly, when the original slow disk detection function in the storage system detects that a certain disk is a faulty slow disk, the slow disk detection function is also required to set the faulty slow disk mark corresponding to the disk in the slow disk detection log. This action can be achieved by simply modifying the code of the slow disk detection function. When the slow disk detection function detects a disk failure, an additional step of accessing the slow disk detection log and setting the faulty slow disk mark is performed. In addition, the slow disk detection function is also required to access the slow disk detection log and write the timestamp corresponding to the slow disk detection time, that is, the slow disk detection timestamp, into it when a faulty disk is detected. The implementation of this step can also be achieved by simply modifying the code of the slow disk detection function.
[0103] Based on this, the above step S124: obtaining the slow disk detection timestamp of the current faulty slow disk is specifically:
[0104] From the slow disk detection log, read all slow disk detection timestamps corresponding to the current faulty slow disk.
[0105] And, after step S126: marking the current faulty slow disk as a faulty bad disk, the method further includes:
[0106] S128: Setting the faulty disk flag corresponding to the current faulty slow disk in the slow disk detection log.
[0107] Furthermore, after step S123: marking the current faulty slow disk as a faulty bad disk, the method further includes:
[0108] S129: Initialize the faulty slow disk mark corresponding to the current faulty slow disk in the slow disk detection log.
[0109] That is, the various steps and processes involved in the above embodiments can be implemented based on the maintenance of the slow disk detection log. The slow disk detection log includes at least but is not limited to three information items: a faulty slow disk mark, a slow disk detection timestamp, and a faulty bad disk mark. It should be noted that these three information items correspond to the disks one by one, that is, each disk corresponds to a set of faulty slow disk marks, slow disk detection timestamps, and faulty bad disk marks in the slow disk detection log. Among them, the faulty slow disk mark is used to indicate whether the disk is a faulty slow disk; the faulty bad disk mark is used to indicate whether the disk is a faulty bad disk; if the disk is neither a faulty slow disk nor a faulty bad disk, it means that the disk is a normal disk or a repaired faulty diffusion disk; the slow disk detection timestamp is used to record the moment when the corresponding disk is reported as faulty by the slow disk detection function, which has specific applications in software detection solutions.
[0110] It should also be noted that this embodiment does not limit the format of the slow disk detection log. The types and functions of various information items contained in the slow disk detection log have been introduced in detail in the above embodiment. You can freely choose a suitable implementation plan according to actual needs, and this embodiment does not limit this. However, this embodiment provides a possible implementation plan. For example, the above slow disk detection log is a JSON format log file.
[0111] JSON: JavaScript Object Notation, JavaScript object notation. JavaScript is a lightweight, interpreted or just-in-time compiled programming language with function priority.
[0112] In this embodiment, by creating and maintaining a slow disk detection log, the above method implements the management of the disk's faulty slow disk mark, faulty bad disk mark, and slow disk detection timestamp to ensure data consistency. Based on the maintenance of the slow disk detection log, the above method can be implemented conveniently and reliably.
[0113] Furthermore, based on the above embodiments, it can be known that the maintenance of the slow disk detection log is the key to ensuring the effective and reliable implementation of the disk fault diffusion processing method provided by the present invention. It should also be further explained that a possible implementation scheme is provided in the above embodiments, and the implementation of the corresponding steps of the present method can be implemented by other processes independent of the original slow disk detection function, such as by triggering a pre-written Python script through crontab. For the convenience of distinction, the process for implementing the slow disk detection function is the first process, and the process for implementing the remaining steps of the present method is the second process.
[0114] Then for the three information items in the slow disk detection log, namely, the faulty slow disk mark, the slow disk detection timestamp and the faulty bad disk mark. Among them, the faulty slow disk mark is written by the first process and needs to be read by the second process (for example, the faulty slow disk scan in step S11) or may be written (for example, the faulty slow disk mark initialization in step S129). The slow disk detection timestamp is written by the first process and needs to be read by the second process (for example, the software detection solution corresponding to step S124). The faulty bad disk mark needs to be written by the second process (for example, the faulty bad disk mark corresponding to step S126).
[0115] It can be seen that the slow disk checkout log needs to receive access from at least two different processes. At the same time, the first process and the second process are two independent processes that continue to run in the operation cycle of the storage system, so there may be concurrent access to the slow disk checkout log. It should be particularly noted that this example is only a possible implementation plan, and the remaining steps of the method can also be implemented through more processes, which is not limited in this embodiment. However, no matter how many processes are used to implement this method, combined with the implementation of the original slow disk checkout function, at least two processes are required, that is, there are concurrent access requests to the slow disk checkout log.
[0116] In view of this, this embodiment provides a further implementation scheme, wherein the method further comprises:
[0117] S23: Processing concurrent access requests to the slow disk checkout logs through the log writing module.
[0118] Among them, the log writing module is a functional module pre-written by a script tool, and the log writing module is deployed with a file lock mechanism.
[0119] That is, this embodiment provides a separately created log writing module to implement the maintenance of the slow disk checkout log, and the access requests of the first process and the second process to the slow disk checkout log are all processed by the log writing module. In addition, the log writing module in this embodiment is a functional module deployed with a file lock mechanism, which can effectively handle the concurrent access problems that may be encountered in the slow disk checkout log, avoid the data inconsistency problems that may occur when multiple processes access the data of the slow disk checkout log at the same time, and ensure the consistency and integrity of the data.
[0120] On the other hand, after the above embodiment determines the faulty disk, the method further provides a possible implementation scheme, and the method further includes:
[0121] S141: Execute alarm action and return fault information.
[0122] The fault information includes the disk information of the faulty disk that triggers the alarm action. It should also be noted that the disk information may also include the smartctl information and hardware fault log obtained in the hardware detection solution of the above embodiment.
[0123] S142: After receiving the fault log export instruction, export the fault log.
[0124] The fault log includes disk information and faulty disk detection time of all faulty disks generated from the last fault log export time to the current time. Further, the faulty disk detection time may be the corresponding slow disk detection timestamp stored in the slow disk detection log in the above embodiment.
[0125] That is, this embodiment provides a fault alarm mechanism, which is a scheme for issuing an alarm and exporting relevant information on the basis of the original slow disk detection function after processing the fault diffusion problem and finally determining the faulty bad disk. It should be noted that this embodiment does not limit the specific implementation form of the alarm action, and different alarm methods can be selected according to actual needs. In general, the specific type of alarm action is related to the type of alarm device. For example, when the alarm device is a light-emitting diode (LED) or other device that can emit a simple light signal, the alarm action can be that the LED light is always on, always off, always on with a specific color, flashing in a specific way, etc. Similarly, if the alarm device is a buzzer or other device that can emit a sound signal, the alarm action can be to emit a specific sound signal.
[0126] At the same time, with regard to the specific content of the fault log, the above-mentioned fault log containing the disk information of all faulty disks generated from the last fault log export time to the current time and the detection time of the faulty disk is only a basic requirement. In other application scenarios, other information can be added to the fault log according to different needs, and this embodiment does not impose any restrictions on this.
[0127] A fault alarm mechanism provided in this embodiment is based on the above-mentioned method for dealing with the fault diffusion problem, and alarms the faulty bad disk detected after eliminating the influence of the fault diffusion problem, thereby achieving more accurate disk failure alarm, so that operation and maintenance personnel can timely grasp the disk operation status in the storage system and take corresponding measures in time, which can improve the stability and reliability of the storage system operation and help ensure the security of customer data.
[0128] In the above embodiment, a disk fault diffusion processing method is described in detail, and the present invention also provides a corresponding embodiment of a disk fault diffusion processing device. It should be noted that the present invention describes the embodiment of the device part from two perspectives, one is based on the functional module perspective, and the other is based on the hardware perspective.
[0129] From the perspective of functional modules, as shown in FIG3 , this embodiment provides a disk failure diffusion processing device, including:
[0130] The slow disk scanning module 11 is used to scan the faulty slow disk in the storage system according to a preset period; wherein the faulty slow disk is a disk marked as a faulty disk by the slow disk detection function and the object storage daemon service is stopped;
[0131] A health detection module 12 is used to perform health detection on the faulty slow disk and mark the faulty slow disk that passes the health detection as a fault diffusion disk;
[0132] The fault recovery module 13 is used to add the fault diffusion disk back to the disk topology through the disk expansion function of the storage system, and start the object storage daemon service of the fault diffusion disk.
[0133] Since the embodiments of the apparatus part correspond to the embodiments of the method part, please refer to the description of the embodiments of the method part for the embodiments of the apparatus part, which will not be repeated here.
[0134] FIG. 4 is a structural diagram of a disk failure diffusion processing device provided by another embodiment of the present invention. As shown in FIG. 4 , a disk failure diffusion processing device includes: a memory 20 for storing a computer program;
[0135] The processor 21 is used to implement the steps of a disk failure diffusion processing method in the above embodiment when executing a computer program.
[0136] The disk failure diffusion processing device provided in this embodiment may include but is not limited to a mobile terminal, a personal computer, a workstation, etc.
[0137] Among them, the processor 21 may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 21 can be implemented in at least one hardware form of a digital signal processor (DSP), a field-programmable gate array (FPGA), and a programmable logic array (PLA). The processor 21 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the awake state, also known as a central processing unit (CPU); the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor 21 may be integrated with a graphics processing unit (GPU), which is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 21 may also include an artificial intelligence (AI) processor, which is used to process computing operations related to machine learning.
[0138] The memory 20 may include one or more computer-readable storage media, which may be non-transitory. The memory 20 may also include a high-speed random access memory, and a non-volatile memory, such as one or more disk storage devices, flash memory storage devices. In this embodiment, the memory 20 is at least used to store the following computer program 201, wherein, after the computer program is loaded and executed by the processor 21, it can implement the relevant steps of a disk fault diffusion processing method disclosed in any of the aforementioned embodiments. In addition, the resources stored in the memory 20 may also include an operating system 202 and data 203, etc., and the storage method may be temporary storage or permanent storage. Among them, the operating system 202 may include Windows, Unix, Linux, etc. The data 203 may include, but is not limited to, a disk fault diffusion processing method, etc.
[0139] In some embodiments, a disk failure diffusion processing device may further include a display screen 22 , an input / output interface 23 , a communication interface 24 , a power supply 25 , and a communication bus 26 .
[0140] Those skilled in the art will appreciate that the structure shown in FIG. 4 does not constitute a limitation on a disk failure diffusion processing device, and may include more or fewer components than those shown in the figure.
[0141] An embodiment of the present invention provides a disk fault diffusion processing device, which includes a memory and a processor. When the processor executes a program stored in the memory, it can implement the following method: a disk fault diffusion processing method.
[0142] In addition to the embodiments of the disk failure diffusion processing method and device provided in the above embodiments, the present invention also provides an embodiment corresponding to a computer program product. A computer program product includes a computer program / instruction, and when the computer program / instruction is executed by a processor, the steps of the disk failure diffusion processing method described in any of the above embodiments can be implemented.
[0143] Since the embodiments of the computer program product part correspond to the embodiments of the method part, please refer to the description of the embodiments of the method part for the embodiments of the computer program product part, which will not be repeated here.
[0144] Finally, the present invention also provides an embodiment corresponding to a non-volatile storage medium. The non-volatile storage medium stores a computer program, and when the computer program is executed by a processor, the steps recorded in the above method embodiment are implemented.
[0145] It is understandable that if the method in the above embodiment is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a non-volatile storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium to execute all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc. Various media that can store program codes.
[0146] The above is a detailed introduction to a disk fault diffusion processing method, device and medium provided by the present invention. The various embodiments in the specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same and similar parts between the various embodiments can be referenced to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part description. It should be pointed out that for ordinary technicians in this technical field, without departing from the principle of the present invention, the present invention can also be improved and modified in a number of ways, and these improvements and modifications also fall within the scope of protection of the present invention.
[0147] It should also be noted that, in this specification, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "comprise", "include" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the statement "comprises a ..." does not exclude the presence of other identical elements in the process, method, article or device including the element.
Claims
1. A disk failure diffusion processing method, characterized in that: include: Scanning the faulty slow disk in the storage system according to a preset cycle; wherein the faulty slow disk is a disk marked as a faulty disk by a slow disk detection function and the object storage daemon service is stopped; Performing a health check on the faulty slow disk, and marking the faulty slow disk that passes the health check as a fault diffusion disk; The fault diffusion disk is added back to the disk topology through the disk expansion function of the storage system, and the object storage daemon service of the fault diffusion disk is started.
2. The disk failure diffusion processing method according to claim 1, characterized in that: Performing a health check on the faulty slow disk and marking the faulty slow disk that passes the health check as a fault diffusion disk includes: For any of the faulty slow disks, read the self-test information and hardware fault log of the current faulty slow disk; wherein the self-test information is information output by the self-monitoring, analysis and reporting technology command line tool; Determine whether the current faulty slow disk is a faulty disk according to the self-check information and the hardware fault log; If not, the current faulty slow disk is marked as the fault diffusion disk.
3. The disk failure diffusion processing method according to claim 2, characterized in that: Before marking the current faulty slow disk as the fault diffusion disk, the method further includes: Obtaining a slow disk detection timestamp of the current faulty slow disk; wherein the slow disk detection timestamp is a timestamp generated based on the slow disk detection moment when the slow disk detection function detects the faulty slow disk; According to the slow disk detection timestamp of the current faulty slow disk, determining whether the current faulty slow disk satisfies the requirement that the slow disk detection function marks the faulty disk as a faulty disk because the number of times the faulty disk occurs exceeds a preset time period; If not, the current faulty slow disk is marked as the fault diffusion disk.
4. The disk failure diffusion processing method according to claim 3, characterized in that: After determining whether the current faulty slow disk is a faulty disk according to the self-check information and the hardware fault log, the method further includes: If yes, mark the current faulty slow disk as the faulty bad disk; After determining, according to the slow disk detection timestamp of the current faulty slow disk, whether the current faulty slow disk satisfies the condition that the slow disk is marked as a faulty disk by the slow disk detection function for more than a preset number of times within a preset time period, the method further includes: If the conditions are met, the current faulty slow disk is marked as the faulty bad disk; The method further comprises: The failed disk is removed from the disk topology, and the object storage daemon service corresponding to the failed disk is stopped.
5. The disk failure diffusion processing method according to claim 4, characterized in that: Also includes: Create a log file as a slow disk checkout log; The log information items in the slow disk detection log include: a faulty slow disk mark, a slow disk detection timestamp, and a faulty bad disk mark; The faulty slow disk mark and the faulty bad disk mark include two states: a set state and an initial state; when the faulty slow disk mark and the faulty bad disk mark are both in the initial state, the corresponding disk is a normal disk; when the faulty slow disk mark is in the set state and the faulty bad disk mark is in the initial state, the corresponding disk is the faulty slow disk; when the faulty bad disk mark is in the set state, the corresponding disk is the faulty bad disk; When the slow disk detection function detects a certain disk as the faulty slow disk, it also sets the faulty slow disk mark corresponding to the disk in the slow disk detection log; and writes the correspondingly generated slow disk detection timestamp into the slow disk detection log; Then the step of obtaining the slow disk detection timestamp of the currently faulty slow disk includes: Reading all the slow disk detection timestamps corresponding to the currently faulty slow disk from the slow disk detection log; After marking the current faulty slow disk as the faulty bad disk, the method further includes: Setting the faulty disk mark corresponding to the currently faulty slow disk in the slow disk detection log; After marking the current faulty slow disk as the fault diffusion disk, the method further includes: The faulty slow disk mark corresponding to the current faulty slow disk in the slow disk detection log is initialized.
6. The disk failure diffusion processing method according to claim 5, characterized in that: Also includes: Processing concurrent access requests to the slow disk checkout log by a log writing module; The log writing module is a functional module pre-written by a script tool, and the log writing module is deployed with a file lock mechanism.
7. The disk failure diffusion processing method according to any one of claims 1 to 6, characterized in that: After marking the current faulty slow disk as the faulty bad disk, the method further includes: Execute an alarm action and return fault information; wherein the fault information includes disk information of the faulty disk that triggers the alarm action this time; After receiving the fault log export instruction, the fault log is exported; wherein the fault log includes the disk information of all the faulty disks generated from the last time the fault log was exported to the current time and the time when the faulty disks were detected.
8. A disk failure diffusion processing device, characterized in that: include: A slow disk scanning module is used to scan a faulty slow disk in the storage system according to a preset period; wherein the faulty slow disk is a disk marked as a faulty disk by a slow disk detection function and the object storage daemon service is stopped; A health detection module, used to perform health detection on the faulty slow disk, and mark the faulty slow disk that passes the health detection as a fault diffusion disk; The fault recovery module is used to add the fault diffusion disk back to the disk topology through the disk expansion function of the storage system, and start the object storage daemon service of the fault diffusion disk.
9. A disk failure diffusion processing device, characterized in that: include: Memory for storing computer programs; A processor, configured to implement the steps of the disk failure diffusion processing method as claimed in any one of claims 1 to 7 when executing the computer program.
10. A non-volatile storage medium, characterized in that: The non-volatile storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the disk failure diffusion processing method according to any one of claims 1 to 7 are implemented.
Citation Information
Cited By
Distributed storage collaborative optimization method and device, electronic equipment and storage medium
CN121387208A
