Disk failure detection method and device

By monitoring the data write delay and setting the electromagnetic disk within the preset time, the data write blocking problem caused by array caine waiting for repair is solved, the faulty disk is isolated, and the server reliability and business processing capabilities are improved.

CN113608915BActive Publication Date: 2024-12-03新华三技术有限公司成都分公司
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202111015989.X
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2021-08-31
Publication Date
2024-12-03
Estimated Expiration
2041-08-31

AI Technical Summary

Technical Problem

In the server, when the disk fails, the array card will block data writing due to waiting for repair, causing all connected disks to be placed in a blocked state, reducing the reliability of the server.

Method used

By monitoring the data write delay, if the preset time exceeds the preset time, use BMC to control the power off the disk, and perform disk failure detection and isolation within the repair startup time. Select a backup OSD for business migration to avoid data write blockage.

Benefits of technology

Effectively isolate the failed disk, avoid data write blocking, and improve server reliability and business processing capabilities.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN113608915B_ABST
    Figure CN113608915B_ABST
Patent Text Reader

Abstract

The present specification provides a disk fault detection method and device, which relate to the field of communication technology. A disk fault detection method, applied to a server, includes: sending a write request to a first disk, monitoring the data writing of the array card to the first disk; if it is monitored that the data writing to the first disk exceeds a first preset time, the BMC in the server controls the first disk to be powered off, wherein the first preset time is less than the repair startup time set in the array card. The above method can improve the reliability of the server.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of communication technology, and in particular to a disk failure detection method and device. Background Art

[0002] With the popularization of Internet applications, the application of servers as service providers has also increased, and servers can also set up a large number of disks as storage devices to save massive amounts of data. As the number of disks increases, multiple disks can be managed through array cards. At this time, the array card can take over the disk to cache data first and provide feedback to the program running in the server, and then write the cached data to the actual disk for storage.

[0003] When a failure occurs when writing data to a disk, in order to avoid the loss of subsequent data written, the program running in the server can block the data written to the disk and wait for the repair of the faulty disk. When blocking, upon receiving a blocking message, the array card connected to the faulty disk will be affected by the blocking message and stop writing data to the disk connected to itself. However, an array card may be connected to multiple disks, which will cause all the connected disks to be placed in a blocked state, reducing the server's ability to process business, thereby reducing the server's reliability. Summary of the invention

[0004] In order to overcome the problems existing in the related art, this specification provides a disk failure detection method and device.

[0005] In conjunction with the first aspect of the implementation manner of this specification, the present application provides a disk failure detection method, applied to a server, comprising:

[0006] Sending a write request to the first disk and monitoring the array card writing data to the first disk;

[0007] If it is monitored that the data writing to the first disk exceeds a first preset time, the first disk is powered off by controlling the BMC in the server, wherein the first preset time is less than the repair startup time set in the array card.

[0008] Optionally, after monitoring the data written by the array card to the first disk, the method further includes:

[0009] If it is monitored that the data writing to the first disk exceeds a second preset time length, the object storage device OSD corresponding to the first disk is restarted, wherein the second preset time length is smaller than the first preset time length.

[0010] Furthermore, if it is detected that the data writing to the first disk exceeds the second preset time, the method further includes:

[0011] Select an OSD corresponding to the second disk from multiple OSDs created in the server;

[0012] Migrate the service carried by the OSD corresponding to the first disk to the OSD corresponding to the second disk.

[0013] Optionally, after controlling the first disk to be powered off through the BMC in the server, the method further includes:

[0014] Accumulating the number of times the first disk is powered off;

[0015] If the number of times the first disk is powered off exceeds a threshold, the OSD associated with the first disk is cleared.

[0016] Furthermore, after the first disk is powered off by controlling the BMC in the server, the method further includes:

[0017] Controlling the power-on of the first disk through the BMC in the server;

[0018] Performing startup detection on the first disk;

[0019] If it is determined within the third preset time period that the first disk fails the startup detection, the OSD associated with the first disk is cleared.

[0020] In conjunction with the second aspect of the implementation manner of this specification, the present application provides a disk failure detection device, applied to a server, comprising:

[0021] A write unit, used for sending a write request to the first disk;

[0022] A monitoring unit, used for monitoring data written by the array card to the first disk;

[0023] The control unit is used to control the first disk to be powered off through the BMC in the server if it is detected that the data writing to the first disk exceeds a first preset time, wherein the first preset time is less than the repair startup time set in the array card.

[0024] Optionally, the device further includes:

[0025] The restart unit is used to restart the OSD corresponding to the first disk if it is monitored that the data writing to the first disk exceeds a second preset time length, wherein the second preset time length is smaller than the first preset time length.

[0026] Furthermore, the device further comprises:

[0027] A selection unit, configured to select an OSD corresponding to the second disk from a plurality of OSDs created in the server;

[0028] The migration unit is used to migrate the service carried by the OSD corresponding to the first disk to the OSD corresponding to the second disk.

[0029] Optionally, the device further includes:

[0030] A counting unit, used for accumulating the number of times the first disk is powered off;

[0031] The clearing unit is configured to clear the OSD associated with the first disk if the number of times the first disk is powered off exceeds a threshold.

[0032] Furthermore, the control unit is further used to control the first disk to be powered on through the BMC in the server;

[0033] The device further comprises:

[0034] A detection unit, performing a startup detection on the first disk;

[0035] The clearing unit is further configured to clear the OSD associated with the first disk if it is determined within a third preset time period that the first disk fails the startup detection.

[0036] The technical solutions provided by the embodiments of this specification may have the following beneficial effects:

[0037] In the implementation manner of the present specification, by monitoring the data writing of the disk and powering off the disk within a first preset time that is less than the repair startup time, it is avoided that when a disk in the server fails, the data writing blocking set by the array card is activated, resulting in a decrease in the service processing capacity of the server, so that the faulty disk can be isolated, thereby improving the reliability of the server.

[0038] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present specification. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the specification and, together with the description, serve to explain the principles of the specification.

[0040] Figure 1 is a flow chart of a disk failure detection method involved in the present application;

[0041] Figure 2 It is a structural diagram of a server involved in this application;

[0042] Figure 3 It is a structural schematic diagram of a disk failure detection device involved in the present application. DETAILED DESCRIPTION

[0043] Here, exemplary embodiments are described in detail, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this specification.

[0044] In order to overcome the problems existing in the related art, this specification provides a disk failure detection method and device.

[0045] The present application provides a disk failure detection method, which is applied to a server, such as Figure 1 As shown, including:

[0046] S100: Send a write request to a first disk, and monitor data written by the array card to the first disk.

[0047] like Figure 2 As shown, a mainboard and a backplane plugged into the mainboard can be provided in a server. Devices such as a processor are provided on the mainboard, and a BMC (Baseboard Management Controller) and a CPLD (Complex Programmable Logic Device) are provided on the backplane. Multiple disks can also be plugged into the backplane through a data interface, and an array card can be plugged into the backplane through connectors such as PCIE (Peripheral Component Interconnect Express), and the array card can also be called a RAID (Redundant Array of Independent Disks) card. An I / O (Input / Output) interface is provided between the array card and the multiple disks managed by the array card, and data can be written into the disk for storage through the I / O interface. Through the array card, data writing and data reading management for multiple disks of the server can be realized.

[0048] During the operation of the server, the processor can send a write request to the disk to write the data to be written to the target disk, which is subsequently referred to as the first disk. At this time, the write request will be first received by the array card, and the disk identifier of the first disk and the data to be written and other information will be obtained from the write request. After receiving the write request, the array card can first cache the data to be written carried in the write request.

[0049] In order to determine the status of the disk, the processor can monitor the I / O interface between the array card and the disk it manages. For example, it can monitor the delay of data writing between the array card and the disk through the I / O interface. When the array card writes data to the disk through the I / O interface, it will send the data to be written through the I / O interface. After receiving the data, the target disk will respond to the array card to inform the array card of the result of the data writing. The delay can be the time between the array card writing data to the disk and monitoring the feedback response of the I / O interface. If the array card initiates a data write to the disk and does not receive feedback from the disk within a period of time, it can be considered that the data write initiated this time has timed out, and the data write timeout can indicate that the target disk required to be written by the write request may have a fault. The monitoring performed here can be implemented based on threads, processes, etc. created by the processor.

[0050] In order to prevent disk failures from being unrecognized, a repair mode can be set in the current array card to start when the array card determines that a disk has not responded for a long time. For the repair mode, the array card can set the repair startup time. When a disk does not respond for a period of time exceeding the repair startup time, the array card will no longer receive write requests to the disk it manages and start repair. Since multiple disks managed by the array card have stopped external services, the server's business processing capabilities will be reduced, thereby affecting the reliability of the server.

[0051] S101: If it is monitored that data writing to a first disk exceeds a first preset time, control the first disk to be powered off through a BMC in a server.

[0052] In the server, BMC can control the devices through CPLD. For example, CPLD contains registers, which include registers for controlling the voltage of the disk, called power supply registers. For each disk in the server, a power supply register is set. When the register is set to 1, the backplane can be controlled to output 12 volts and 5 volts to the disk. When the register is set to 0, the backplane is controlled to stop supplying power to the disk. Since a disk is supplied with 12 volts and 5 volts at the same time, the disk may be in different working modes under different power supply voltages. Therefore, in order to take a disk offline, it is necessary to disconnect the 12 volt and 5 volt power supplies respectively.

[0053] The processor monitors the delay of the I / O interface. When the delay exceeds the first preset time length (for example, set to 90 seconds), it can be said that after writing data to the first disk, the first disk failed to feedback a response to the array card during a period of time exceeding the first preset time length. Since the response from the first disk has not been received for a long time, it can be considered that the first disk is likely to have failed, and continued waiting may trigger the repair mode of the array card. At this time, the processor can send instructions to the BMC based on the IPMI (Intelligent Platform Management Interface) protocol through the connection with the BMC, thereby controlling the CPLD to set the power supply register corresponding to the first disk to 0 to achieve power off.

[0054] It should be noted that the first preset duration is less than the repair startup duration set in the array card. The repair startup duration set in a general array card is 180 seconds. Therefore, setting the first preset duration to 90 seconds can satisfy the requirement of controlling the first disk to be powered off before the repair mode is started, thereby avoiding the data writing being blocked due to the array card starting the repair mode. Moreover, after controlling the first disk to be powered off, the first disk can continue to be powered on, and a write request can be sent to the first disk again, thereby testing whether the first disk can be restored by restarting.

[0055] By monitoring the data writing of the disk and powering off the disk within a first preset time that is less than the repair startup time, it is avoided that when a disk in the server fails, the data writing blocking set by the array card is activated, resulting in all disks managed by the array card being blocked. The failed disk can be isolated, thereby improving the reliability of the server.

[0056] In the server, the processor can run multiple OSDs (Object-based Storage Devices) for each disk. The OSD can be understood as a virtual device bound to the disk. At the software level, an OSD can be understood as an independent process, which is allocated with corresponding processing resources and process identifiers. When the server needs to write data, access to a disk can be achieved by calling the OSD. For an OSD, the I / O interface between the array card and the disk can be monitored. In specific implementation, a monitoring point can be set in the timeout function in the kernel state. The monitoring point can be understood as monitoring the state of the timeout function. When the timer times out, the timeout function is triggered and the timer timeout state is passed to other functions that have registered the timeout function. At this time, the monitoring point is triggered, thereby passing the timer timeout state to the OSD, so that the OSD determines the delay timeout of the I / O. During the operation of the server, the OSD may fail and the write request cannot be sent to the corresponding disk. At this time, the physical disk may not fail. In order to avoid the problem that the write request cannot be sent due to the OSD failure.

[0057] Optionally, after step S101, monitoring the data written by the array card to the first disk, the method further includes:

[0058] S102: If it is monitored that the data writing to the first disk exceeds a second preset time length, restart the OSD corresponding to the first disk.

[0059] like Figure 2 As shown, in the server, 4 OSDs (i.e., OSD1 to OSD4) can be created, and 4 disks are also deployed in the server, where disk 1 corresponds to OSD1, disk 2 corresponds to OSD2, disk 3 corresponds to OSD3, and disk 4 corresponds to OSD4. A second preset duration can be set in the server, and the second preset duration is less than the first preset duration. For example, the second preset duration can be set to 30 seconds. After the processor sends a write request, it is received by the array card, and data is sent to the disk through the array card, and the response of the first disk (disk 1 corresponding to OSD1 can be regarded as the first disk) is monitored.

[0060] The processor will start timing. The timing mentioned here needs to record two kinds of time. The first one is based on the first preset time, that is, it has been monitoring whether disk 1 has responded to data writing. When the first preset time is reached, the power on and power off of disk 1 are realized by whether power is supplied. The second one is based on the second preset time, that is, the OSD1 corresponding to disk 1 in operation is intermittently monitored. During the monitoring process, the timing first reaches the second preset time. At this time, the processor can clear the OSD1 corresponding to disk 1 based on software, and restart the OSD1 to be associated with disk 1. The OSD restart mentioned here refers to resetting the status bit in OSD once. For example, when it is determined that the I / O interface has timed out, OSD first switches OSD to offline state through the status bit, and then switches OSD to online state through the status bit. In this process, the configuration related to OSD and the data that have been generated will not be cleared, and only the state will be switched.

[0061] For the two timings, when the processor sends a write request and transmits data to the disk through the array card, two timers (i.e., timer 1 and timer 2) are started respectively for timing. After the monitoring reaches the second preset time, timer 2 is cleared and timing is started again when OSD1 is recreated. Timer 1 continues to timing without processing after reaching the second preset time.

[0062] At this time, the processor can resend a write request to disk 1 through the restarted OSD1 to determine whether the failure of disk 1 is caused by OSD1. If the failure to receive a response to the write request is caused by OSD1, restarting OSD1 may solve this problem, thereby restoring the server to a normal working state. In addition, the failure to receive a response from disk 1 may also be caused by a failure of disk 1 itself. In this case, the repair cannot be completed by restarting OSD1. Then, since the processor is still continuing to time the first preset time, when the timing time of timer 1 reaches the first preset time, disk 1 can be restarted by cutting off power through the BMC to determine whether disk 1 is unable to complete data writing due to its own failure.

[0063] In this way, the server can determine the problem of data writing failure from the software level and the hardware level respectively, further improving the reliability of the server.

[0064] In addition, since restarting the OSD and reissuing a write request to the disk corresponding to the OSD will inevitably cause a delay in data writing, in order to avoid data writing delays caused by OSD failures. Further, if it is monitored that the data writing to the first disk exceeds the second preset time, it also includes:

[0065] S103: Select an OSD corresponding to the second disk from multiple OSDs created in the server.

[0066] S104: Migrate the service carried by the OSD corresponding to the first disk to the OSD corresponding to the second disk.

[0067] In such Figure 2 In the scenario shown, since multiple disks and corresponding OSDs are deployed in the server, when the processor determines based on timer 2 that the second preset time has been exceeded, an OSD is selected from OSD2 to OSD4 as an alternative, for example, OSD2 is selected as an alternative, and OSD2 is associated with disk 2. At this time, the server can implement service migration based on its own existing functions, that is, migrate the service (including write request) carried by OSD1 to OSD2 for implementation. At this time, OSD2 can execute the service already carried by OSD1 based on service migration. For example, the processor can send a write request to disk 2 through OSD2 to write data to the normally working disk 2 for storage.

[0068] In this way, data writing delays caused by OSD failures can be avoided, improving the timeliness of server data writing.

[0069] After the disk is powered off, it can be powered on automatically or manually by the staff. During this process, if it is determined that the failure to write is caused by the disk itself, then the same disk may be repeatedly powered on and off, and the disk may not be able to be restored. Instead, the repeated power-on and power-off of the disk will increase the server detection requirements and cause instability. In order to avoid the above problems, after the first disk is powered off through the BMC in the server, it is optionally further included:

[0070] S105: Accumulate the number of times the first disk is powered off.

[0071] S106: If the number of times the first disk is powered off exceeds a threshold, clear the OSD associated with the first disk.

[0072] A counter can be set for each disk in the server, and the counter is used to record the number of times a disk is powered off. That is to say, each time the timer reaches the first preset duration, the counter is accumulated accordingly. In addition, for the counter set for a disk, a threshold value can also be set for the disk, such as 5 times. If the number of power-offs exceeds 5 times, it is considered that the disk is indeed faulty and will not be powered on again. Since management software can generally be run in the server to display and manage information such as the status of devices in the server, when a disk failure is determined, in addition to clearing the OSD corresponding to the disk, an alarm can be issued in the management software so that the staff can identify the faulty disk and perform manual maintenance as soon as possible, further improving the reliability of the server operation.

[0073] Furthermore, after step S101, in which the BMC in the server controls the first disk to be powered off, the method further includes:

[0074] S107 . Control the first disk to power on through the BMC in the server.

[0075] S108: Perform a startup test on the first disk.

[0076] S109: If it is determined within the third preset time period that the first disk fails the startup detection, clearing the OSD associated with the first disk.

[0077] After the processor performs a power-off operation through the BMC, it can continue to control the BMC to power on the disk again, that is, to reset the corresponding register in the CPLD to 1. At this time, the repowered disk may temporarily resume working state, but since the disk has already failed, it cannot be ruled out that the disk itself may be in an unreliable state. Then, after the disk is powered on, a more stringent disk in-place detection is started and the detection is continued, for example, the cycle of the in-place detection is 5 seconds.

[0078] In addition, a third preset duration for in-place detection can be set in the server, which can be set to a longer time, such as 20 minutes. Within these 20 minutes, if the disk is always in a normal state, it can be considered that the disk has recovered and can continue to be used. If the disk fails again within these 20 minutes, for reliability considerations, an alarm can be issued to inform the staff that the disk is in an unreliable state, so that the staff can decide whether to continue to use the disk.

[0079] Alternatively, the server may directly power off the disk and generate an alarm to notify the staff to directly replace the faulty disk.

[0080] Correspondingly, the present application provides a disk failure detection device, which is applied to a server, such as Figure 3 As shown, including:

[0081] A write unit, used for sending a write request to the first disk;

[0082] A monitoring unit, used for monitoring data written by the array card to the first disk;

[0083] The control unit is used to control the first disk to be powered off through the BMC in the server if it is detected that the data writing to the first disk exceeds a first preset time, wherein the first preset time is less than the repair startup time set in the array card.

[0084] Optionally, the device further includes:

[0085] The restart unit is used to restart the OSD corresponding to the first disk if it is monitored that the data writing to the first disk exceeds a second preset time length, wherein the second preset time length is smaller than the first preset time length.

[0086] Furthermore, the device further comprises:

[0087] A selection unit, configured to select an OSD corresponding to the second disk from a plurality of OSDs created in the server;

[0088] The migration unit is used to migrate the service carried by the OSD corresponding to the first disk to the OSD corresponding to the second disk.

[0089] Optionally, the device further includes:

[0090] A counting unit, used for accumulating the number of times the first disk is powered off;

[0091] The clearing unit is configured to clear the OSD associated with the first disk if the number of times the first disk is powered off exceeds a threshold.

[0092] Furthermore, the control unit is further used to control the first disk to be powered on through the BMC in the server;

[0093] The device further comprises:

[0094] A detection unit, performing a startup detection on the first disk;

[0095] The clearing unit is further configured to clear the OSD associated with the first disk if it is determined within a third preset time period that the first disk fails the startup detection.

[0096] The technical solutions provided by the embodiments of this specification may have the following beneficial effects:

[0097] In the implementation manner of the present specification, by monitoring the data writing of the disk and powering off the disk within a first preset time that is less than the repair startup time, it is avoided that when a disk in the server fails, the data writing blocking set by the array card is activated, resulting in all disks managed by the array card being blocked, so that the failed disk can be isolated, thereby improving the reliability of the server.

[0098] It should be understood that the present description is not limited to the exact construction that has been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof.

[0099] The above description is only a preferred implementation of this specification and is not intended to limit this specification. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of this specification should be included in the scope of protection of this specification.

Claims

1. A disk failure detection method, characterized in that: Applicable to servers, including: Sending a write request to the first disk, and monitoring the array card writing data to the first disk; If it is monitored that the data writing to the first disk exceeds a second preset time length, restarting the object storage device OSD corresponding to the first disk, wherein the second preset time length is less than the first preset time length; If it is monitored that the data writing to the first disk exceeds a first preset time length, the first disk is powered off by a baseboard management controller BMC in the server, wherein the first preset time length is less than the repair startup time length set in the array card; If it is detected that the data writing to the first disk exceeds a second preset time, the method further includes: Selecting an OSD corresponding to the second disk from a plurality of OSDs created in the server; Migrate the service carried by the OSD corresponding to the first disk to the OSD corresponding to the second disk, wherein the service includes the write request.

2. The method according to claim 1, characterized in that: After the first disk is powered off by controlling the BMC in the server, the method further includes: Accumulating the number of times the first disk is powered off; If the number of times the first disk is powered off exceeds a threshold, the OSD associated with the first disk is cleared.

3. The method according to claim 2, characterized in that After the first disk is powered off by controlling the BMC in the server, the method further includes: Controlling the first disk to power on by using the BMC in the server; Performing startup detection on the first disk; If it is determined within the third preset time period that the first disk fails the startup detection, the OSD associated with the first disk is cleared.

4. A disk failure detection device, characterized in that: Applicable to servers, including: A write unit, used for sending a write request to the first disk; a restart unit, configured to restart the OSD corresponding to the first disk if it is detected that the data writing to the first disk exceeds a second preset time length, wherein the second preset time length is less than the first preset time length; A monitoring unit, used for monitoring data written by the array card to the first disk; a control unit, configured to control the first disk to be powered off through the BMC in the server if it is detected that the data writing to the first disk exceeds a first preset time, wherein the first preset time is less than the repair startup time set in the array card; The device further comprises: A selection unit, configured to select an OSD corresponding to the second disk from a plurality of OSDs created in the server; A migration unit is used to migrate the service carried by the OSD corresponding to the first disk to the OSD corresponding to the second disk, wherein the service includes the write request.

5. The device according to claim 4, characterized in that Also includes: A counting unit, used for accumulating the number of times the first disk is powered off; The clearing unit is configured to clear the OSD associated with the first disk if the number of times the first disk is powered off exceeds a threshold.

6. The device according to claim 5, characterized in that The control unit is further used to control the first disk to power on through the BMC in the server; The device further comprises: A detection unit, performing a startup detection on the first disk; The clearing unit is further configured to clear the OSD associated with the first disk if it is determined within a third preset time period that the first disk fails the startup detection.