Monitoring system, monitoring method, program, and fault-tolerant server
The monitoring system addresses inaccurate hard disk diagnostics by desynchronizing subsystems for detailed diagnosis, ensuring accurate identification of abnormalities and maintaining system performance.
Patent Information
- Application Number
- JP2020141150
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- Filing Date
- 2020-08-24
- Publication Date
- 2024-11-19
- Estimated Expiration
- 2040-08-24
AI Technical Summary
Existing hard disk diagnostic methods in RAID systems often inaccurately identify normal disks as faulty due to temporary load factors, leading to unnecessary replacements and impacting business processing performance.
A monitoring system that issues IO requests to synchronized subsystems, determines abnormal tendencies based on response time differences, and desynchronizes the subsystems to allow independent operation for detailed diagnosis without affecting business processing.
The system effectively identifies actual abnormalities in hard disks without disrupting business operations, preventing unnecessary replacements and maintaining system performance.
Smart Images

Figure 0007585659000001 
Figure 0007585659000002 
Figure 0007585659000003
Abstract
Description
[Technical field]
[0001] The present disclosure relates to surveillance systems and the like. [Background technology]
[0002] One method for identifying hard disks that are not yet broken but have degraded performance among the hard disks that make up a RAID (Redundant Array of Inexpensive Disks) system is to measure the response time of the hard disk. With this method, a hard disk whose response time is equal to or exceeds a predetermined threshold is determined to have degraded performance. However, the response time exceeding the threshold may be caused by accidental factors such as temporary load. Therefore, even when there is actually no abnormality in the hard disk, it may be determined to be abnormal, leading to unnecessary disk replacement.
[0003] Patent Document 1 discloses a method for detecting storage devices whose disk load is below a threshold and whose response time is above a threshold in order to find storage devices in a potential failure state in a storage device. The method of Patent Document 1 determines that a storage device may be broken even if the response time exceeds the threshold due to an accidental factor other than the disk load.
[0004] After detecting an abnormal trend in a hard disk, a diagnosis of the hard disk may be performed to determine the abnormal state. For example, a method may be used in which similar measurements are performed multiple times, and only a specific hard disk is diagnosed as having an abnormality after confirming that the performance of the specific hard disk has decreased in all measurements. Furthermore, a method may be used in which average response times are compared, and a hard disk whose average response time exceeds a threshold value is diagnosed as having an abnormality.
[0005] Patent Document 2 discloses that when a disk error such as a timeout occurs, the disk is put into a provisional degenerate state and four types of disk diagnostic processing are performed. [Prior art documents] [Patent documents]
[0006] [Patent Document 1] JP 2019-036163 A [Patent Document 2] JP 2002-108573 A Summary of the Invention [Problem to be solved by the invention]
[0007] Since the hard disk diagnostic process and business processing are performed in parallel, performing the hard disk diagnostic process without disconnecting the hard disk may result in a decrease in the performance of the entire system, which may cause problems in business processing. Furthermore, repeating diagnostic input and output for a hard disk whose performance may be degraded and measuring the response time places an additional load on the system. In other words, the more detailed the hard disk diagnostic is performed, the more the load on the system will be, which may cause problems in business processing.
[0008] The diagnostic process in Patent Document 2 is performed in parallel with commands from the host computer, so it must be performed in parallel with business processing on the same disk array device. Therefore, the diagnostic method in Patent Document 2 may affect the performance of business processing due to the diagnostic process.
[0009] One object of the present disclosure is to provide a monitoring system for a storage device that avoids an impact on business processing. [Means for solving the problem]
[0010] The monitoring system of the present disclosure comprises a processing means for issuing IO (Input Output) requests to storage devices respectively included in a synchronized first and second subsystems, a determination means for determining an abnormal tendency of the storage device having a longer response time based on the difference in response times to the IO requests, and a synchronization control means for, when an abnormal tendency is determined, releasing the synchronization of the first and second subsystems and controlling the first and second subsystems so that they can operate independently, and the processing means performs a diagnostic process for determining whether or not the storage device is abnormal in the subsystem including the storage device determined to have an abnormal tendency.
[0011] The monitoring method of the present disclosure issues IO (Input Output) requests to storage devices respectively included in synchronized first and second subsystems, and determines an abnormal tendency of the storage device with a longer response time based on the difference in response times to the IO requests. When an abnormal tendency is determined, the synchronization of the first and second subsystems is released, and the first and second subsystems are controlled so as to be able to operate independently, and in the subsystem including the storage device determined to have an abnormal tendency, a diagnosis process is performed to determine whether or not there is an abnormality in the storage device.
[0012] The program of the present disclosure causes a computer to execute the following processes: issuing IO (Input Output) requests to storage devices respectively included in a synchronized first and second subsystem; determining an abnormal tendency of the storage device having a longer response time based on the difference in response times to the IO requests; when an abnormal tendency is determined, releasing the synchronization of the first and second subsystems and controlling the first and second subsystems so that they can operate independently; and diagnosing the presence or absence of an abnormality in the storage device in the subsystem including the storage device determined to have an abnormal tendency.
[0013] A fault-tolerant server according to the present disclosure comprises a first subsystem having a first storage device, a first CPU module, and a first fault-tolerant (FT) controller, and a second subsystem having a second storage device, a second CPU module, and a second FT controller, wherein the first and second storage devices each respond to an IO (Input Output) request issued from a monitoring system, and when an abnormality tendency of the first storage device is determined based on a difference in response time to the IO requests, the first FT controller controls desynchronization of the first and second subsystems, diagnoses the abnormality of the first storage device in the first subsystem, and performs other processing in the second subsystem. Effect of the Invention
[0014] According to the present disclosure, it is possible to provide a monitoring system for a storage device that avoids an impact on business processing. [Brief description of the drawings]
[0015] [Figure 1] FIG. 2 is a diagram illustrating a configuration of an FT server 1 according to the present disclosure. [Diagram 2] FIG. 1 shows the FT server 1 in a synchronized state. [Diagram 3] FIG. 1 shows the FT server 1 in a desynchronized state. [Figure 4A] 1 is a block diagram illustrating a configuration of a monitoring system 50 according to a first embodiment. [Figure 4B] 5 is a flowchart showing an example of the operation of the monitoring system 50 according to the first embodiment. [Diagram 5] 5 is a flowchart showing an abnormal trend detection process according to the first embodiment. [Figure 6] 11 is a flowchart showing the operations of the system 10 and the system 20 when releasing the synchronization. [Figure 7] 4 is a flowchart showing a diagnosis process according to the first embodiment. [Figure 8]FIG. 11 is a block diagram illustrating a configuration of a monitoring system 50 according to a second embodiment. [Figure 9] 10 is a flowchart showing an abnormal trend detection process according to the second embodiment. [Figure 10] 10 is a flowchart showing a diagnosis process according to the second embodiment. [Figure 11] 11 is a flowchart showing the operations of the system 10 and the system 20 when releasing the synchronization. [Figure 12] FIG. 2 is a block diagram showing an example of a hardware configuration of a monitoring system 50. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0016] In mission-critical systems, the continuity of service is required even when a fault occurs, so fault-tolerant technology is introduced. A fault-tolerant server (FT server) is known as a computer that employs fault-tolerant technology.
[0017] The hardware components that make up an FT server are duplicated. If a failure occurs in one of the hardware components, the part where the failure occurred is logically isolated. The part that is operating normally continues processing, so duplication improves fault tolerance.
[0018] The monitoring system according to the present disclosure can be used, for example, to monitor the hard disk of a FT server. Fig. 1 is a diagram showing the configuration of a FT server 1 according to the present disclosure. The FT server 1 according to the present disclosure includes a system 10 and a system 20 as two subsystems.
[0019] The system 10 includes a CPU module 11, an FT controller 12, and an IO module 13. The CPU module 11 includes a CPU (Central Processing Unit) and a memory. The IO module 13 includes various IO (Input Output) devices including a NIC (Network Interface Card) and a hard disk 14.
[0020] The system 20 includes a CPU module 21, an FT controller 22, and an IO module 23. The CPU module 21 includes a CPU and a memory. The IO module 23 includes IO devices including a NIC and a hard disk 24.
[0021] The CPU module 11 and the CPU module 21 are controlled by the FT controller 12 and the FT controller 22 so that they operate synchronously with the same clock. The IO devices of the IO module 13 and the IO module 23 are configured redundantly by software. The NICs are configured redundantly by using teaming technology, and the hard disks are configured redundantly by using mirroring technology.
[0022] Fig. 2 is a diagram showing the FT server 1 in a synchronized state. As shown in Fig. 2, when the FT server 1 is operating in a synchronized state, the CPU module 11 and CPU module 21 are controlled by the FT controller 12 and the FT controller 22 so that they operate synchronously with the same clock. An OS (Operating System) 30 runs on the FT server 1, and business processing is performed on the OS 30. If a hardware failure occurs in one of the systems, the part where the failure occurred is logically isolated, allowing the system that is operating normally to continue processing.
[0023] If the FT server 1 is operating in a synchronized state without any hardware failure, it is possible to split the FT server 1 into two systems that operate independently by intentionally temporarily disabling the synchronization of the FT server 1.
[0024] Fig. 3 is a diagram showing the FT server 1 in a state where synchronization is released. As shown in Fig. 3, in a state where synchronization is intentionally released, the CPU module 11 operates in the system 10, and IO processing corresponding to each IO device of the IO module 13 is performed. The OS 31 runs on the system 10. The CPU module 21 operates in the system 20, and IO processing corresponding to each IO device of the IO module 23 is performed. The OS 32 runs on the system 20.
[0025] [First embodiment] [composition] Fig. 4A is a block diagram illustrating an example of the configuration of a monitoring system 50 according to the first embodiment. The monitoring system 50 is connected, for example, by wire or wirelessly to the FT server 1 shown in Fig. 1, and monitors the hard disks 14 and 24 of the FT server 1. The monitoring system 50 includes a processing unit 51, a recording unit (not shown), a determination unit 53, and a synchronization control unit 54. The processing unit 51, the determination unit 53, and the synchronization control unit 54 are embodiments of the processing means, the determination means, and the synchronization control means, respectively, according to the present disclosure. The hard disk 14 and the hard disk 24 are embodiments of the storage device, respectively, according to the present disclosure.
[0026] The processing unit 51 issues IO requests to the storage devices included in the two synchronized subsystems (hereinafter, this may simply be referred to as issuing IOs or issuing IOs). Specifically, the processing unit 51 periodically issues IOs to, for example, the hard disks 14 and 24. The processing unit 51 issues IOs to measure the response time from sending an IO request to receiving a response to the IO request. Then, the processing unit 51 records the measured response time in the recording unit. Furthermore, the processing unit 51 performs a diagnostic process to diagnose the presence or absence of an abnormality in a subsystem including a storage device determined to have an abnormal tendency.
[0027] The recording unit is, for example, an auxiliary storage device such as a random access memory (RAM), etc. The recording unit may be realized by a storage device such as a hard disk.
[0028] The determination unit 53 determines whether the storage device with the longer response time has an abnormal tendency based on the difference in response times to IO requests. Specifically, the determination unit 53 monitors the response times of each hard disk recorded in the recording unit, for example, and determines whether the difference in response time is equal to or greater than a threshold. If the difference in response time exceeds the threshold, the determination unit 53 transmits to the synchronization control unit 54 a notification that the FT server 1 needs to be desynchronized, together with information on which hard disk has an abnormal tendency. Furthermore, the determination unit 53 performs a determination to determine whether the hard disk has an abnormality that reduces performance based on the results of the diagnostic processing of the hard disk.
[0029] When it is determined that one of the storage devices is prone to an abnormality, the synchronization control unit 54 releases the synchronization between the two subsystems, and performs control so that systems 10 and 20 can operate independently. Specifically, the synchronization control unit 54 controls the synchronization of the FT server 1, for example, based on a notification from the determination unit 53. When releasing the synchronization, the system on the hard disk side that is not prone to an abnormality (short response time) becomes the business continuation side, and the system on the hard disk side that is prone to an abnormality (long response time) becomes the hard disk diagnosis side, and releases the synchronization. Note that the control of the synchronization of the FT server 1 is performed via the FT controllers 12 and 22.
[0030] [Operation] FIG. 4B is a flowchart showing an example of the operation of the monitoring system 50 according to the first embodiment. First, the processing unit 51 issues an IO request to each of the storage devices included in the systems 10 and 20, which are two synchronized subsystems (step S101). The determination unit 53 determines whether the storage device with the longer response time is prone to an abnormality based on the difference in response time to the IO request (step S102). When the synchronization control unit 54 determines that the storage device is prone to an abnormality, it cancels the synchronization between the systems 10 and 20 and controls the two subsystems to be able to operate independently (step S103). The processing unit 51 diagnoses whether the storage device in the subsystem that includes the storage device determined to be prone to an abnormality has an abnormality (step S104).
[0031] A specific example of a processing flow according to the first embodiment will be described with reference to FIGS.
[0032] 5 is a flowchart showing the process (check C1) of detecting a hard disk having a tendency to be abnormal according to the first embodiment. First, the processing unit 51 checks whether the FT server 1 is operating in a synchronous state. If it is not operating in a synchronous state (step S1; NO), the processing unit 51 notifies the display unit (not shown) of an error indicating that the FT server 1 is not operating in a synchronous state (step S2). If it is operating in a synchronous state (step S1; YES), the processing unit 51 periodically issues IO to the hard disks 14 and 24 (step S3). The processing unit 51 measures the response time of each and records it in the recording unit (step S4).
[0033] Next, the determination unit 53 performs response time monitoring processing. Specifically, the determination unit 53 monitors whether the difference in response time between each hard disk exceeds a threshold value. In order to monitor the abnormal tendency of performance degradation of the hard disk, the determination unit 53 may, for example, calculate the difference in response time between each hard disk and monitor whether the difference exceeds a threshold value (M ms (ms: millisecond)). Also, the determination unit 53 may, for example, calculate the ratio of the response time of each hard disk and monitor whether the ratio exceeds a threshold value (N times) (1 < N). Other indicators may be used as long as the purpose of monitoring the performance degradation of the hard disk can be achieved. When the difference in response time exceeds the threshold value, the determination unit 53 detects that there is an abnormal tendency of performance degradation in one of the hard disks.
[0034] When the difference in response time does not exceed the threshold value (step S5; NO), the monitoring system 50 checks the synchronization state of the FT server 1 again, and the determination unit 53 waits for the next IO issue.
[0035] When the difference in response time exceeds the threshold value (step S5; YES), the determination unit 53 determines which hard disk has a longer response time. When the determination unit 53 detects an abnormal tendency of performance degradation, it notifies the synchronization control unit 54 that it is necessary to release the synchronization of the FT server 1. When the response time of the hard disk 24 is longer (step S6; YES), the synchronization control unit 54 releases the synchronization of the FT server 1 with the system 10 as the business continuation side and the system 20 as the hard disk diagnosis side (step S7). When the response time of the hard disk 14 is longer (step S6; NO), the synchronization control unit 54 releases the synchronization of the FT server 1 with the system 20 as the business continuation side and the system 10 as the hard disk diagnosis side (step S8).
[0036] FIG. 6 is a flowchart showing the operations of the system 10 and the system 20 when releasing the synchronization. In FIG. 6, the case where an abnormal tendency of performance degradation is detected in the hard disk 24 on the system 20 side is described as an example.
[0037] First, systems 10 and 20 of FT server 1 operate in a synchronized state (step S11). FT controller 12 instructs FT controller 22 to cancel synchronization based on control by synchronization control unit 54 (step S12). In response to this instruction, FT controller 22 cancels synchronization (step S16), and FT server 1 is divided into systems 10 and 20 which operate independently. System 10 continues operations (step S13), while system 20 stops operations (step S17). At this time, in order to continue operations on the system 10 side, the IP address that was used during the synchronous operation of FT server 1 continues to be used. On the system 20 side, a different IP address is temporarily assigned.
[0038] In the system 20, a hard disk diagnostic process A1 is executed for the hard disk 24 in which an abnormal tendency has been detected, in order to determine whether or not the hard disk is actually abnormal (step S18). If the result of the hard disk diagnostic process A1 is that the hard disk 24 is determined to be normal (step S19; NO), the system 20 notifies the system 10 via the network that the hard disk diagnostic process has ended (step S20). When the system 10 receives the notification (step S14), the FT controller 12 instructs the FT controller 22 to synchronize (step S15). The system 20 receives the synchronization instruction and performs synchronization based on the system 10 that has been continuing business processing (step S21). When the synchronization process is completed, the FT server 1 returns to a synchronized state (step S23).
[0039] If the hard disk diagnostic process A1 determines that the hard disk 24 is abnormal (step S19; YES), the system 20 notifies a display unit (not shown) of an error that the hard disk needs to be replaced (step S22).
[0040] 7 is a flowchart showing a diagnostic process (hard disk diagnostic process A1) according to the first embodiment. The hard disk diagnostic process A1 is performed on a hard disk 24 that has been determined to have an abnormal tendency of performance degradation in order to confirm whether the abnormal tendency has been detected due to an accidental factor or whether an abnormality of performance degradation has actually occurred.
[0041] The hard disk diagnostic process A1 can be performed independently of the system on the business continuation side by releasing the synchronization of the FT server 1, so it is possible to perform a detailed diagnosis without affecting business operations. In the first embodiment, a diagnostic method is shown in which an IO similar to check C1 is issued multiple times and judged based on the average response time, but the diagnostic method is not limited to this method. For example, it is also possible to perform a more detailed diagnosis such as a full read and full write test of the hard disk.
[0042] In the hard disk diagnostic process A1, first, the processing unit 51 issues an IO to the hard disk 24 (step S31), measures the response time, and records it in the recording unit (step S32). The monitoring system 50 repeats the processes of steps S31 and S32 until the measurement is completed a specified number of times (X times) (step S33; NO). After the measurement is completed a specified number of times, the monitoring system 50 proceeds to a judgment process by the judgment unit 53 (step S33; YES).
[0043] The determination unit 53 calculates an average response time of the hard disk 24 from the response times of the specified number of times (X times) of the hard disk 24 (step S34). The determination unit 53 also calculates an average response time of the hard disk 24 from the response times of the most recent X times of the hard disk 24 recorded in the recording unit (step S35).
[0044] The judgment unit 53 compares the average response time of the hard disk 14 and the hard disk 24. If the average response time of the hard disk 24 is longer and the difference exceeds a threshold (step S36; YES), the judgment unit 53 judges the hard disk 24 to be abnormal (step S37). If this condition is not met (step S36; NO), the judgment unit 53 judges the hard disk 24 to be normal. The threshold used here may be the same as the threshold used in check C1, but a threshold smaller than that of check C1 may be used to perform a more accurate diagnosis.
[0045] [effect] According to the monitoring system 50 of the first embodiment, it is possible to avoid the impact on business processing due to a decrease in performance of one of the storage devices included in each of the two synchronized subsystems. This is because, in the monitoring system 50, the processing unit 51 issues an IO request to each storage device, the determining unit 53 determines whether the storage device is prone to abnormality based on the difference in response time, and the synchronization control means releases the synchronization of the two subsystems. In addition, the subsystems from which synchronization has been released become capable of operating independently, and the processing unit 51 performs a diagnostic process for determining whether or not the storage device is abnormal in the subsystem including the storage device determined to be prone to abnormality.
[0046] According to the first embodiment, it is possible to avoid a performance drop in the entire FT server 1 and its impact on business processing due to a performance drop in one of the hard disks. This is because the monitoring system 50 measures the response time of IO requests from each hard disk, monitors for abnormal trends in performance drops, and if an abnormal trend is detected, releases synchronization of the FT server 1.
[0047] Furthermore, according to the first embodiment, by diagnosing a hard disk that shows signs of abnormality, it is possible to confirm whether or not an abnormality has actually occurred in the hard disk, thereby making it possible to avoid unnecessary replacement of the hard disk.
[0048] Furthermore, the business system can concentrate on business, and the hard disk diagnostic system can perform more detailed diagnosis of the hard disk without affecting business operations, because the FT server 1 is desynchronized and divided into systems 10 and 20 which operate independently.
[0049] [Second embodiment] In the first embodiment, a case has been described in which if an abnormal trend of performance degradation is detected in the hard disk at the check C1 stage, synchronization of the FT server 1 is cancelled. This is because priority is given to preventing degradation of hard disk performance from interfering with business processing (performance-first mode).
[0050] In the second embodiment, a case will be described in which synchronization is released while monitoring the load conditions of the systems 10 and 20, such as the CPU load and IO load. In the second embodiment, even if the monitoring system 50 detects an abnormal trend of performance degradation in the hard disk, if the system load is lower than the threshold and the impact on business processing is minor, the monitoring system 50 does not release synchronization of the FT server 1 and continues diagnosing the hard disk. By releasing synchronization of the FT server 1 only when the load exceeds the threshold, it is possible to prioritize maintaining the synchronized state of the FT server 1 (synchronization priority mode).
[0051] [composition] Fig. 8 is a block diagram illustrating a configuration of a monitoring system 50 according to a second embodiment. In Fig. 8, the monitoring system 50 is connected by wire or wirelessly to two subsystems, system 10 and system 20, which are capable of operating synchronously. Note that, regarding the monitoring system 50 according to the second embodiment, a description of the same configuration as that of the monitoring system 50 according to the first embodiment will be omitted. The monitoring system 50 according to the second embodiment differs from the monitoring system 50 according to the first embodiment in that it includes a recording unit 52 and further includes a load measuring unit 55.
[0052] The load measurement unit 55 measures the system load, such as the CPU load or the IO load. If the system load does not exceed a predetermined threshold when an abnormal trend is determined, the synchronization control unit 54 does not release the synchronization of the subsystem, and the processing unit 51 performs diagnostic processing of the storage device in the synchronized subsystem. If the system load exceeds the predetermined threshold, the synchronization control unit 54 releases the synchronization of the subsystem.
[0053] In the second embodiment, the determination unit 53 monitors the load status in the load measurement unit 55. When the storage devices have an abnormal tendency of performance degradation and the load exceeds a threshold, the determination unit 53 transmits to the synchronization control unit 54 a notification that desynchronization of the subsystems is necessary together with information on which storage device has an abnormal tendency.
[0054] [Operation] The process flow according to the second embodiment for monitoring the FT server 1 will be described with reference to the flowcharts of FIG. 9 to FIG.
[0055] 9 is a flowchart showing the processing of the monitoring system 50 for detecting a hard disk having an abnormal tendency of performance degradation according to the second embodiment. First, the processing unit 51 checks whether the FT server 1 is operating in a synchronous state. If it is not operating in a synchronous state (step S41; NO), the processing unit 51 notifies the display unit (not shown) of an error that the FT server 1 is not operating in a synchronous state (step S42). If it is operating in a synchronous state (step S41; YES), the processing unit 51 periodically issues IO to the hard disks 14 and 24 (step S43), measures the response time of each, and causes the recording unit 52 to record it (step S44).
[0056] The determination unit 53 monitors whether the difference in response time between the hard disks exceeds a threshold. The threshold for monitoring the performance degradation of the hard disks may be the same as that in the first embodiment. If the difference in response time exceeds the threshold, the determination unit 53 detects that one of the hard disks has an abnormal tendency of performance degradation.
[0057] If the difference in response time does not exceed the threshold value (step S45; NO), the process returns to step S41 to check the synchronization state, and the determination unit 53 waits for the next IO issuance.
[0058] If the difference in response time exceeds the threshold (step S45; YES), the determination unit 53 monitors whether the load exceeds the threshold. The load to be monitored may be selected according to the characteristics of the business, such as the CPU load, the IO load, or the like, which is necessary for monitoring the degree of the impact on the business processing. There may be one object to be monitored, or a combination of necessary objects may be used as the objects to be monitored.
[0059] If the load exceeds the threshold (step S46; YES), the judgment unit 53 judges which hard disk has a longer response time. If the response time of the hard disk 24 is longer (step S47; YES), the synchronization control unit 54 cancels the synchronization of the FT server 1 with the system 10 as the business continuation side and the system 20 as the hard disk diagnosis side (step S48). If the response time of the hard disk 14 is longer (step S47; NO), the synchronization control unit 54 cancels the synchronization of the FT server 1 with the system 20 as the business continuation side and the system 10 as the hard disk diagnosis side (step S49).
[0060] If the load does not exceed the threshold (step S46; NO), FT server 1 performs hard disk diagnostic process A2 while maintaining a synchronized state (step S50). If the result of hard disk diagnostic process A2 is that there is an abnormality in the hard disk (step S51; YES), FT server 1 separates the abnormal hard disk from mirroring and notifies an error that the hard disk needs to be replaced (step S52). At this time, the CPU module of FT server 1 operates while maintaining a synchronized state. Furthermore, necessary hardware components other than the CPU module may also operate while maintaining a synchronized state.
[0061] As a result of the hard disk diagnostic process A2, if it is determined that there is no abnormality in the hard disk (step S51; NO), the process returns to step S41 to check the synchronization state and wait for the next IO issuance.
[0062] 10 is a flowchart showing the diagnostic process (hard disk diagnostic process A2) according to the second embodiment. The hard disk diagnostic process A2 is performed when it is determined that one of the hard disks has an abnormal tendency of performance degradation, in order to check whether this is due to an accidental factor or whether an abnormality in performance degradation has actually occurred. The hard disk diagnostic process A2 is performed when the FT server 1 is synchronized.
[0063] In the hard disk diagnostic process A2, first, I / O is issued to the hard disk 14 and the hard disk 24, and the response time is measured and recorded (steps S61, S62). Next, it is monitored whether the load exceeds a threshold value.
[0064] If the load exceeds the threshold (step S63; YES), a determination is made as to which hard disk has a longer response time. If the response time of hard disk 24 is longer (step S71; YES), system 10 is set as the business continuation side and system 20 is set as the hard disk diagnosis side, and FT server 1 is released from synchronization (step S72). If the response time of hard disk 14 is longer (step S71; NO), system 20 is set as the business continuation side and system 10 is set as the hard disk diagnosis side, and FT server 1 is released from synchronization (step S73).
[0065] If the load does not exceed the threshold (step S63; NO), this process is repeated until a specified number of measurements (X times) are completed (step S64; NO), and after the specified number of measurements are completed, the process proceeds to a judgment process step (step S64; YES).
[0066] In the judgment process, first, the average response time of the hard disk 14 and the hard disk 24 for a specified number of times (X times) is calculated (step S65). The average response times of the hard disk 14 and the hard disk 24 are compared, and if the difference does not exceed a threshold value (step S66; NO), the hard disk is judged to be normal (step S70).
[0067] The average response times of the hard disk 14 and the hard disk 24 are compared, and if the difference exceeds a threshold (step S66; YES), a comparison is made to see which hard disk has the longer average response time, and if the average response time of the hard disk 14 is longer (step S67; NO), the hard disk 14 is determined to be abnormal (step S68).If the average response time of the hard disk 24 is longer (step S67; YES), the hard disk 24 is determined to be abnormal (step S69).
[0068] 11 is a flowchart showing the operations performed by system 10 and system 20 when releasing synchronization of FT server 1 when the difference in response time between the hard disks exceeds a threshold and the load also exceeds a threshold. Note that an example will be explained here in which an abnormal trend of performance degradation is detected in the hard disk 24 on the system 20 side.
[0069] First, systems 10 and 20 of FT server 1 are operating in a synchronized state (step S81). When an abnormal trend of performance degradation is detected in the hard disk 24, the load exceeds a threshold, and a notification is received that FT server 1 needs to be desynchronized, FT controller 12 instructs FT controller 22 to desynchronize (step S82). This instruction causes FT server 1 to be divided into systems 10 and 20, which operate independently (step S86). System 10 continues operations (step S83), and system 20 stops operations (step S87). At this time, in order to continue operations on the system 10 side, the IP address that was used during the synchronous operation of FT server 1 continues to be used. On the system 20 side, a different IP address is temporarily assigned.
[0070] In the system 20, for the hard disk 24 in which an abnormal tendency has been detected, a hard disk diagnostic process A1 is executed to determine whether or not the hard disk is actually abnormal. If the result of the hard disk diagnostic process A1 is that the hard disk 24 is determined to be normal (step S89; NO), the system 10 is notified via the network that the hard disk diagnostic process has ended (step S90). The system 10 that receives the notification (step S84) issues a synchronization instruction from the FT controller 12 to the FT controller 22 (step S85). The system 20 receives the synchronization instruction and performs synchronization based on the system 10 that was continuing business processing (step S91). When the synchronization process is completed, the FT server 1 returns to a synchronized state (step S92).
[0071] If the result of the hard disk diagnostic process A1 indicates that the hard disk 24 is abnormal (step S89; YES), the FT server 1 notifies the system 10 of an error that the hard disk is abnormal and needs to be replaced (step S93). At this time, the hard disk 24 determined to be abnormal is placed in a state where synchronization is not possible (step S94), and the system 10 is notified via the network that the hard disk diagnostic process has ended (step S90). The system 10 receives the notification (step S84), and issues a synchronization instruction from the FT controller 12 to the FT controller 22 (step S85). The system 20 receives the synchronization instruction and performs synchronization based on the system 10 that continued business processing (step S91). When the synchronization process is completed, the hardware components of the FT server 1 other than the hard disk are restored to a synchronized state (step S92).
[0072] [effect] According to the second embodiment, the present disclosure can also be applied when it is desired to give priority to the synchronized state of the FT server 1. This is because the FT server 1 diagnoses the hard disks while maintaining the synchronization of the FT server 1 until the load exceeds a threshold value. If it is determined that there is an abnormality in a hard disk, only that hard disk is separated from the mirroring.
[0073] In the second embodiment, when the load exceeds a threshold, the FT server 1 cancels synchronization of the FT server 1 and diagnoses the hard disks. If the diagnosis determines that there is an abnormality in the hard disk, the FT server 1 makes the hard disk unsynchronizable and causes the other hardware components to return to a synchronized state.
[0074] [Hardware configuration] In each of the above-described embodiments, each component of the monitoring system 50 is shown as a functional block. A part or all of each component of the monitoring system 50 may be realized by any combination of a computer 500 and a program. FIG. 12 is a block diagram showing an example of a hardware configuration of the monitoring system 50. Referring to FIG. 12, the computer 500 includes, for example, a CPU (Central Processing Unit) 501, a ROM (Read Only Memory) 502, a RAM (Random Access Memory) 503, a program 504, a storage device 505, a drive device 507, a communication interface 508, an input device 509, an input / output interface 511, and a bus 512.
[0075] The program 504 includes instructions for implementing each function of the monitoring system 50. The program 504 is stored in advance in the ROM 502, the RAM 503, or the storage device 505. The CPU 501 implements each function of the monitoring system 50 by executing instructions included in the program 504. For example, the CPU 501 of the monitoring system 50 implements a function of the monitoring system 50 by executing instructions included in the program 504. The RAM 503 may also store data to be processed in each function of the monitoring system 50. For example, the RAM 503 of the computer 500 may store a response time to an IO request.
[0076] The drive device 507 reads and writes data from and to the recording medium 506. The communication interface 508 provides an interface with a communication network. The input device 509 is, for example, a mouse or a keyboard, and accepts information input from a user. The output device 510 is, for example, a display, and outputs (displays) information to a user. The input / output interface 511 provides an interface with peripheral devices. The bus 512 connects these hardware components. The program 504 may be supplied to the CPU 501 via a communication network, or may be stored in advance in the recording medium 506, read by the drive device 507, and supplied to the CPU 501. For example, the computer 500 and the subsystem in the above-mentioned embodiment may be connected via a communication network, or may be connected via the input / output interface 511.
[0077] It should be noted that the hardware configuration shown in FIG. 12 is merely an example, and other components may be added, or some components may not be included.
[0078] [Variations] There are various modified examples of the method of realizing the monitoring system 50. The monitoring system 50 may be realized using the resources of the FT server 1. For example, the programs for the monitoring system 50 may be loaded onto each of the hard disks 14, 24. Also, for example, the monitoring system 50 may be realized by any combination of different computers and programs for each component. Also, multiple components included in the monitoring system 50 may be realized by any combination of a single computer and program.
[0079] In addition, some or all of the components of the monitoring system 50 may be realized by general-purpose or dedicated circuits including a processor or a combination of these. These circuits may be configured by a single chip, or may be configured by multiple chips connected via a bus. Some or all of the components of the monitoring system 50 may be realized by a combination of the above-mentioned circuits and a program.
[0080] Furthermore, when some or all of the components of the monitoring system 50 are realized by a plurality of computers, circuits, etc., the plurality of computers, circuits, etc. may be arranged in a centralized manner or in a distributed manner.
[0081] Although the present disclosure has been described above with reference to the embodiments, the present disclosure is not limited to the above embodiments. Various modifications that can be understood by a person skilled in the art can be made to the configuration and details of the present disclosure within the scope of the present disclosure. Furthermore, the configurations in each embodiment can be combined with each other without departing from the scope of the present disclosure. [Explanation of symbols]
[0082] 1 FT Server 10, 20 Subsystem 11, 21 CPU module 12, 22FT Controller 13, 23 IO Module 14, 24 Hard Disk 50 Surveillance System 51 Processing section 52 Recording Section 53 Judgment section 54 Synchronization control section 55 Load measurement section
Claims
1. A monitoring system for monitoring a fault-tolerant server having a first subsystem and a second subsystem, comprising: a processing means for issuing an IO (Input Output) request to a storage device included in each of the first and second subsystems which perform business processing in synchronization with each other; a determination means for determining an abnormal tendency of the storage device having a longer response time based on a difference in the response time to the I / O request; a load measuring means for measuring a load of the first and second subsystems; a synchronization control means for releasing the synchronization between the first and second subsystems and controlling the first and second subsystems to be able to operate independently when the load exceeds a predetermined threshold value when the determination means determines that there is an abnormal tendency, The processing means includes: When the load exceeds a predetermined threshold value when the abnormal tendency is determined, a diagnosis process is performed on the storage device in either the first or second subsystem, in which synchronization is released, the storage device being determined to have the abnormal tendency, to determine whether or not the storage device has an abnormality; When the abnormal tendency is determined, if the load does not exceed a predetermined threshold, the first and second subsystems are synchronized and the diagnostic process is performed. Surveillance system.
2. The processing means determines whether the first and second subsystems are operating synchronously, and when it is determined that the first and second subsystems are operating synchronously, periodically issues the I / O requests and measures and records the response times of each of the I / O requests; The determining means determines whether the load exceeds a predetermined threshold value when the difference in the response times exceeds a threshold value. The monitoring system of claim 1 .
3. A monitoring method for monitoring a fault-tolerant server having a first subsystem and a second subsystem, comprising: issuing an IO (Input Output) request to a storage device provided in each of the first and second subsystems which perform business processing in synchronization with each other; determining an abnormal tendency of the storage device having a longer response time based on the difference in response time to the IO request; Measuring loads on the first and second subsystems; when the load exceeds a predetermined threshold when the abnormal tendency is determined, the synchronization of the first and second subsystems is released, and the first and second subsystems are controlled so as to be able to operate independently, and in the subsystem including the storage device determined to have the abnormal tendency, a diagnosis process is performed to determine whether or not the storage device has an abnormality; When the load does not exceed a predetermined threshold value when the abnormal tendency is determined, the synchronization between the first and second subsystems is not released, and the diagnosis process is performed in a state in which the first and second subsystems are synchronized. Monitoring method.
4. A program for monitoring a fault-tolerant server having a first subsystem and a second subsystem, comprising: a process of issuing an IO (Input Output) request to a storage device included in each of the first and second subsystems which perform business processing in synchronization with each other; A process of determining an abnormal tendency of the storage device having a longer response time based on a difference in the response time to the I / O request; A process of measuring loads of the first and second subsystems; a process of releasing synchronization between the first and second subsystems and controlling the first and second subsystems to be able to operate independently when the load exceeds a predetermined threshold when an abnormal trend is determined; a diagnostic process for diagnosing whether or not there is an abnormality in the storage device in either the first or second subsystem in which synchronization has been released, the storage device being determined to have the abnormal tendency; a diagnostic process for diagnosing the presence or absence of an abnormality in the storage device while the first and second subsystems are in a synchronized state, without releasing the synchronization between the first and second subsystems if the load does not exceed a predetermined threshold when the abnormality tendency is determined; A program that causes a computer to execute the following.
5. a first subsystem having a first storage device, a first CPU module, and a first FT (Fault Tolerant) controller; a second subsystem having a second storage device, a second CPU module, and a second FT controller; the first and second storage devices respectively included in the first and second subsystems which perform business processing in synchronization with each other respond to an IO (Input Output) request issued from a monitoring system; an abnormality tendency of the first storage device is determined based on the difference in response time to the I / O request, and when the load of the first and second subsystems exceeds a predetermined threshold, the first FT controller controls the release of synchronization of the first and second subsystems, and a diagnosis of an abnormality of the first storage device is performed in the first subsystem from which synchronization has been released; An abnormal tendency of the first storage device is determined based on the difference in response time to the I / O request, and if the load of the first and second subsystems does not exceed a predetermined threshold, a diagnosis of an abnormality of the first storage device is performed in a state in which the first and second subsystems are synchronized. Fault-tolerant servers.
Citation Information
Patent Citations
Disk array device, its error control method, and recording medium where control program thereof is recorded
JP1999338648A
Disk array device and method for controlling its error and recording medium with its control program recorded thereon
JP2002108573A
Information processing device, program, and verification method
JP2013178713A
Backup method in fault tolerant server
JP2013206052A
Fault tolerant system, fault tolerant method, and program
JP2016115239A
