Failure monitoring device and failure monitoring method

The failure monitoring device in information processing devices uses an oldest information holding unit and failure counters to swiftly detect and switch to standby devices, addressing the inefficiencies in existing systems and ensuring data integrity and real-time performance.

JP2024168590AInactive Publication Date: 2024-12-05FUJITSU LTD
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
JP2023085409
Authority / Receiving Office
JP · JP
Patent Type
Applications
Current Assignee / Owner
Filing Date
2023-05-24
Publication Date
2024-12-05
Estimated Expiration
Not applicable · inactive patent

AI Technical Summary

Technical Problem

In information processing devices with multiple active and standby devices connected via shared memory, the load on the management device increases with the number of active devices, leading to longer failure detection cycles and slower switching, which can result in data overflow and loss.

Method used

A failure monitoring device is installed in the information processing device, utilizing an oldest information holding unit and failure counters to detect failures promptly by monitoring the update status of data in shared memory, allowing for rapid switching to standby devices.

Benefits of technology

This approach reduces delays in failure detection and minimizes data loss by enabling quick switching to standby devices, maintaining real-time performance and reducing power consumption.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 2024168590000001_ABST
    Figure 2024168590000001_ABST
Patent Text Reader

Abstract

To suppress delay in detection of a failure even when multiple devices are installed by performing a detection operation of the failure correspondingly to the operation of each device.SOLUTION: A failure monitoring device mounted on an information processing device that includes a shared memory, a first device that sequentially writes data to the shared memory and a second device that sequentially reads the data from the shared memory comprises: an oldest information holding unit that holds oldest information indicating the oldest data not read by the second device in the data written to the shared memory by the first device; an oldest information updating unit that updates the oldest information held in the oldest information holding unit when the data read from the shared memory by the second device is output to a device subsequent to the second device; and a first failure detection unit that detects a failure in the second device when the oldest information held in the oldest information holding unit is not updated even after the first device writes the data to the shared memory by the predetermined times.SELECTED DRAWING: Figure 2
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] The present invention relates to a fault monitoring device and a fault monitoring method. [Background technology]

[0002] A method is known in which, when a failure is detected in an active device, data is obtained from a shared storage where the data of the active device is duplicated, and a standby device is operated, so that the function of the active device can be continued by the standby device (see, for example, Patent Document 1).A method is also known in which a shared disk device is provided in which the same contents as those of the built-in disk device of an active device are written, and when a failure occurs in the active device, the standby device copies the contents of the shared disk device and takes over the processing of the active device (see, for example, Patent Document 2). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2016-110173 A [Patent Document 2] Japanese Patent Application Publication No. 6-175788 Summary of the Invention [Problem to be solved by the invention]

[0004] For example, in an information processing device including a plurality of operational devices and a standby device connected via a shared memory, the upstream operational device writes processed data to the shared memory in sequence, and the downstream operational device processes data read from the shared memory in sequence. For example, a management device mounted on the information processing device monitors the plurality of operational devices to detect a failure, and switches the failed operational device to a standby device. However, the greater the number of operational devices, the greater the load on the management device, the longer the monitoring period for failures of each operational device, and the slower the switching from the operational device to the standby device when a failure occurs. For example, if the detection of the failure and the switching to the standby device are delayed when a downstream operational device fails, the shared memory may overflow, resulting in data loss.

[0005] In one aspect, the present invention aims to prevent delays in failure detection even when multiple devices are installed, by performing failure detection operations in accordance with the operation of each device. [Means for solving the problem]

[0006] According to one aspect, the fault monitoring device is mounted on an information processing device having a shared memory, a first device that sequentially writes data to the shared memory, and a second device that sequentially reads data from the shared memory, and has an oldest information holding unit that holds oldest information indicating the oldest data among the data written to the shared memory by the first device that has not been read by the second device, an oldest information update unit that updates the oldest information held in the oldest information holding unit when data read from the shared memory by the second device is output to a device subsequent to the second device, and a first fault detection unit that detects a fault in the second device when the oldest information held in the oldest information holding unit is not updated even after the first device has written data to the shared memory a predetermined number of times. Effect of the Invention

[0007] By performing failure detection operations in accordance with the operation of each device, delays in failure detection can be suppressed even when multiple devices are installed. [Brief description of the drawings]

[0008] [Figure 1] FIG. 1 is a block diagram illustrating an example of an information processing device according to an embodiment. [Diagram 2] FIG. 2 is an explanatory diagram showing an example of the operation of the information processing device in FIG. [Diagram 3] FIG. 3 is an explanatory diagram showing a continuation of FIG. 2. [Figure 4] 1. FIG. 4 is an explanatory diagram showing another example of the operation of the information processing device in FIG. [Diagram 5] FIG. 5 is an explanatory diagram showing a continuation of FIG. [Figure 6] 6 is an explanatory diagram showing an example of the operation of another information processing device not having the functions described with reference to FIGS. 2 to 5. FIG. [Figure 7] FIG. 11 is a block diagram illustrating an example of an information processing device according to another embodiment. [Figure 8] 8 is a block diagram showing an example of a data structure of a shared memory in FIG. 7. [Figure 9] 8 is a sequence diagram showing an example of an operation of the information processing device in FIG. 7. [Figure 10] FIG. 10 is a sequence diagram showing a continuation of FIG. [Figure 11] FIG. 11 is a sequence diagram showing a continuation of FIG. [Figure 12] 8 is a flow chart showing an example of the overall operation of the data management method of the information processing device of FIG. 7. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Hereinafter, embodiments will be described with reference to the drawings.

[0010] FIG. 1 shows an example of an information processing device in an embodiment. The information processing device 100 shown in FIG. 1 has a CPU, multiple shared memories CM (CM1, CM2, CM3), and multiple cards CD (CD1, CD2, CD3). Each card CD reads data from the shared memory CM connected upstream, processes it, and writes the processed data to the shared memory CM connected downstream. The information processing device 100 is included in a system 300 together with a pre-stage device 210 and a post-stage device 220. The card CD that writes data to the shared memory CM is an example of a first device, and the card that reads data from the shared memory CM is an example of a second device.

[0011] Card CD1 is either one of N cards CD11-CD1N assigned to the operating system or card CD1S assigned to the standby system. Card CD2 is either one of N cards CD21-CD2N assigned to the operating system or card CD2S assigned to the standby system. Card CD3 is either one of N cards CD3 (CD31-CD3N) assigned to the operating system or card CD3S assigned to the standby system. Cards CDS (CD1S, CD2S, CD3S) assigned to the standby system are an example of a third device.

[0012] The CPU controls the overall operation of the information processing device, and also manages cards CD connected to card slots (not shown) via a monitor / control path CP. Although not limited thereto, each card CD may be an accelerator card such as an FPGA (Field-Programmable Gate Array) card or a GPU (Graphics Processing Unit) card.

[0013] Each card CD may adopt a PCIe (Peripheral Component Interconnect express) interface or a CXL (Compute Express Link) interface, or may adopt an optical interface. Note that the information processing device 100 may have a processing device detachably connected to a connector or the like, instead of the card CD.

[0014] Data transfer between the shared memory CM and the cards CD is performed via a data communication path DP. Each of the cards CD is assigned to one of N active systems or one standby system. The standby system is used in place of the active system if any of the active systems fails, and is not used if the active system does not fail.

[0015] An input of the first-stage shared memory CM1 is connected to the preceding stage device 210 via a data communication path DP of the active system, and receives data supplied from the preceding stage device 210 for each active system. An output of the first-stage shared memory CM1 is connected to cards CD11-CD1N, CD1S via the data communication path DP. An input of the second-stage shared memory CM2 is connected to cards CD11-CD1N, CD1S via the data communication path DP. An output of the second-stage shared memory CM2 is connected to cards CD21-CD2N, CD2S via the data communication path DP.

[0016] The input of the third stage shared memory CM3 is connected to the cards CD21-CD2N, CD2S via a data communication path DP. The output of the third stage shared memory CM3 is connected to the cards CD31-CD3N, CD3S via a data communication path DP. The cards CD31-CD3N, CD3S are connected to the input of the shared memory CM4 provided corresponding to the subsequent stage device 220.

[0017] Although not particularly limited, for example, each of the operational cards CD1, CD2, and CD3 sequentially processes images received from the previous stage device 210 and outputs the processed images to the subsequent stage device 220. For example, the card CD1 reads compressed image data stored in the shared memory CM1 by the previous stage device 210 from the shared memory CM1, decompresses it, and writes the decompressed image data to the shared memory CM2. The card CD2 reads the decompressed image data from the shared memory CM2, reduces the size of the image data, and writes the obtained reduced image data to the shared memory CM3.

[0018] The card CD3 reads the reduced image data from the shared memory CM3, detects an object contained in the reduced image data, and writes information indicating the detected object together with the reduced image data to the shared memory CM4. In the following, the processing of the card CD1 is performed in the stage STG1, the processing of the card CD2 is performed in the stage STG2, and the processing of the card CD3 is performed in the stage STG3. In addition, when there is no distinction between the stages STG1 to STG3, they will be referred to as the stage STG.

[0019] Each of the cards CD1-CD3 completes its own image processing within a preset period. That is, the writing of data DT to the shared memory CM by each of the cards CD1-CD3 and the reading of data DT from the shared memory CM by each of the cards CD1-CD3 are repeatedly performed in the same period.

[0020] By each of the cards CD1-CD3 sequentially performing image processing at each cycle, the information processing device 100 can process image data transferred sequentially from the previous stage device 210 without data being retained in the shared memories CM1-CM3. Then, the information processing device 100 can sequentially transfer the processed image data to the shared memory CM4 connected to the subsequent stage device 220. In other words, the information processing device 100 can transfer the processed images to the subsequent stage device 220 within the guaranteed time without missing any images.

[0021] As described in Figures 2 and 3, the standby cards CD2S, CD3S can detect a failure of the active system card CD of their own stage STG in cooperation with the active system card CD of the stage STG one stage upstream and the active system card CD of their own stage STG. Also, as shown in Figures 4 and 5, the standby cards CD1S, CD2S can detect a failure of the active system card CD of their own stage STG in cooperation with the active system card CD of the stage STG one stage downstream and the active system card CD of their own stage STG.

[0022] 2 and 3 show an example of the operation of the information processing device 100 of FIG. 1. FIG. 2 and 3 show an example of a fault monitoring method by a fault monitoring device mounted on the information processing device 100. Each card CD repeatedly performs data processing including reading data DT from the upstream shared memory CM, processing the read data DT, and writing the processed data DT to the downstream shared memory CM at a preset period. The data processing period of each card CD is set so as to satisfy the data transfer rate from the front-stage device 210 to the rear-stage device 220 determined by the specifications of the system 300, and is common to all cards CD.

[0023] Each shared memory CM has a management area for storing the oldest information OLD, the latest information NEW, and the values ​​of the failure counters CNT1 and CNT2, and a data area for storing multiple data DT, for each active system. For example, each of the areas for storing the oldest information OLD, the latest information NEW, and the values ​​of the failure counters CNT1 and CNT2 is about 1 byte.

[0024] The area for storing the oldest information OLD is an example of an oldest information storage section, and the area for storing the latest information NEW is an example of a latest information storage section. In Figures 2 and 3, each of the shared memories CM3 and CM4 shows a management area and a data area corresponding to one operational system.

[0025] The data area holds data DT processed by the upstream connected card CD. The letters a, b, c, and d at the end of the data DT indicate the order in which the data DT is processed by each card CD (DTa, DTb, DTc, and DTd). For the sake of simplicity, it is assumed that the shared memory CM3 can hold a maximum of three pieces of data DT.

[0026] The area OLD holds the oldest information OLD indicating the oldest data DT among the data DT written by the card CD connected upstream. The area NEW holds the latest information NEW indicating the latest data DT among the data DT written by the card CD connected upstream.

[0027] The value of the failure counter CNT1 is incremented every data processing cycle when the card CD connected upstream detects that the oldest information OLD has not been updated. The value of the failure counter CNT2 is incremented every data processing cycle when the card CD connected downstream detects that the latest information NEW has not been updated. The failure counter CNT1 is an example of a first counter, and the failure counter CNT2 is an example of a second counter.

[0028] 2 and 3 show an example in which the oldest information OLD and the failure counter CNT1 are used to detect a failure of the active card CD3 by the standby card CD3S. An example in which the latest information NEW and the failure counter CNT2 are used to detect a failure of the active card CD2 by the standby card CD2S is shown in FIG. 4 and FIG.

[0029] Card CD2 is any one of cards CD21-CD2N in Fig. 1, and card CD3 is any one of cards CD31-CD3N in Fig. 1. The oldest information OLD and failure counter CNT1 are used, and the operation in which a failure of active card CD2 is detected by standby card CD2S is also the same as in Fig. 2 and Fig. 3. The values ​​of failure counters CNT1 and CNT2 are initialized to 0 in the initial state when each card CD starts processing.

[0030] In state ST1, the shared memory CM3 holds the data DTa and DTb processed by the card CD2, and the oldest information OLD=a indicating the oldest data DTa. The card CD3 reads the data DTa corresponding to the oldest information OLD=a and performs processing (RD). At this time, the card CD2 is processing the data DTc.

[0031] Next, in state ST2, after completing the processing of the data DTa, the card CD3 writes the processed data DTa to the shared memory CM4. The standby card CD3S reads the value of the failure counter CNT1 at each data processing cycle and judges whether the increment of the counter value is equal to or greater than a threshold value. In the example shown in FIG. 2 and FIG. 3, the threshold value is 2.

[0032] When the value of the failure counter CNT1 is equal to or greater than the threshold value, the card CD3S determines that a failure has occurred in the active card CD3 corresponding to the failure counter CNT1 in the own stage STG. In state ST2, since the value of the failure counter CNT1 is 0, the card CD3S determines that the card CD3 is not at fault.

[0033] Next, in state ST3, the card CD3 erases (DEL) the processed data DTa and updates the oldest information OLD to OLD=b corresponding to the data DTb. After outputting the processed data DTa to the shared memory CM4, which is the subsequent device, the card CD3 updates the oldest information OLD, and also functions as an oldest information update unit.

[0034] Before writing the data DTc to the shared memory CM3, the card CD2 checks whether the oldest information OLD has been updated since the previous time the data DT was written. In state ST3, since the oldest information OLD has been updated to OLD=b, the card CD2 writes the processed data DTc to the shared memory CM3 (WR) without incrementing the failure counter CNT1.

[0035] The operations shown in states ST1 to ST3 are repeatedly performed in all active cards CD1, CD2, CD3, so that data is processed sequentially by cards CD1, CD2, CD3 for each active system and transferred downstream.

[0036] 3, the card CD3 fails, and processing by the card CD3 stops. The failed card CD3 cannot read data DTb from the shared memory CM3, process the read data DTb, erase the data DTb held in the shared memory CM3, or update the oldest information OLD from b to c. The standby card CD3S reads the value of the failure counter CNT1, and since it is not equal to or greater than the threshold value (=2), it determines that the card CD3 is not failed.

[0037] In state ST5, before writing data DTd to shared memory CM3, card CD2 checks that the oldest information OLD=b has not been updated since the previous data DT was written. Since the oldest information OLD has not been updated, card CD2 increments (+1) the failure counter CNT1 and writes data DTd to shared memory CM3. The standby card CD3S reads the value of failure counter CNT1 and determines that card CD3 is not faulty because it is not equal to or greater than the threshold value (=2).

[0038] In state ST6, before writing the next processed data DTE to the shared memory CM3, the card CD2 checks that the oldest information OLD=b has not been updated since the previous data DT was written. Since the oldest information OLD has not been updated, the card CD2 increments (+1) the failure counter CNT1 and writes the data DTE to the shared memory CM3.

[0039] The shared memory CM3 has no free space left because the data DTE has been written thereto. Therefore, when the card CD2 next writes the processed data DTf to the shared memory CM3, the data area of ​​the shared memory CM3 overflows. Alternatively, when the card CD2 stops writing data to the shared memory CM3, data processing stagnates. To prevent such a problem, as shown below, a failure of the card CD3 is detected by the standby card CD3S, the standby card CD3S switches to the active card, and processing continues from the point in time when the failure occurred.

[0040] Since the value of the failure counter CNT1 is equal to or greater than the threshold value (=2), the card CD3S detects a failure of the card CD3 of its own stage STG. The card CD3S also functions as a first failure detection unit that determines that the oldest information OLD is not updated based on the failure counter CNT1 and detects a failure of the card CD3. Based on the detection of the failure, the card CD3S separates the failed card CD3 from the active system and operates its own card CD3S as the active system card CD3. After this, the card CD3 that has newly started operating as the active system reads out the data DTb corresponding to the oldest information OLD=b and resumes processing.

[0041] 2 and 3, the card CD3 that updates the oldest information OLD, the card CD2 that increments the failure counter CNT1, and the card CD3S that detects a failure of the card CD3 based on the value of the failure counter CNT1 function as a failure monitoring device. Then, the failed card CD3 can be replaced on a card-by-card basis.

[0042] Since a failure can be detected by storing about 1 byte of oldest information OLD and a failure counter CNT1 in the shared memory CM3 corresponding to each card CD3, the effect of providing a failure monitoring device on the operational system can be minimized.

[0043] By equipping cards CD2, CD3, and CD3S with the function of a fault monitoring device that detects faults in card CD3, the time from when a fault occurs to when the fault is detected can be shortened compared to when the fault is detected by the CPU using the data communication path DP.

[0044] Since one standby card CD3S can detect failures of multiple cards CD3 for each stage STG, a so-called N:1 redundant system can be realized. Since the number of standby cards CD3S mounted can be minimized, the power consumption of the information processing device 100 can be reduced compared to a case where multiple standby cards CD3S are mounted for each stage STG. Note that the information processing device 100 may employ an N:2 redundant system having two standby cards CD3S.

[0045] The threshold value of the failure counter CNT1, which is used by the card CD3S to determine a failure of the card CD3, is set to a value that prevents the oldest data DT from being overwritten even when the data DT in the shared memory CM3 is overwritten by the card CD2. Also, the threshold value of the failure counter CNT1 is set so that the data transfer rate from the front-stage device 210 to the back-stage device 220 determined by the specifications of the system 300 is satisfied even when the data DT stagnates in the shared memory CM due to a failure of the card CD.

[0046] Figures 4 and 5 are explanatory diagrams showing another example of the operation of the information processing device 100 of Figure 1. Figures 4 and 5 show an example of a failure monitoring method by a failure monitoring device mounted on the information processing device 100. Detailed explanations of operations similar to those of Figures 2 and 3 are omitted. In Figures 4 and 5, the latest information NEW and a failure counter CNT2 are used, and a failure of the active card CD2 is detected by the standby card CD2S. The oldest information OLD and the value of the failure counter CNT1 are the same as those of Figures 2 and 3.

[0047] 4 and 5 also show an example of an operation in which data written from the card CD2 to the shared memory CM3 is processed by the card CD3 and written to the shared memory CM4. However, the processing of the other cards CD1 and CD2 may be performed in the same manner as the processing shown in Fig. 4 and Fig. 5. The card CD2 is any one of the cards CD21-CD2N in Fig. 1, and the card CD3 is any one of the cards CD31-CD3N in Fig. 1.

[0048] State ST1 is the same as state ST1 in Fig. 2. In state ST2, after completing the processing of data DTa, card CD3 writes the processed data DTa to shared memory CM4. In Fig. 4, the standby card CD2S reads the value of the failure counter CNT2 at each data processing cycle and judges whether the increment of the counter value is equal to or greater than a threshold value. In the examples shown in Figs. 4 and 5, the threshold value is 2.

[0049] When the value of the failure counter CNT2 is equal to or greater than the threshold value, the card CD2S determines that a failure has occurred in the active card CD2 corresponding to the failure counter CNT2 in the own stage STG. In the state ST2, since the value of the failure counter CNT2 is 0, the card CD2S determines that the card CD2 is not at fault.

[0050] Next, in state ST3, the card CD3 erases (DEL) the processed data DTa and updates the oldest information OLD to OLD=b corresponding to the data DTb. The card CD2 writes the data DTc to the shared memory CM3 and updates the latest information NEW to c corresponding to the data DTc. The card CD2 that updates the latest information NEW also functions as a latest information update unit.

[0051] The operations shown in states ST1 to ST3 are repeatedly performed in all active cards CD1, CD2, CD3, so that data is processed sequentially by cards CD1, CD2, CD3 for each active system and transferred downstream.

[0052] In state ST4 of Fig. 5, card CD2 fails, and processing by card CD2 stops. The failed card CD2 cannot read data DTa from shared memory CM2 (not shown), process the read data DTa, erase data DTa held in shared memory CM2, or update the oldest information OLD=e. The failed card CD2 cannot write data DTd to shared memory CM3 or update the latest information NEW. The standby card CD2S reads the value of the failure counter CNT2, and since it is not equal to or greater than the threshold value (=2), determines that card CD2 is not failed.

[0053] In state ST5, the card CD3 reads the data DTb and the latest information NEW=c from the shared memory CM3 and confirms that the latest information NEW=c has not been updated. Since the latest information NEW has not been updated, the card CD3 increments (+1) the failure counter CNT2. The standby card CD2S reads the value of the failure counter CNT2 and determines that the card CD2 is not at fault because it is not equal to or greater than the threshold value (=2).

[0054] In state ST6, the card CD3 writes the processed data DTb to the shared memory CM4 and updates the oldest information OLD to c. The card CD3 also reads the data DTc and the latest information NEW=c from the shared memory CM3 and confirms that the latest information NEW=c has not been updated. Since the latest information NEW has not been updated, the card CD3 increments (+1) the failure counter CNT2.

[0055] The standby card CD2S reads the value of the failure counter CNT2, and since it is equal to or greater than the threshold value (=2), detects a failure of the card CD2 of its own stage STG. The card CD2S also functions as a second failure detection unit that determines that the latest information NEW is not updated based on the failure counter CNT2 and detects a failure of the card CD2. Based on the detection of the failure, the card CD2S separates the failed card CD2 from the active system and operates its own card CD2S as the active system card CD2. After this, the card CD2 that has newly started operating as the active system reads the data DTd from the shared memory CM2 and performs processing.

[0056] 4 and 5, the card CD2 that updates the latest information NEW, the card CD3 that increments the failure counter CNT2, and the card CD2S that detects a failure of the card CD2 based on the value of the counter CNT2 function as a failure monitoring device. Then, a failed card CD2 can be replaced on a card-by-card basis.

[0057] Since a failure can be detected by storing about 1 byte of the latest information NEW and the failure counter CNT2 in the shared memory CM3 corresponding to each card CD2, the effect of providing a failure monitor on the operational system can be minimized.

[0058] By equipping cards CD2, CD3, and CD2S with the function of a fault monitoring device that detects faults in card CD2, the time from when a fault occurs to when the fault is detected can be shortened compared to when the fault is detected by the CPU using the data communication path DP.

[0059] Since one standby card CD2S can detect failures of multiple cards CD2 for each stage STG, so-called N:1 redundancy can be realized. Since the number of standby cards CD2S mounted can be minimized, the power consumption of the information processing device 100 can be reduced compared to the case where multiple standby cards CD2S are mounted for each stage STG.

[0060] Figure 6 shows an example of the operation of another information processing device that does not have the functions described in Figures 2 to 5. The operation shown in Figure 6 is similar to the operation shown in Figures 2 to 5, except that each shared memory CM does not have an area for the failure counters CNT1, CNT2, and a failure of each card CD is detected by a CPU mounted on the information processing device, not by the card CD3S of the standby system.

[0061] For example, in state ST3, assume that the card CD3 fails and processing by the card CD3 stops. If the card CD3 fails, data DT3 is written from the card CD2 to the shared memory CM3 without being read from the shared memory CM3 to the card CD3, which may cause the shared memory CM3 to overflow.

[0062] For example, the CPU monitors each card CD at a cycle slower than the cycle of data processing by each card CD, and detects a failure of the card CD3 after the shared memory CM3 overflows. The CPU performs redundancy switching to switch the standby card CD3s to the active card in place of the failed card CD3, but if the oldest data DT is lost before the redundancy switching, the information processing device 100 will not normally process the data, causing the system 300 to malfunction.

[0063] As described above, in this embodiment, the active card CD and the standby card CDS connected via the shared memory CM can detect and replace failures on a card CD basis. Therefore, compared to when a failure is detected by a CPU, the standby card CDS can be switched to the active card at high speed, and even if the active card CD fails, data DT can be prevented from being lost or duplicated. And even if the active card CD fails, the data transfer rate from the pre-stage device 210 to the post-stage device 220 can be prevented from exceeding the specifications, and the real-time nature of the system 300 can be maintained.

[0064] For example, by adopting an optical interface for each card CD, it will be possible to install it in an IOWN (Innovative Optical and Wireless Network), which will contribute to the realization of disaggregated computing.

[0065] FIG. 7 shows an example of an information processing device in another embodiment. The same elements as those in FIG. 1 are given the same reference numerals, and detailed description is omitted. The system 300 shown in FIG. 7 has the same configuration and function as the system 300 shown in FIG. 1, except that it has an information processing device 100A instead of the information processing device 100 in FIG. 1. In the information processing device 100A, each card CD of the operation system has a data processing unit DPRC and an operation control unit OPCNT, and each card CDS of the standby system has a data processing unit DPRC and a standby control unit SBCNT. The information processing device 100A has the same configuration and function as the information processing device 100 in FIG. 1.

[0066] The information processing device 100A includes a CPU, multiple shared memories CM (CM1, CM2, CM3), and multiple active cards CD1 (CD11-CD1N), CD2 (CD21-CD2N), and CD3 (CD31-CD3N) in the same manner as the information processing device 100 in Fig. 1. The information processing device 100A also includes multiple standby cards CD1S, CD2S, and CD3S in the same manner as the information processing device 100 in Fig. 1.

[0067] The data processing unit DPRC of each active card CD has the function of processing data DT read from the upstream shared memory CM and writing it to the downstream shared memory CM. The operation control unit OPCNT of each active card CD manages the reading of data DT from the upstream shared memory CM and the writing of data DT to the downstream shared memory CM. The operation control unit OPCNT of each active card CD also refers to and updates the management information (Fig. 8) held in the shared memory CM.

[0068] The data processing unit DPRC of each card CDS of the standby system processes data DT when the standby card CDS is switched to the active system due to a failure of the active card CD. The standby control unit SBCNT of each card CDS of the standby system refers to the values ​​of the failure counters CNT1 and CNT2 held in the shared memory CM and detects a failure of the active card CD.

[0069] Fig. 8 shows an example of the data structure of the shared memory CM in Fig. 7. Each shared memory CM has a holding area corresponding to each of N active systems and m data areas. The holding areas have the same data structure. Each holding area has the oldest information OLD, the latest information NEW, and failure counters CNT1 and CNT2 as management information.

[0070] The card CD that writes data DT to the shared memory CM initially sets the oldest information OLD to information indicating the initial data DT when writing the data DT for the first time (WR). The card CD that writes data DT to the shared memory CM writes information indicating the data DT to be written as the latest information NEW in the holding area when writing the data DT to the shared memory CM.

[0071] The card CD which writes the data DT into the shared memory CM monitors whether the oldest information OLD has been updated since the previous time the data DT was written, when writing the data DT into the shared memory CM.

[0072] The failure counters CNT1 and CNT2 are initialized to "0" when the information processing device 100A starts operating and when the standby card CDS switches to the active card CD. The card CD that writes data DT to the shared memory CM increments the value of the failure counter CNT1 if the oldest information OLD has not been updated since the previous write when writing the data DT. The card CD that reads data DT from the shared memory CM increments the value of the failure counter CNT2 if the latest information NEW has not been updated since the previous read when reading the data DT.

[0073] The standby card CD1S detects a failure of the corresponding card CD11-CD1N based on the value of the failure counter CNT2 stored for each of the cards CD11-CD1N in the shared memory CM2. The standby card CD3S detects a failure of the corresponding card CD31-CD3N based on the value of the failure counter CNT1 stored for each of the cards CD31-CD3N in the shared memory CM3.

[0074] The standby card CD2S detects a failure of the corresponding card CD21-CD2N based on the value of the failure counter CNT2 stored for each of the cards CD21-CD2N in the shared memory CM3. Alternatively, the standby card CD2S detects a failure of the corresponding card CD21-CD2N based on the value of the failure counter CNT1 stored for each of the cards CD21-CD2N in the shared memory CM2.

[0075] A card CD which reads data from the shared memory CM writes the processed data DT into a data area of ​​the downstream shared memory CM, erases the processed data DT from the shared memory CM, and sets the oldest information OLD to information indicating the data DT to be read next.

[0076] When the value of the failure counter CNT1 (or CNT2) is equal to or greater than a preset threshold value for determining a failure, the standby card CDS detects the failure of the corresponding card CD and performs a process of switching its own card CDS to the active card CD. After that, the standby card CDS operates as the active card CD.

[0077] 9 to 11 show an example of the operation of the information processing device 100A of FIG. 7. FIG. 9 to 11 show an example of a fault monitoring method by a fault monitoring device mounted on the information processing device 100A. As in the above-mentioned embodiment, each card CD repeatedly performs data processing including reading data DT from the upstream shared memory CM, processing the read data DT, and writing the processed data DT to the downstream shared memory CM at a preset period. The data processing period of each card CD is set so as to satisfy the data transfer rate from the front-stage device 210 to the rear-stage device 220 determined by the specifications of the system 300, and is common to all cards CD.

[0078] 9 to 11 show an example in which the card CD3S detects a failure of the card CD3 in its own stage STG by using the oldest information OLD and the failure counter CNT1. That is, Fig. 9 to 11 show the operation corresponding to Fig. 2 and Fig. 3 described above. The operation in which the card CD2S detects a failure of the card CD2 in its own stage STG by using the latest information NEW and the failure counter CNT2 is the same as that in Fig. 4 and Fig. 5.

[0079] 9 to 11 show an example of an operation in which data written from the card CD2 to the shared memory CM3 is processed by the card CD3 and written to the shared memory CM4. However, the processing of the other cards CD1 and CD2 is also performed in the same manner as the processing shown in Fig. 9 to 11. The card CD2 is any one of the cards CD21-CD2N in Fig. 7, and the card CD3 is any one of the cards CD31-CD3N in Fig. 7.

[0080] In the following, the data DT is identified by alphabets. The alphabets a, b, ..., m indicate the order of the data DT supplied from the front-stage device 210 in Fig. 7. Moreover, the data DT held in the shared memories CM2, CM3, and CM4 are indicated by alphabets only. The frame indicated by the dashed dotted line indicates a repeating period CYC in which the update operation of the failure counter CNT1 based on the oldest information OLD by the card CD2 and the failure determination operation of the card CD3 by the standby card CD3S are performed.

[0081] Writing data DT to the shared memory CM is indicated by the symbol WR, and reading data DT from the shared memory CM is indicated by the symbol RD. Data DT enclosed in a dashed frame in the shared memory CM indicates that it is to be erased. The value of the failure counter CNT1 is shown only in the shared memory CM3 of interest. The description of the latest information NEW is omitted.

[0082] In cycle CYC1, data DTE, DTf, DTg and oldest information OLD=e are stored in shared memory CM2. After completing the processing of data DTE, card CD2 writes the processed data DTE to shared memory CM3 (FIG. 9(a)). As a result, data c, d, e and oldest information OLD=c are stored in shared memory CM3.

[0083] The card CD3 reads the oldest data DTc from the shared memory CM3, processes it, and writes the processed data DTc to the shared memory CM4 (FIG. 9(b)). The shared memory CM4 holds the data a, b, and c and the oldest information OLD=a.

[0084] The card CD2 reads out the oldest information OLD=c from the shared memory CM3 and stores it as the current oldest information OLD (FIG. 9(c)). After writing the data DTE to the shared memory CM3, the card CD2 updates the oldest information OLD of the shared memory CM2 to f and erases the data DTE held in the shared memory CM2 (FIGS. 9(d) and (e)).

[0085] The card CD2 compares the oldest information OLD=b from the previous time when the data DTd was written to the shared memory CM3 with the oldest information OLD=c this time, and since they are not the same, it confirms that the oldest information OLD has been updated (Figure 9(f)). The dashed line frame indicates that the process "CNT1+1" is not performed. Since the card CD3 has written the processed data DTc to the shared memory CM4, it updates the oldest information OLD in the shared memory CM3 to d and erases the data DTc (Figures 9(g) and (h)).

[0086] In cycle CYC2, as in cycle CYC1, card CD2 writes the processed data DTf to shared memory CM3 and then updates the oldest information OLD in shared memory CM2 to g (Figures 9(i) and (j)). Card CD2 reads the oldest information OLD=d from shared memory CM3, saves it as the current oldest information OLD, and erases the data DTf held in shared memory CM2 (Figures 9(k) and (l)). Card CD2 compares the previous oldest information OLD=c with the current oldest information OLD=d, and since they are not the same, confirms that the oldest information OLD has been updated (Figure 9(m)).

[0087] The card CD3 reads the oldest data DTd from the shared memory CM3, processes it, and writes the processed data DTd to the shared memory CM4 (FIG. 9(n)). The card CD3 updates the oldest information OLD of the shared memory CM3 to e, and erases the data DTd (FIGS. 9(o) and (p)).

[0088] In cycle CYC3 in Fig. 10, card CD3 fails after reading the oldest data DTi from shared memory CM3 (Fig. 10(a)). Card CD2 cannot recognize the failure of card CD3, and therefore operates in the same manner as in Fig. 9. Note that in Fig. 10, access to shared memory CM2 by card CD2 is omitted.

[0089] The card CD2 reads out the oldest information OLD=e from the shared memory CM3 and stores it as the current oldest information OLD (FIG. 10(b)). The card CD2 compares the previous oldest information OLD=d with the current oldest information OLD=e, and since they are not the same, it confirms that the oldest information OLD has been updated (FIG. 10(c)).

[0090] The standby card CD3S compares the failure counter CNT1=0 read from the shared memory CM3 with the threshold value VT and determines that it is smaller than the threshold value VT (FIG. 10(d)). For example, the threshold value VT is n (an integer equal to or greater than 2). Although not shown in FIG. 9, the standby card CD3S compares the failure counter CNT1=0 read from the shared memory CM3 with the threshold value VT for each cycle CYC.

[0091] In cycle CYC4, card CD2 reads out the oldest information OLD=e that has not been updated due to a failure of card CD3, and saves it as the current oldest information OLD (Fig. 10(e)). Card CD2 compares the previous oldest information OLD=e with the current oldest information OLD=e, and since they are the same, determines that the oldest information OLD has not been updated (Fig. 10(f)). Note that if there is a possibility that the processing timing of cards CD2 and CD3 may differ within cycle CYC, card CD2 may determine that the oldest information OLD has not been updated if the previous and current oldest information OLD are the same for a predetermined number of consecutive times.

[0092] Since the oldest information OLD has not been updated, the card CD2 increments the value of the failure counter CNT1 in the shared memory CM3 and sets the failure counter CNT1 to 1 (FIG. 10(g)). The standby card CD3S compares the failure counter CNT1=1 read from the shared memory CM3 with the threshold value VT and determines that it is smaller than the threshold value VT (FIG. 10(h)).

[0093] After that, in cycles CYC5 (not shown) and after, card CD2 compares the previous oldest information OLD=e with the current oldest information OLD=e, and since they are the same, it sequentially increments the value of the failure counter CNT1. After that, in cycle CYCm, card CD2 determines that the previous oldest information OLD=e and the current oldest information OLD=e are the same, as in cycle CYC4 (FIGS. 10(i) and (j)). Card CD2 increments the value of the failure counter CNT1 and sets it to failure counter CNT1=n (FIG. 10(k)).

[0094] The standby card CD3S compares the failure counter CNT1=n read from the shared memory CM3 with the threshold value VT and determines that it is equal to or greater than the threshold value VT (FIG. 10(l)). Therefore, the card CD3S performs a process to switch its own card from the standby system to the active system. The standby card CD3S checks whether the oldest information OLD of all the active cards CD3 in the same stage STG has been updated, and detects a failure in the card CD3 whose oldest information OLD has not been updated.

[0095] 1 reads out the data DT from the shared memory CM4 in cycle CYC3, the rear stage device 220 in Fig. 1 updates the oldest information OLD and erases the data DT. As a result, in the shared memory CM4, the oldest information OLD=c, data DTc, DTd in cycle CYC3, the oldest information OLD=d, data DTd in cycle CYC4, and the oldest information OLD=-, returning to the initial state of no data in cycle CYC5.

[0096] In the cycle CYCm+1 in Fig. 11, the card CD3 that has been switched from the standby system to the active system performs the takeover process indicated by the dashed frame. The card CD3 reads data DTE corresponding to the oldest information OLD=e stored in the shared memory CM3 from the shared memory CM3, and processes the read data DTE (Fig. 11(a) and (b)).

[0097] 9(b) and (h), the card CD3 before the failure does not erase the data DT immediately after reading it from the shared memory CM3, but processes the data DT, writes it to the shared memory CM4, and then erases it. Therefore, the card CD3 that has been switched from the standby system to the active system can read the data DTe that was unprocessed before the failure from the shared memory CM3 and process it, thereby preventing malfunction of the information processing device 100.

[0098] After writing the processed data DTE to the shared memory CM4, the card CD3 updates the oldest information OLD of the shared memory CM3 to f and erases the data DTE (FIGS. 11(c), (d), and (e)).

[0099] Card CD2 writes the processed data DTj into shared memory CM3 and updates the oldest information OLD of shared memory CM2 to k (Fig. 11(f) and (g)). Card CD2 reads out the oldest information OLD=f from shared memory CM3, stores it as the current oldest information OLD, and erases the data DTj held in shared memory CM2 (Fig. 11(h) and (i)). Card CD2 compares the previous oldest information OLD=e with the current oldest information OLD=f, and since they are not the same, confirms that the oldest information OLD has been updated (Fig. 11(j)).

[0100] Fig. 12 shows an example of the overall operation of the data management method of the information processing device of Fig. 7. The flow shown in Fig. 12 is a summary of the operations of Fig. 9 to Fig. 11, and each card CD of the active system and a card CD switched from the standby system to the active system perform the same operation. The following describes the operation of the active card CD2 and the operation of the standby card CD3S before it switches to the active system.

[0101] Card CD2 is initialized and starts operation when information processing device 100 is started or reset (FIG. 12(a)). Cards CD3 and CD3S are also initialized and start operation when information processing device 100 is started or reset. FIG. 12 shows the state after a period of operation from the initial setting. Card CD2 reads out the oldest data DTe corresponding to the oldest information OLD=e from shared memory CM2 (FIG. 12(b)).

[0102] Here, the card CD2 does not erase the data DTE held in the shared memory CM2 until it completes processing the read data DTE and writes it to the shared memory CM3. If the card CD2 does not have the oldest data DT (i.e., there is no data to process), it waits until the oldest data DT is stored in the shared memory CM2 (FIG. 12(c)).

[0103] Card CD2 processes data DTE and writes the processed data DTE to shared memory CM3 (FIG. 12(d)). Card CD2 reads out the oldest information OLD=c from shared memory CM3 and stores it in its own card CD2's work memory or the like (FIG. 12(e)). Card CD2 updates the oldest information OLD of shared memory CM2 to f and erases data DTE held in shared memory CM2 (FIGS. 12(f) and (g)).

[0104] Card CD2 compares the previous and current oldest information OLD, and if they are the same, increments the value of failure counter CNT1 (Figures 12(h) and (i)). Card CD2 then repeats the operations of Figures 12(b) to 12(i). For example, if downstream card CD3 fails, the operation of card CD3 stops and the update operation of the oldest information OLD of shared memory CM3 stops. In this case, as shown in Figures 12(h) and (i), the previous and current oldest information OLD match, so the value of failure counter CNT1 is incremented.

[0105] The standby card CD3S repeatedly reads the failure counter CNT1 of the shared memory CM3 (FIG. 12(j)). When the value of the failure counter CNT1 becomes equal to or greater than the threshold value VT, the card CD3S determines that the active card CD3 has failed. The card CD3S takes over the information of the failed card CD3, and performs a process of switching its own card CD3S from the standby card to the active card, and performs initial settings for operating as the active card (FIG. 12(k)).

[0106] The card CD3 that has become the active system takes over the processing that the card CD3 before the failure was performing. Here, after each card CD reads the data DT to be processed from the upstream shared memory CM, it does not erase the data DT held in the upstream shared memory CM until it completes processing the read data DT and writes it to the downstream shared memory CM. This allows the card CD3 that has switched from the standby system to the active system to sequentially perform processing of the data DTc, DTd, and DTa that were unprocessed before the failure without missing or overlapping, thereby preventing malfunction of the information processing device 100.

[0107] As described above, this embodiment can also provide the same effects as the above-mentioned embodiment. For example, by making the active card CD and the standby card CDS mounted on the information processing device 100A function as fault monitoring devices, it is possible to replace a faulty card CD on a card-by-card basis.

[0108] Compared to a case where a failure is detected by a CPU, the standby system can be switched to the active system at high speed, and loss or duplication of data DT due to switching can be prevented. As a result, even if the active system card CD fails, the data transfer rate from the pre-stage device 210 to the post-stage device 220 can be prevented from exceeding the specifications, and the real-time performance of the system 300 can be maintained.

[0109] Since so-called N:1 redundancy can be realized by using one standby card CDS, the number of standby card CDSs mounted can be minimized, and the power consumption of the information processing device 100 can be reduced.

[0110] The features and advantages of the embodiments will be apparent from the above detailed description. It is intended that the claims cover the features and advantages of the embodiments as described above without departing from the spirit and scope of the claims. In addition, any improvements and modifications can be easily made by a person having ordinary skill in the art. Therefore, it is not intended to limit the scope of the inventive embodiments to the above, and appropriate improvements and equivalents within the scope of the embodiments disclosed can be used. [Explanation of symbols]

[0111] 100 Information processing device 210 Pre-stage equipment 220 Post-stage equipment 300 Systems CD operation card CD1 (CD11-CD1N) card (operational system) CD2 (CD21-CD2N) card (operational system) CD3 (CD31-CD3N) card (operational system) CDS, CD1S, CD2S, CD3S cards (standby system) CM (CM1, CM2, CM3, CM4) Shared memory CNT1, CNT2 fault counters CP Supervisory and control path DP Data Communication Path DPRC Data Processing Unit DT Data NEW Latest information OPCNT Operation Control Unit OLD Oldest information STG1-STG3 Stages VT Threshold

Claims

1. A fault monitoring device mounted on an information processing device having a shared memory, a first device for sequentially writing data to the shared memory, and a second device for sequentially reading data from the shared memory, an oldest information storage unit that stores oldest information indicating the oldest data that has not been read by the second device among the data written to the shared memory by the first device; a oldest information updating unit that updates the oldest information stored in the oldest information storage unit when the data read from the shared memory by the second device is output to a device subsequent to the second device; a first failure detection unit that detects a failure of the second device when the oldest information held in the oldest information holding unit is not updated even if data is written to the shared memory a predetermined number of times by the first device; A fault monitoring device having the same.

2. The writing of data to the shared memory by the first device and the reading of data from the shared memory by the second device are repeatedly performed at the same cycle; the shared memory includes a first counter whose counter value is updated when the oldest information is not updated in response to writing of data to the shared memory by the first device; the first failure detection unit detects a failure of the second device when an increment of the counter value of the first counter becomes equal to or greater than a threshold value; The fault monitoring device according to claim 1 .

3. a plurality of operational systems each including the first device and the second device; a standby system including a third device having the first failure detection unit; the shared memory is provided in common to the plurality of operational systems; the oldest information storage unit and the first counter are provided in the shared memory for each of the plurality of operational systems, the oldest information update unit is included in each of the second devices of the plurality of active systems, When the first failure detection unit detects a failure in any of the second devices, the third device switches to an active system in place of the failed second device. The fault monitoring device according to claim 2 .

4. the second device processes the data read from the shared memory, outputs the processed data, and then erases the unprocessed data stored in the shared memory. The fault monitoring device according to claim 1 .

5. an updated information storage unit that stores updated information indicating the latest data written to the shared memory by the first device; a latest information update unit that updates the latest information stored in the latest information storage unit when data is written to the shared memory by the first device; a second failure detection unit that detects a failure of the first device when the latest information stored in the latest information storage unit is not updated even if the data is read from the shared memory a predetermined number of times by the second device; 2. The fault monitoring device according to claim 1, further comprising:

6. The writing of data to the shared memory by the first device and the reading of data from the shared memory by the second device are repeatedly performed at the same cycle; the shared memory includes a second counter whose counter value is updated when the latest information is not updated in response to the second device reading data from the shared memory; the second failure detection unit detects a failure of the second device when an increment of the counter value of the second counter becomes equal to or greater than a threshold value. The fault monitoring device according to claim 5.

7. a plurality of operational systems each including the first device and the second device; a standby system including a third device having the second failure detection unit; the shared memory is provided in common to the plurality of operational systems; the latest information storage unit and the second counter are provided in the shared memory for each of the plurality of operational systems, the latest information update unit is included in each of the first devices of the plurality of active systems, When the second failure detection unit detects a failure in any of the first devices, the third device switches to an active system in place of the failed first device. The fault monitoring device according to claim 6.

8. 1. A fault monitoring method for a fault monitoring device mounted on an information processing device having a shared memory, a first device for sequentially writing data to the shared memory, and a second device for sequentially reading data from the shared memory, comprising: storing, in an oldest information storage unit included in the information processing device, oldest information indicating the oldest data that has not been read by the second device among the data written to the shared memory by the first device; an oldest information update unit included in the information processing device updates the oldest information held in the oldest information holding unit when data read from the shared memory by the second device is output to a device subsequent to the second device; a first failure detection unit included in the information processing device detects a failure of the second device when the oldest information held in the oldest information holding unit is not updated even if data is written to the shared memory a predetermined number of times by the first device; Failure monitoring method.

Citation Information

Patent Citations

  • Method and device for backup

    JP1994175788A

  • Fault tolerant system, operation system device, standby system device, fail-over method, and fail-over program

    JP2016110173A