Fault processing device, processing method for fault processing device, and program

Abstract

To provide a fault processing device that can automate fault recovery for a device to be monitored while reducing manual work.SOLUTION: A fault processing device includes: a first processing unit configured to detect the occurrence of a fault in a device to be monitored and generate fault information about the fault in the device to be monitored; a second processing unit configured to receive, from an operator terminal, one or more action commands executed to address the fault in the device to be monitored based on the fault information; a third processing unit configured to store the one or more action commands received by the second processing unit in the order in which they were executed; and a fourth processing unit configured to extract an action command that is effective for the fault in the device to be monitored from the one or more action commands stored by the third processing unit and store the extracted effective action command in association with the fault information.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a fault processing device that executes a fault handling process when a fault occurs. [Background technology]

[0002] As a failure recovery technology for a failure processing device, there is a device that automatically executes failure recovery. For example, Patent Document 1 discloses an information processing device that stores failure information about failures that occurred in the past in association with recovery procedures that were executed for the failures, and when a new failure occurs, if there is a similar failure in the stored past failure cases, reads out and automatically executes the recovery procedure corresponding to the similar failure. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2001-034509 Summary of the Invention [Problem to be solved by the invention]

[0004] However, with conventional failure recovery techniques such as that disclosed in Patent Document 1, if there are no accumulated failure cases similar to the newly occurring failure, maintenance personnel must be present on-site, which incurs the cost of manual labor. Also, if multiple failures occur simultaneously, it can take a long time to resolve the failures due to limited human resources.

[0005] Therefore, an object of the present invention is to address such issues and provide a fault processing device, processing method, and program that can automate fault recovery of monitored equipment while reducing the amount of manual work required. [Means for solving the problem]

[0006] The fault processing device of the present invention is characterized by comprising: a first processing unit that detects the occurrence of a fault in a monitored device and generates fault information about the fault in the monitored device; a second processing unit that receives from an operator terminal one or more treatment commands that have been executed to deal with the fault in the monitored device based on the fault information; a third processing unit that stores the one or more treatment commands received by the second processing unit in the order in which they were executed; and a fourth processing unit that extracts a treatment command that is effective for the fault in the monitored device from the one or more treatment commands stored in the third processing unit, and stores the extracted effective treatment command in association with the fault information.

[0007] The processing method of the present invention is a processing method of a fault processing device, characterized in that the fault processing device executes the following steps: a first step of detecting the occurrence of a fault in a monitored device and generating fault information about the fault in the monitored device; a second step of receiving from an operator terminal one or more treatment commands executed to deal with the fault in the monitored device based on the fault information; a third step of storing the one or more treatment commands received in the second step in the order of their execution; and a fourth step of extracting a treatment command that is effective for the fault in the monitored device from the one or more treatment commands stored in the third step, and storing the extracted effective treatment command in association with the fault information.

[0008] The program of the present invention is a program used in a fault processing device, and is characterized in that it causes a computer to function as a first processing unit that detects the occurrence of a fault in a monitored device and generates fault information about the fault in the monitored device, a second processing unit that receives from an operator terminal one or more treatment commands that have been executed to deal with the fault in the monitored device based on the fault information, a third processing unit that stores the one or more treatment commands received by the second processing unit in the order in which they were executed, and a fourth processing unit that extracts a treatment command that is effective for the fault in the monitored device from the one or more treatment commands stored in the third processing unit, and stores the extracted effective treatment command in association with the fault information. [Effects of the Invention]

[0009] According to the fault processing device, fault processing method, and program of the present invention, a job for dealing with a fault in a monitored device can be automatically registered simply by the operator operating the worker terminal, and if the same or similar fault occurs in the monitored device thereafter, the registered job can be used to deal with the fault, thereby automating fault recovery of monitored devices while reducing human work. [Brief explanation of the drawings]

[0010] [Figure 1] 1 is a block diagram showing a schematic configuration of a communication system including a fault processing device according to the present invention; [Figure 2] 10 is a flowchart showing an operation of the fault processing device of FIG. 1 when a fault occurs in the monitoring server; [Figure 3] 10 is a flowchart showing the operation of the job management server of the fault processing device in FIG. 1 when checking the operation and implementing a temporary measure; [Figure 4] 10 is a flowchart showing a job registration operation of the job management server. [Figure 5] 2 is a flowchart showing a fault handling operation by an operator terminal of the communication system of FIG. 1; [Figure 6] 5 is a flowchart specifically showing the job registration operation of FIG. 4. [Figure 7] 10 is a flowchart showing a basic response when a failure occurs in a monitored device. DETAILED DESCRIPTION OF THE INVENTION

[0011] Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

[0012] 1 shows a communication system including a fault processing device 10 according to the present invention. In addition to the fault processing device 10, this communication system also includes a group of monitored devices 110 and an operator terminal 141.

[0013] The fault processing device 10 includes a monitoring server 101 , a group of monitored devices 110 , a job management server 121 , and a problem management and change management system 131 .

[0014] The monitoring server 101, monitored device group 110, job management server 121, problem management and change management system 131, and worker terminal 141 are connected to each other via an internal line 150 such as a LAN (local area network) line so that they can communicate with each other. The monitored device group 110 consists of multiple monitored devices 111, 112, ..., 11n, where n is an integer of 2 or greater.

[0015] The monitoring server 101 is a device consisting of a computer for monitoring and recording the operating status of each of the monitored devices 111 to 11n. Types of monitoring include alive monitoring (Ping), monitoring of CPU and memory resource usage, and log monitoring. When the monitoring server 101 detects a failure such as an abnormal operation of the monitored devices 111 to 11n, it sends failure information about the detected failure to the job management server 121.

[0016] Each of the monitored devices 111 to 11n is configured by a network device including various servers, other hardware, and the like.

[0017] The job management server 121 is a device consisting of a computer that receives fault information sent from the monitoring server 101, registers and analyzes the fault information, and if the fault matches a registered known fault, executes a pre-registered job. The job management server 121 has a job DB 122, in which known fault information is pre-registered as well as fault information received by the job management server 121.

[0018] The problem management and change management system 131 is a computer system that records failure information of each of the monitored devices 111 to 11n detected by the monitoring server 101 and the temporary measures that have been implemented.

[0019] The worker terminal 141 is a terminal for issuing instructions to the monitoring server 101, the monitored devices 111 to 11n, the problem management and change management system 131, and the job management server 121. The worker terminal 141 is, for example, a PC (personal computer).

[0020] The monitoring server 101, the job management server 121, and the problem management and change management system 131 are provided with a support tool 200. The support tool 200 is a program for the above-mentioned functions, and also has other functions such as a function to send information to related parties, automatic registration in the problem management and change management system 131, and a search and matching engine for past cases.

[0021] Next, the fault handling operation of the fault processing device 10 according to the present invention will be described with reference to the flowcharts of FIGS.

[0022] When a failure (including an incident) occurs in a monitored device 11j (where 1≦j≦n) among the monitored devices 111 to 11n, the monitoring server 101 detects the failure in the monitored device 11j (step S101), as shown in Fig. 2. Then, the monitoring server 101 generates failure information about the detected failure and transmits the failure information to the job management server 121 (step S102). Steps S101 and S102 correspond to, for example, a first processing unit.

[0023] 3, the job management server 121 receives fault information and posts the fault information to the problem management and change management system 131 (step S201). The job management server 121 also searches the job DB 122 to determine whether the fault indicated in the fault information is a known case (fault) by the support tool 200 (step S202). If the search in step S202 determines that the fault is a known case, the job management server 121 executes a job registered as a temporary measure corresponding to the known case for the monitored device 11j (step S203). The job management server 121 records the execution result of the job in step S203 in the problem management and change management system 131 (step S204).

[0024] If, as a result of the search in step S202, the fault indicated in the fault information is not a known case, that is, for a fault for which no corresponding job is registered, the operator operates the worker terminal 141 to register a job corresponding to the fault in the job DB 122 (step S301). The job management server 121 can register the job in the job DB 122 in response to an input operation by the operator to the worker terminal 141. By registering the job in step S301, it is possible to automate processing when the same fault occurs thereafter.

[0025] When registering a job, an operator uses the operator terminal 141 to trace and register the job that has dealt with the failure.

[0026] In tracing, as shown in FIG. 5, an operator inputs data to the operator terminal 141, which then executes one or more corrective commands for the failure in sequence on the monitored device 11j (step S311). The operator terminal 141 determines whether an error message has occurred for each executed corrective command (step S312). A corrective command for which an error message has occurred is determined to be an invalid corrective command. The operator terminal 141 records "invalid" for a corrective command for which an error message has occurred (step S313). On the other hand, a corrective command for which no error message has occurred is determined to be a valid corrective command. After executing step S313, the operator terminal 141 determines whether execution of all corrective commands has been completed (step S314). If execution of all corrective commands has not been completed, the operator terminal 141 proceeds to step S311 and executes the next corrective command. If execution of all corrective commands has been completed, the operator terminal 141 has finished tracing, and therefore transmits one or more corrective commands in sequence to the job management server 121 to register the job (step S315). In step S315, the result of determining whether each treatment command is valid or invalid is associated with each treatment command in the order of execution and transmitted.

[0027] 6, the job management server 121 receives one or more action instructions and the associated valid / invalid determination results transmitted from the operator terminal 141 (step S321), and stores the one or more action instructions and the associated valid / invalid determination results in an internal memory (not shown) of the job management server 121 in the order in which the action instructions were executed (step S322). Step S321 corresponds to, for example, a second processing unit, and step S322 corresponds to, for example, a third processing unit. The job management server 121 extracts valid action instructions in the order in which the action instructions were executed from all the action instructions stored in step S322, associates the extracted action instructions with fault information, and stores the extracted action instructions in the job DB 122 (step S323). Step S323 corresponds to, for example, a fourth processing unit. The one or more action instructions extracted and stored in step S323 become the job corresponding to the fault referred to in step S301.

[0028] In this way, a job for dealing with a fault in the monitored device 11j can be automatically registered in the job DB 122 simply by the operator operating the worker terminal 141, and if the same or similar fault occurs in any of the monitored devices 111 to 11n thereafter, the registered job can be used to deal with the fault, thereby automating the recovery of faults in the monitored devices while reducing human work.

[0029] Furthermore, the worker terminal 141 and the job management server 121 may perform the following processing.

[0030] In step S315 of FIG. 6, the worker terminal 141 may further sequentially send one or more treatment commands and information linked to the one or more treatment commands to the job management server 121 to register the job.

[0031] In step S323 of FIG. 6, the job management server 121 may further associate the extracted treatment command with information linked to the extracted treatment command and store the association information in the job DB 122.

[0032] Here, examples of information linked to the treatment command include (1) a screenshot of the worker terminal 141, (2) audio / video recordings made when responding to the failure, and (3) text converted from audio recordings made when responding to the failure.

[0033] The job contents include information for identifying the failure, such as a host name that can identify the monitoring target, a monitoring type, an error message, and the like, and a script that can be executed by the job management server 121 as the job to be performed.

[0034] Tasks that can be executed by the job management server 121 using the support tool 200, such as temporary measures to be taken when a failure occurs, are registered in the script. Examples of registered tasks include operations on monitored devices (restarting the device, acquiring log files, and checking whether services are running normally) and sending emails to the maintenance department and other relevant parties of the monitored devices. Specifically, there is a process for automatically creating and sending emails to relevant parties by clicking, for example, a "Generate Discovery Report Email" button in the navigation window on the support tool 200.

[0035] 7 shows a basic response flow when a failure occurs in a monitored device. In this response flow, when a failure occurs in a monitored device, first, the failure notification information is confirmed (step S401). The confirmed failure notification information is posted to the problem management and change management system (step S402). Then, the operating status of the monitored device where the failure occurred, obtained from the notification information, is confirmed (step S403). The confirmed operating status is entered into the problem management and change management system (step S404), and the details of the detected failure are reported to the relevant departments (step S405). Furthermore, a predetermined recovery measure is implemented for the monitored device where the failure occurred (step S406).

[0036] Next, a temporary solution corresponding to the detected failure is confirmed (step S407). Information about the confirmed temporary solution is entered into the problem management and change management system (step S408), and a response regarding the temporary solution is sent to the relevant department (step S409). Then, the temporary solution is implemented on the monitored device where the failure occurred (step S410), and the relevant department is notified that the temporary solution has been implemented (step S411).

[0037] Several steps in the response flow of FIG. 7 (especially steps S402, S404 to S406, and S407 to S410) have conventionally been performed manually, but the fault response operation of the fault processing device 10 according to the present invention makes it possible to automate most of the manual work, and also to achieve a reduction in time through automation. [Explanation of symbols]

[0038] 10. Fault processing device 101 Monitoring Server 110 Monitored Equipment 111~11n monitored devices 121 Job Management Server 122 Job DB 131 Problem Management and Change Control Systems 141 Worker terminal 150 internal lines 200 Support Tools

Claims

1. a first processing unit that detects the occurrence of a failure in a monitored device and generates failure information regarding the failure in the monitored device; a second processing unit that receives, from an operator terminal, one or more treatment commands that have been executed to deal with the failure of the monitored device based on the failure information; a third processing unit that stores the one or more treatment commands received by the second processing unit in the order of execution; a fourth processing unit that extracts a processing command that is effective for a failure of the monitored device from the one or more processing commands stored in the third processing unit, and stores the extracted effective processing command in association with the failure information; A fault processing device comprising:

2. The second processing unit receives the determination result of whether the one or more treatment commands are valid or not, which is associated with the one or more treatment commands; 2. The fault processing device according to claim 1, wherein the determination result is information in which it is determined whether each of the one or more processing commands is effective for the fault in the monitored device.

3. 3. The fault processing device according to claim 2, wherein the fourth processing unit determines whether the processing command is valid or not based on the result of the determination of whether the processing command is valid or not.

4. 4. The fault processing device according to claim 1, wherein the valid treatment command is a treatment command for which no error message has been generated.

5. a fifth processing unit that receives, from the operator terminal, at least one of video information, audio information, and text information generated from the audio information when the one or more treatment commands are executed; a sixth processing unit that manages the at least one piece of information and the effective treatment command in association with each other; 2. The fault processing device according to claim 1, further comprising:

6. the fourth processing unit has a database that stores the effective treatment command as a job in association with the fault information, 2. The fault processing device according to claim 1, further comprising a seventh processing unit that, when fault information identical or similar to the fault information generated by the first processing unit exists in the database, reads out the job associated with the fault information from the database and executes the read job on the monitored device.

7. A method for handling a faulty processing device, comprising: The fault processing device comprises: a first step of detecting an occurrence of a fault in a monitored device and generating fault information regarding the fault in the monitored device; a second step of receiving, from an operator terminal, one or more action instructions executed to deal with the failure of the monitored device based on the failure information; a third step of storing the one or more treatment instructions received in the second step in order of their execution; a fourth step of extracting a treatment command effective for the fault of the monitored device from the one or more treatment commands accumulated in the third step, and associating the extracted effective treatment command with the fault information and accumulating the same; A processing method characterized by carrying out the above.

8. A program used in a fault processing device, Computer, a first processing unit that detects the occurrence of a failure in a monitored device and generates failure information regarding the failure in the monitored device; a second processing unit that receives, from an operator terminal, one or more treatment commands that have been executed to deal with the failure of the monitored device based on the failure information; a third processing unit that stores the one or more treatment commands received by the second processing unit in the order of execution; A program characterized by functioning as a fourth processing unit that extracts an effective treatment command for a fault in the monitored device from the one or more treatment commands stored in the third processing unit, and stores the extracted effective treatment command in association with the fault information.

Images