Log management system and log management method
The log management system addresses log management challenges during device startup and operation by adapting filtering and output processes based on the management table's state, ensuring efficient log retention and preventing CPU overload, enabling centralized log verification and resource management.
Patent Information
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- HITACHI LTD
- Filing Date
- 2024-12-17
- Publication Date
- 2026-06-29
AI Technical Summary
Existing log management systems fail to manage logs during information processing device startup, leading to buffering issues, loss of logs during system failures, and CPU overload due to excessive log filtering, which interferes with device operation.
A log management system with a management table holding unit, input filter unit, buffer unit, and log output processing unit that adapt their operations based on the management table's state, allowing log filtering and output based on source and severity, with mechanisms to prevent overload and ensure log retention during startup and normal operation.
Enables efficient log management during device startup and operation, preventing CPU overload and ensuring log retention, even in system failures, allowing centralized log verification and efficient resource allocation.
Smart Images
Figure 2026106315000001_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to a log management system and a log management method.
Background Art
[0002] Patent Document 1 (Japanese Patent Application Laid-Open No. 2015-033148) discloses a log printing filter device that greatly improves the usefulness as a record by filtering and printing only necessary log information, and reduces the number of replenishments of limited printing media, paper, etc.
[0003] The log printing filter device includes a receiving unit that receives event information, a data buffer unit that stores the received event information, a log printing filter condition acquisition unit that acquires log printing filter conditions, a comparison operation unit that compares the event information stored in the data buffer unit with the acquired log printing filter conditions and performs a predetermined logical operation process, a log printing data shaping unit that shapes log printing data based on the operation process result, and a transmission unit that transmits the shaped log printing data.
Prior Art Documents
Patent Documents
[0004]
Patent Document 1
Summary of the Invention
Problems to be Solved by the Invention
[0005] Even if a log occurs during the startup of an information processing device, filtering processing cannot be performed, and there is a limitation on the output destination, so the logs cannot be buffered in chronological order and managed centrally. Even if buffering is possible, when a system failure occurs and the information processing device fails to start up, the logs in the buffer disappear, and there is a problem that the logs during startup cannot be confirmed.
[0006] Furthermore, if a large volume of logs is generated, filtering can overload the CPU, potentially making it difficult for the information processing system to operate. Even if filtering is successful, if the number of logs passing through exceeds the design limit, there is a problem of insufficient storage space in the destination tables or files.
[0007] The objective of the present invention is to provide a log management system that enables logging to high-level I / O even if the logs occur during the startup of an information processing device, allows for log verification even in the event of startup failure, and prevents logs from interfering with the operation of the information processing device. [Means for solving the problem]
[0008] The above problem is solved by a log management system comprising: a management table holding unit that holds a management table storing processing methods for filtering generated logs; an input filter unit that performs log filtering based on the processing methods held in the management table; a buffer unit that holds the logs filtered by the input filter unit; a log output processing unit that performs log output processing based on the processing methods held in the management table; and a management table state determination unit that determines the state of the management table holding unit, wherein the operation of at least one of the input filter unit and the log output processing unit is changed based on the determination of the management table state determination unit. [Effects of the Invention]
[0009] Log information generated during the startup of the information processing device can also be output according to its source and severity. [Brief explanation of the drawing]
[0010] [Figure 1] This is an example of a system configuration diagram in an embodiment. [Figure 2] This is an example of a block diagram of a log management system in operation, where the information processing device in the embodiment is running. [Figure 3] This is an example of a block diagram of a log management system in operation, where the information processing device in the embodiment is running. [Figure 4] This is an example of a block diagram showing the configuration of the input filter section in the embodiment. [Figure 5] This is an example diagram showing the hardware configuration of the log management system in the embodiment. [Figure 6] This is an example diagram showing an example of the settings for the management table in the embodiment. [Figure 7] This is an example of a diagram showing an example of log generation in an embodiment. [Figure 8] This is an example of a log output to high-level I / O in the implementation example. [Figure 9] This is an example of logs output to low-level I / O in the implementation example. [Figure 10] This is a state machine diagram showing the state transitions of the information processing device in the embodiment. [Figure 11] This is an example flowchart showing the processing flow of the input filter section in the embodiment. [Figure 12] This is an example flowchart showing the processing flow of the log output processing unit in the embodiment. [Figure 13] This is an example flowchart showing the detailed processing flow of the traffic monitoring system related to the embodiment. [Modes for carrying out the invention]
[0011] Hereinafter, embodiments of the present invention will be described with reference to the drawings. In each figure used to illustrate the embodiments, the same names and reference numerals will be used for identical components as much as possible, and repeated explanations will be omitted.
[0012] The present invention is not limited to the embodiments described below, but includes various modifications and equivalent configurations within the spirit of the attached claims. For example, the embodiments described above are explained in detail for clarity, and the present invention is not necessarily limited to those having all the configurations described.
[0013] In addition, the processing units and processing modules described in the embodiments may be implemented in hardware by designing some or all of them, for example, using an integrated circuit, or may be implemented in software by a processor interpreting and executing a program that realizes each function.
[0014] The information described in the embodiments may be a table or a database (DB), or may be data stored in the main memory.
[0015] Hereinafter, embodiments for implementing the present invention will be described with reference to the drawings. Note that the following embodiments do not limit the invention according to the claims, and not all combinations of features described in the embodiments are essential for the solution of the invention.
[0016] FIG. 1 is an example of a system overall configuration diagram in an embodiment. By connecting the log management system independently from other subsystems such as IPL (Initial Programing Load) 2, OS (Operating System) 3, driver 4, and application 5 in this way, all logs in the information processing device output from IPL, OS, driver, and application can be centrally managed, and important logs can be output to various high-level I / Os and low-level I / Os of I / O 6.
[0017] FIG. 2 is an example of a block diagram of the log management system while the information processing device in the embodiment is operating.
[0018] The log management system while the information processing device is operating assumes that logs occur multiply, buffers them in the buffer unit 102 in time series, and outputs each log generated by the log source 110 to any I / O such as network communication 112, file system 108, and serial communication 109.
[0019] Since the buffer capacity is limited, an input filter unit 101 is provided before the buffer unit 102 to select logs based on the priority of the logs registered in the management table, and duplicate logs and logs with low priority are discarded 111.
[0020] Alternatively, when outputting logs that have been stored in the buffer unit 102, the log output processing unit 103 may output them based on the priority of the logs registered in the management table.
[0021] The management of the management table holding unit 104 is handled by a file system, a database, or any similar storage unit.
[0022] Log buffering and output based on the priority of logs registered in such a management table presupposes that log priorities are registered in the management table. Therefore, the input filter unit 101 and the log output processing unit 103 must query the management table status determination unit 107 for the status of the management table holding unit, and process the information after confirming that the information in the management table holding unit is available.
[0023] Figure 3 shows an example block diagram of the log management system while the information processing device in the embodiment is running. The "X" marks indicate that referencing and data exchange are not possible.
[0024] While the information processing device is running, the management table status determination unit 107 determines that the management table holding unit 104 is uninitialized, and prevents the input filter unit 101 from accessing the management table holding unit 104. The input filter unit 101 does not perform filtering and instead stores the logs received from the log source 110 in the buffer unit 102.
[0025] By preventing the log output processing unit 103 from accessing the buffer unit 102 and thus not receiving logs, logs are accumulated in the buffer unit 102. Alternatively, the log output processing unit 103 may be prevented from accessing the management table holding unit 104.
[0026] Figure 4 is an example of a block diagram showing the configuration of the input filter section in the embodiment.
[0027] When a large amount of logs pass through the input filter unit 101, the accumulation of logs in the buffer unit 102 and the output of logs to each I / O can place a load on the CPU, making it difficult for the information processing device to operate.
[0028] In such cases, a log management method is needed that solves the problem of difficulty in fault analysis using logs and prevents the log management system itself from becoming overloaded even when a large amount of logs are generated.
[0029] As shown in Figure 4, the system includes a traffic monitoring unit 502 that monitors the amount of logs flowing into the input filter unit 101, a suppression status management unit 503 that manages logs whose output is suppressed, and an output stop processing unit 504 that stops log output by referring to the suppression status management unit 503.
[0030] This configuration provides a traffic monitoring function that detects anomalies in log inflow by monitoring the log inflow rate per hour and comparing it with a predetermined threshold, thereby stopping log acceptance.
[0031] The predetermined threshold is an arbitrary setting of the amount of inflow that allows the system to operate. The suppression state management unit 503 holds information indicating what type of log output should be stopped.
[0032] For example, the source, severity, and conditions for stopping log output are associated with the log and stored. Furthermore, the information indicating which type of log output should be stopped is updated by the traffic monitoring unit 502.
[0033] The output stop processing unit 504 refers to the information held by the suppression state management unit 503 and selects which logs to include.
[0034] In addition to the above, if the amount of log output passing through the input filter unit 101 exceeds the design value, problems such as insufficient storage table or file capacity may occur. To solve this problem, a log management method is provided that visualizes the amount of log passing through using statistical methods.
[0035] As shown in Figure 4, the system includes a statistical processing unit 505 that acquires log transit information from the traffic monitoring unit 502 and visualizes it using statistical methods. By outputting the number of logs per hour and per log source from an output unit (not shown) to the user, the system can check the trend of log transit volume.
[0036] The information collected here can be used, for example, to inform constants such as the storage table and file capacity, which can then be used to modify resources related to logging.
[0037] In this embodiment, even if the information processing device terminates abnormally during startup and cannot transition during operation, there is a means to enable the output of a log of the system failure cause to the low-level I / O 106.
[0038] The system has mechanisms to prevent the logging system itself from becoming overloaded when a large volume of logs are generated.
[0039] In addition to the above, a means of visualizing the amount of data passing through the log using statistical methods may also be provided.
[0040] Figure 5 is an example of a diagram showing the hardware configuration of the log management system in the embodiment.
[0041] The log management system 1 is implemented on a computer that includes a processor 301 such as a CPU (Central Processing Unit) or GPU (Graphics Processing Unit) that controls the entire log management system 1, memory 302 such as ROM (Read Only Memory) or RAM (Random Access Memory) that stores various processing programs for realizing the functions of the log management system 1, external storage devices 303 such as an HDD (Hard Disk Drive) or SSD (Solid State Drive), input / output devices 304 such as a keyboard, mouse, or touch panel, and a network interface 305 such as a NIC (Network Interface Card).
[0042] The processor 301 executes various processing programs stored in memory 302 while referring to data stored in external storage device 303, thereby realizing the functions of the learning data recommendation system.
[0043] Some or all of these programs may be introduced from other devices via non-temporary storage media or communication lines, or they may be pre-stored. In this embodiment, an example of implementation using a standalone computer is described, but it may also be implemented using a cloud service that provides computing resources.
[0044] Figure 6 is an example diagram showing the configuration of a management table in an embodiment. The sources of events such as KERNEL, DRIVER, and APPLICATION are associated with the output destinations such as FILE and SERIAL.
[0045] Severity levels can be specified for each source and destination, ranging from DEBUG to EMERGENCY (eight levels). Logs of the checked severity levels are permitted to be output to that destination. FILE is a high-level I / O destination, while SERIAL is a low-level I / O destination.
[0046] In addition, if the system also holds information indicating the startup status of the file system and the initialization status of high-level I / O, the input filter unit 101 and the log output processing unit 103 can select an output destination based on this information.
[0047] Information on the source of the incident and information on its severity only need to be retained. This information allows the input filter unit 101 and the log output processing unit 103 to stop outputting logs.
[0048] Figure 7 is an example of a diagram showing log generation examples in the embodiment. Logs 1 to 7 are logs generated while the information processing device is running. When these logs are generated, the management table status determination unit 107 indicates that the management table holding unit 104 is unavailable, so the input filter unit 101 stores the logs in the buffer unit 102 without performing filtering based on the source and severity.
[0049] When the status of the information processing device transitions to operational 802, the log is retrieved from the buffer unit 102, and output is output after filtering based on the source and severity, which was not performed by the output destination distribution and input filter unit 101.
[0050] Logs 8-11 are generated during operation. During operation, the management table status determination unit 107 indicates that the management table holding unit 104 is available, so the input filter unit 101 performs filtering based on the source and severity.
[0051] Subsequently, the data is routed through the buffer unit 102 and then output to the log output processing unit 103, which then distributes the data to the appropriate output destination.
[0052] Figure 8 shows an example of logs output to low-level I / O in the embodiment.
[0053] For low-level I / O such as SERIAL output, high-severity logs set to EMERGENCY, such as kernel panics, storage failures, and application memory exhaustion, are output. Low-severity logs set to DEBUG are not output.
[0054] Figure 9 shows an example of logs output to high-level I / O in the embodiment.
[0055] For high-level I / O such as FILE output, not only are severe logs designated as EMEREGENCY, such as kernel panics, output, but also less severe logs designated as DEBUG, such as changes in variable values.
[0056] Figure 10 is a state machine diagram showing the state transitions of the information processing device in the embodiment. While in operation, device 802 can output logs to the file system.
[0057] However, if a log is generated in the startup 801, the management table holding unit 104 cannot be referenced, and high-level I / O is also uninitialized, so the input filter unit 101 does not perform filtering and instead accumulates the log in the buffer unit 102. After transitioning to the operational 802 state, the log output processing unit 103 performs the filtering that was not performed by the input filter unit 101 and outputs the log.
[0058] Figure 11 is an example flowchart showing the processing flow of the input filter unit in the embodiment. First, a log output request is received from the log source 110 (step 900). Information indicating whether the management table holding unit 104 has been initialized is received from the management table status determination unit 107, and it is determined whether the management table has been initialized (step 901).
[0059] If the management table holding unit 104 has been initialized, the information in the management table holding unit 104 is used to filter the target log based on its origin and severity, determine whether output is necessary and where it should be output (step 902), and store it in the buffer unit 102 (step 903).
[0060] If the management table holding unit 104 has been initialized, the system may determine that the OS is running and begin receiving logs from the OS. In systems where the OS cannot output log output to low-level I / O 106 while the OS is running, this can reduce the load on the log management system.
[0061] If the management table holding unit 104 is not initialized, information indicating that filtering is required during log output processing (hereinafter referred to as a filter flag) is added to the log header, etc. (step 904).
[0062] Next, logs are filtered based on thresholds determined based on the set severity level (step 905), and the filtered logs are output to low-level I / O (step 906).
[0063] Finally, in step 905, all logs are stored in a buffer, regardless of whether they were output to low-level I / O or not (step 903).
[0064] Figure 12 is an example flowchart showing the processing flow of the log output processing unit in the embodiment. First, information indicating whether the management table holding unit 104 has been initialized is read from the management table status determination unit 107 (step 920).
[0065] If the management table holding unit 104 is not initialized, the system waits until initialization is complete. If the management table holding unit 104 is initialized, the system reads the log from the buffer unit 102 and determines whether it has been filtered using the filter flag (step 922).
[0066] If a filter flag exists, i.e., if the log requires filtering, the management table holding unit 104 is referenced, and filtering is performed on the read log to determine whether output is necessary and where it should be output based on its source and severity (step 923).
[0067] Based on whether or not log output is necessary and the output destination, the read log is processed for output (step 925).
[0068] In this example, if the management table has been initialized, the input filter unit 101 and the log output processing unit 103 perform filtering and output processing based on the log output destination and log importance held in the management table.
[0069] It is not necessary to perform processing in both the input filter unit 101 and the log output processing unit 103. The input filter unit 101 may perform filtering when the management table is initialized, and the log output processing unit may perform output processing when the management table is not initialized.
[0070] Furthermore, when the log output processing unit performs output processing, if the log management system is connected to low-level I / O 106, it may output a signal to low-level I / O 106 indicating the occurrence of a log based on its severity.
[0071] Figure 13 is an example of a flowchart showing the detailed processing flow of the traffic monitoring system related to the embodiment. This is an example of using source information when suppressing log output. First, the traffic monitoring unit 502 monitors the log inflow for each log source and stores the log inflow for each source and time period in the statistical processing unit 505 (step 950).
[0072] The traffic monitoring unit 502 determines whether the inflow volume from the target log source exceeds the upper limit (step 951), and the output stop processing unit 504 stops outputting logs from the target source if the log inflow volume exceeds the upper limit (step 952). Information indicating that output has been stopped is stored in the suppression status management unit 503.
[0073] The traffic monitoring unit 502 refers to the suppression status management unit 503 and, if log output has already been suppressed, determines whether the release criterion, which is the number of logs within a predetermined time, has been met (step 953). If the release criterion has been met, the output stop processing unit 54 releases the output stop (step 954). The process returns to step 950 and the above process is repeated.
[0074] This makes it possible to reduce the processing load by suppressing or discarding logs from a specific source, even if a large volume of logs are generated there. As a result, even when a large volume of logs are generated, it is possible to prevent the log management system itself from becoming overloaded.
[0075] Furthermore, the statistical processing unit 505 receives the data monitored in step 950, records the number of log occurrences for each source and severity level, and visualizes it in graphs, etc. This makes it possible to improve the reliability of each system during the development and testing phases.
[0076] Therefore, by using statistical methods to visualize the data, it is possible to identify changes in resources related to logs.
[0077] As a result, a large amount of logs passing through the input filter unit 101 is generated, solving the problem of the information processing device becoming difficult to operate when the CPU is overloaded. This enables the realization of a log management system that does not interfere with the operation of the information processing device due to logs.
[0078] With the above configuration, the input filter unit 101 and the log output processing unit 103 change their processing operation based on the information from the management table status determination unit 107, so that logs generated during startup of the information processing device can be saved based on their source and severity, just like logs generated during operation.
[0079] In this embodiment, logs were filtered based on the source and severity, but logs may also be filtered using at least one arbitrary piece of information, not limited to the source and severity.
[0080] Furthermore, with the above configuration, even if the information processing device terminates abnormally during startup and cannot transition to operational mode, the user can collect logs of the cause of the failure by referring to low-level I / O 106.
[0081] In step 901, the management table state determination unit 107 performed a process to check the initialization state of the management table holding unit 104, but various implementation forms are possible for this step.
[0082] In order for log output to be performed correctly while the information processing device is running, the initialization of the management table holding unit 104 and the high-level I / O 105 must be completed in the configuration of this embodiment.
[0083] Therefore, if the high-level I / O 105 is initialized after the management table holder is initialized, the initialization status of the high-level I / O 105 may be checked. Furthermore, both the initialization status of the management table holder and the initialization status of the high-level I / O may be checked.
[0084] Step 901 can be rephrased as a step to determine the state of the management table holding unit 104. Furthermore, it can be rephrased as a step to determine whether or not log output can be performed normally while the information processing device is running.
[0085] Next, the effects of this embodiment will be described. The first effect of this embodiment is that logs can be output under the same conditions, regardless of whether the information processing device is being started or running, according to the information from the management table status determination unit 107.
[0086] Furthermore, a second effect of this embodiment is that it solves the problem of the conventional technology in which, when a system failure occurs and the information processing device fails to start, the logs in the buffer are lost, making it impossible to check the logs of the system failure cause that occurred during startup.
[0087] In this embodiment, by using low-level I / O206 in combination, it is possible to log system failure factors that occur during the startup of the information processing device.
[0088] Here, "Starting up" 801 can be interpreted as "during OS startup," and "In operation" 802 as "after OS startup is complete." Furthermore, as a result of the above operations, the result of the management table state determination unit 207 determining the state of the management table holding unit 204 will differ during OS startup compared to after OS startup is complete.
[0089] In the above embodiment, the operation of both the input filter unit 101 and the log output processing unit 103 was changed. However, the operation of only the input filter unit 101 may be changed, and log output may be stopped in the input filter unit 101. Alternatively, the operation of only the log output processing unit 103 may be changed, and log output may be stopped in the log output processing unit 103.
[0090] Therefore, this embodiment is characterized in that the operation of at least one of the input filter unit 101 and the log output processing unit 103 is changed.
[0091] Furthermore, in the above embodiment, the determination is made based on both the startup state of the file system and the initialization state of high-level I / O such as the protocol stack, but it is also acceptable to make the determination based on either one of them.
[0092] Furthermore, in the above embodiment, a filter flag was used to determine whether or not filtering processing is necessary in the log output processing unit, but filtering processing may be performed continuously without making such determination.
[0093] Even when the information processing device is in a startup state, filtering and output to high-level I / O are now possible, just as when it is in operation.
[0094] Traditionally, due to the limitation that high-level I / O could not be used for logs of the operating system and other information processing devices while they were running, logs were output separately for each subsystem. However, as shown in Figure 1, by centrally managing log information from other subsystems with this log management system, it becomes possible to handle all logs chronologically, resulting in the efficient collection of information related to the operation of the information processing device, such as operating status and failure occurrences. [Explanation of Symbols]
[0095] 1. Log Management System 2 IPL 3 OS 4 drivers 5 Applications 101 Input Filter Section 102 Buffer section 103 Log output processing unit 104 Management Table Holding Section 105 High-level I / O 106 Low-level I / O 107 Management Table Status Determination Unit 108 file systems 109 Serial communication 110 Log source 111 Disposal 501 Filter section 502 Traffic Monitoring Department 503 Output Stop Processing Unit 504 Deterrence Status Management Department 505 Statistical Processing Unit
Claims
1. A management table holding unit that holds a management table that stores the processing method when filtering the generated logs, An input filter unit performs filtering of the log based on the processing method stored in the management table, A buffer unit that holds the log filtered by the input filter unit, A log output processing unit performs the log output processing based on the processing method stored in the management table, A log management system comprising a management table state determination unit that determines the state of the management table holding unit, and which modifies the operation of at least one of the input filter unit and the log output processing unit based on the determination of the management table state determination unit.
2. In the log management system described in claim 1, The log management system is connected to the OS (operating system). The management table status determination unit determines that the OS is running when it receives information from the management table holding unit indicating that the initial initialization of the management table is complete, and the log management system receives logs from the OS.
3. A log management system according to claim 2, When the management table status determination unit determines that the OS is running, The aforementioned log output processing unit is a log management system that changes the destination of the log output.
4. In the log management system according to claim 1 or claim 2, The input filter unit refers to the management table and stops outputting logs based on at least one of the information held in the management table: information indicating the source of the log and information indicating the severity of the log.
5. In the log management system described in claim 1, The management table status determination unit determines the status of the management table holding unit based on at least one of the information held by the management table, namely information indicating the startup status of the file system and information indicating the initialization status of high-level I / O.
6. A logging system according to claim 1, When the management table status determination unit determines that the state of the management table has been initialized, the input filter unit performs the filtering process on the log based on the information from the management table holding unit. A log management system in which, when the management table state determination unit determines that the state of the management table has not been initialized, the input filter unit does not perform the filtering process, and the log output processing unit refers to the management table in the management table holding unit and performs the filtering process on the log.
7. A log management system according to claim 1, The log management system is connected to low-level I / O. When the management table state determination unit determines that the state of the management table has not been initialized, The log output processing unit is a log management system that outputs a signal to low-level I / O indicating the occurrence of the log based on the severity of the log.
8. In the log management system described in claim 1, The aforementioned input filter unit includes a traffic monitoring unit that monitors the number of logs received, When the traffic monitoring unit determines that it has received a predetermined number of logs, A log management system equipped with an output stop processing unit that stops the output of logs.
9. In the log management system described in claim 1, The aforementioned input filter unit includes a traffic monitoring unit that monitors the number of logs received, A log management system equipped with an output unit that outputs the number of logs per hour per log source.
10. The management table storage unit stores a management table that stores the processing method used when filtering logs that have been generated. The input filter unit performs filtering of the log based on the processing method stored in the management table. The buffer unit holds the log that has been filtered by the input filter unit. The log output processing unit performs the log output processing based on the processing method stored in the management table. The management table status determination unit determines the status of the management table holding unit, A log management method that modifies the operation of at least one of the input filter unit and the log output processing unit based on the determination of the management table state determination unit.