Log transmission control device, log transmission control method, and program

Abstract

This log transmission control device includes a transmission control unit configured to control the transmission of logs collected from a system to be monitored to nodes that execute predetermined processing on the logs. When congestion occurs with respect to the predetermined processing, the transmission control unit is configured to increase the nodes that are transmission destinations of the logs having a relatively high priority among the logs in comparison to a case in which congestion does not occur, thereby reducing the possibility that the handling of a log having a relatively high priority is impaired.

Description

Log transmission control device, log transmission control method and program

[0001] The present invention relates to a log transmission control device, a log transmission control method, and a program.

[0002] Recently, system monitoring platforms using Filebeats, logstash, and Elasticsearch have become widely used (Figure 1). Filebeats collects logs from the log files of each monitored system (Target in Figure 1). Logstash converts the different log formats for each monitored system, enabling centralized management. Elasticsearch searches and analyzes the logs. Filebeats, logstash, and Elasticsearch work together as microservices to provide monitoring platform services.

[0003] Such monitoring infrastructure services have the following characteristics: (1) There are cases where real-time performance is required. Errors may require immediate automatic response processing or operator intervention. (2) There are cases where burst loads occur. There are cases where a problem occurs in the monitored object and a large amount of logs are sent out. These logs contain a mixture of highly important information, such as information about errors, and relatively less important information, such as debug information.

[0004] "Filebeat", [online], Internet <URL: https: / / www.elastic.co / jp / beats / filebeat> "logstash", [online], Internet <URL: https: / / www.elastic.co / jp / logstash / > "Elasticsearch", [online], Internet <URL: https: / / www.elastic.co / jp / elasticsearch / >

[0005] Scaling out is a commonly used load reduction method for dealing with burst loads, but scaling out is not instantaneous and often takes anywhere from tens of seconds to several minutes from start to completion (although this varies depending on the system). During this time, congestion occurs and real-time performance is impaired. There are also cases where events are missed due to internal event backlogs. This can delay automatic response processes and operator actions in response to errors, or can lead to errors not being noticed.

[0006] These problems are due to the contradictory characteristics of (1) "there are cases where real-time performance is required" and (2) "there are cases where burst loads occur."

[0007] The present invention has been made in view of the above points, and has an object to reduce the possibility that a response to a log with a relatively high priority will be missed.

[0008] In order to solve the above problem, the log transmission control device has a transmission control unit configured to control the transmission of logs collected from the system to be monitored to a node that performs a specified process, and the transmission control unit is configured to increase the number of nodes to which logs with relatively high priority are sent when congestion occurs in relation to the specified process, compared to when the congestion does not occur.

[0009] This reduces the possibility that a response to a log with a relatively high priority will be lost.

[0010] FIG. 1 is a diagram showing an example of the configuration of a conventional system monitoring infrastructure; FIG. 2 is a diagram showing an example of the configuration of a system monitoring infrastructure 1 in an embodiment of the present invention; FIG. 3 is a diagram showing an example of the hardware configuration of a computer functioning as a transmission control unit 12 in an embodiment of the present invention; FIG. 4 is a diagram showing an example of the functional configuration of the transmission control unit 12 in an embodiment of the present invention; FIG. 5 is a diagram for explaining a first pattern of behavior depending on the state of the high priority control unit 123a; FIG. 6 is a diagram for explaining a second pattern of behavior depending on the state of the high priority control unit 123a; FIG. 7 is a diagram for explaining a first pattern of behavior depending on the state of the low priority control unit 123b; FIG. 8 is a diagram for explaining a second pattern of behavior depending on the state of the low priority control unit 123b; and FIG. 9 is a diagram for explaining a third pattern of behavior depending on the state of the low priority control unit 123b.

[0011] In this embodiment, for a system monitoring infrastructure that has the characteristics of "there are cases where real-time performance is required" and "there are cases where burst loads occur," attention is paid to the fact that there are "relatively high priority logs" such as fatal error logs that must be notified to operators in real time, and "relatively low priority logs" such as simple Info that have a relatively high degree of leeway, and by performing priority control based on the priority of the logs, real-time performance of high priority logs is ensured when there is congestion in log processing.

[0012] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS An embodiment of the present invention will now be described with reference to the accompanying drawings. Fig. 2 is a diagram showing an example of the configuration of a system monitoring platform 1 according to an embodiment of the present invention.

[0013] 2, the system monitoring infrastructure 1 includes one or more log collection units 11, one or more transmission control units 12, one or more log processing units 13, one or more log analysis units 14, a resource monitoring unit 15, and a congestion control unit 16. Each of these units is realized by a process executed by a program (e.g., an application program) installed on one or more computers (whether physical machines or virtual machines) constituting the system monitoring infrastructure 1. For example, the log collection unit 11 may be realized by the same computer as the monitored system, or by a computer connected via a network to a computer functioning as a monitoring system. The transmission control unit 12 may be realized by a computer connected via a network to the computer functioning as the log collection unit 11. The log processing unit 13 may be realized by a computer connected via a network to the computer functioning as the transmission control unit 12. The log analysis unit 14 may be realized by a computer connected via a network to the computer functioning as the log processing unit 13. The resource monitoring unit 15 may be realized by a computer connected via a network to the computer functioning as the log processing unit 13. The congestion control unit 16 may be realized by a computer that functions as the resource monitoring unit 15 or a computer that is connected to the computer that functions as the resource monitoring unit 15 via a network.

[0014] Furthermore, each of the multiple log collection units 11, the multiple transmission control units 12, the multiple log processing units 13, and the multiple log analysis units 14 may be realized by a single computer or may be distributed across multiple computers.

[0015] The log collection unit 11 collects logs from the log files of each monitored system. For example, the log collection unit 11 may be realized using Filebeats.

[0016] The transmission control unit 12 controls the transmission of each log collected by the log collection unit 11 to the log processing unit 13 .

[0017] The log processing unit 13 is an example of a node that executes processing on logs. For example, the log processing unit 13 converts logs in different formats for each monitored system into a common format so that the logs can be centrally managed. For example, the log processing unit 13 may be realized using logstash.

[0018] The log analysis unit 14 searches and analyzes the logs processed by the log processing unit 13. For example, the log analysis unit 14 may be realized using Elasticsearch.

[0019] The resource monitoring unit 15 monitors (obtains) the usage status of the resources of the log processing unit 13 (resources of the computer functioning as the log processing unit 13 (such as CPU resource usage and queue length)).

[0020] The congestion control unit 16 acquires the resource usage status of the log processing unit 13 from the resource monitoring unit 15 and determines whether congestion (a load greater than a predetermined level) has occurred in the processing by the log processing unit 13. The determination of whether congestion has occurred may be made by comparing the resource usage status with a threshold value. When the congestion control unit 16 determines that congestion has occurred, it performs control to resolve the congestion by, for example, switching the operation of the transmission control unit 12.

[0021] 3 is a diagram showing an example of the hardware configuration of a computer that functions as the transmission control unit 12 in an embodiment of the present invention. The computer 10 in FIG. 3 includes a drive device 100, an auxiliary storage device 102, a memory device 103, a processor 104, and an interface device 105, all of which are interconnected via a bus B.

[0022] A program for realizing processing on the computer 10 is provided by a recording medium 101 such as a CD-ROM. When the recording medium 101 storing the program is set in the drive device 100, the program is installed from the recording medium 101 to the auxiliary storage device 102 via the drive device 100. However, the program does not necessarily have to be installed from the recording medium 101, but may be downloaded from another computer via a network. The auxiliary storage device 102 stores the installed program as well as necessary files, data, etc.

[0023] When an instruction to start a program is received, the memory device 103 reads and stores the program from the auxiliary storage device 102. The processor 104 is a CPU or a GPU (Graphics Processing Unit), or a CPU and a GPU, and executes functions related to the transmission control unit 12 in accordance with the program stored in the memory device 103. The interface device 105 is used as an interface for connecting to a network.

[0024] FIG. 4 is a diagram showing an example of the functional configuration of the transmission control unit 12 according to an embodiment of the present invention. In FIG. 4, the transmission control unit 12 includes a receiving unit 121, a distributing unit 122, a high-priority control unit 123a, a low-priority control unit 123b, a transmitting unit 124a, a transmitting unit 124b, and a state changing unit 125. Each of these units is implemented by the processor 104 executing one or more programs installed on the computer functioning as the transmission control unit 12. The transmission control unit 12 also uses a buffer 126. The buffer 126 can be implemented using, for example, the memory device 103, the auxiliary storage device 102, or a storage device connectable via a network to the computer functioning as the transmission control unit 12.

[0025] The receiving unit 121 receives the logs collected by the log collecting unit 11 from the log collecting unit 11 .

[0026] The distribution unit 122 determines the priority (high priority or low priority) of the log received by the receiving unit 121, and distributes the log to the high-priority control unit 123a or the low-priority control unit 123b based on the determination result. Specifically, the distribution unit 122 distributes high-priority logs to the high-priority control unit 123a and low-priority logs to the low-priority control unit 123b. The priority of a log may be determined based on whether the log contains a specific keyword (a keyword indicating high priority (e.g., Critical, Major, etc.) or a keyword indicating low priority (e.g., Minor, INFO, Debug, etc.)). Furthermore, the priority of a log does not have to be limited to two levels, high priority and low priority, and may be classified into, for example, three or more levels.

[0027] The high-priority control unit 123a performs control to change the destination of the assigned log (high-priority log) depending on the state (normal state or congestion state) set for the assigned log. The high-priority control unit 123a notifies the transmission unit 124a of the determined destination for the assigned log. The transmission unit 124a transmits the log to the destination (any of the high-priority nodes). The high-priority node refers to, for example, a process as the log processing unit 13 that is controlled by an OS (Operating System) and has a relatively high priority set for allocation of computing resources.

[0028] The low-priority control unit 123b performs control to change the destination of the assigned log (low-priority log) depending on the state (normal state or congestion state) set for itself. The low-priority control unit 123b notifies the transmission unit 124b of the determined destination for the assigned log. The transmission unit 124b transmits the log to the destination (any of the low-priority nodes). The low-priority node refers to, for example, a process as the log processing unit 13 controlled by the OS and set with a relatively low priority in terms of allocation of computing resources.

[0029] The resource monitoring unit 15 may monitor only the log processing units 13 corresponding to high-priority nodes for resource usage. The congestion control unit 16 inquires about the resource usage of the resource monitoring unit 15, and when it determines that congestion has occurred in one or more of the monitored log processing units 13 (e.g., the resource usage of one or more log processing units 13 is equal to or greater than a threshold), it instructs the state change unit 125 of the transmission control unit 12 to transition to a congested state. Furthermore, when the transmission control unit 12 has transitioned to a congested state, the congestion control unit 16 instructs the state change unit 125 of the transmission control unit 12 to transition to a normal state when it determines that the congestion has been released for all of the monitored log processing units 13 (e.g., the resource usage of all of the monitored log processing units 13 is less than a threshold).

[0030] When the state change unit 125 receives an instruction to transition to a congested state from the congestion control unit 16, it changes its own state to the "congested state" and sets the states of both the high priority control unit 123a and the low priority control unit 123b to the "congested state." Furthermore, when the state change unit 125 receives an instruction to transition to a normal state from the congestion control unit 16 while it is in the "congested state," it changes its own state to the "normal state" and sets the states of both the high priority control unit 123a and the low priority control unit 123b to the "normal state."

[0031] Next, the behavior of the high priority control unit 123a and the low priority control unit 123b in response to the "normal state" or "congestion state" will be described.

[0032] 5 is a diagram illustrating a first pattern of behavior depending on the state of the high priority control unit 123a. The upper side of Fig. 5 shows the behavior of the high priority control unit 123a in a normal state, and the lower side shows the behavior of the high priority control unit 123a in a congested state.

[0033] In the normal state, the high-priority control unit 123a selects only the "high-priority node (normal)" as a candidate for the destination of the high-priority log.

[0034] When a congestion state occurs, the high-priority control unit 123a also includes the "high-priority node (backup)" as a candidate destination for the high-priority log. In other words, when a congestion state occurs, the high-priority control unit 123a increases the number of candidate destinations for the high-priority log compared to when it is in a normal state. This makes it possible to suppress congestion occurring in the "high-priority node (normal)."

[0035] Scaling out by increasing the number of physical machines or virtual machines is effective in the event of congestion, but increasing the number of processes such as containers may conversely make the congestion redundant, so a "high priority node (backup)" is started in advance and put on standby to execute processing as the log processing unit 13. Even if a "high priority node (backup)" is started in advance, it is thought that if general processes are only running idle, almost no resources will be consumed.

[0036] The pattern in Figure 5 can also be said to be a pattern in which, under normal conditions, the high-priority control unit 123a selects only a portion of all high-priority nodes as candidates for destinations for logs, and when congestion occurs, increases the number of candidates for destinations for high-priority logs.

[0037] FIG. 6 is a diagram for explaining a second pattern of behavior depending on the state of the high priority control unit 123a.

[0038] 6, a process with a medium priority and a process with a relatively high priority relative to other processes are activated in advance as a node (log processing unit 13) to which a high-priority log is to be sent. The priority here refers to, for example, the priority regarding allocation of computing resources controlled by an OS (Operating System).

[0039] In a normal state, the high-priority control unit 123a selects only high-priority nodes with medium priority as candidates for log transmission destinations. When congestion occurs, the high-priority control unit 123a switches the candidate log transmission destinations to high-priority nodes with relatively high priority. This allows computational resources to be preferentially allocated to processing by the log processing unit 13, which is expected to reduce congestion.

[0040] 6 may be obtained by the congestion control unit 16 changing the priority of the high-priority node (log processing unit 13). In this case, the candidates for log destinations are the same in the normal state and the congestion state, but the priority of each candidate is changed.

[0041] FIG. 7 is a diagram for explaining a first pattern of behavior depending on the state of the low priority control unit 123b.

[0042] 7, the low-priority control unit 123b discards low-priority logs when congestion occurs. Therefore, in this case, low-priority logs are not sent to the low-priority node (log processing unit 13). The first pattern is effective when the requirements allow for the discarding of low-priority logs.

[0043] FIG. 8 is a diagram for explaining a second pattern of behavior depending on the state of the low priority control unit 123b.

[0044] 8, when a congestion state occurs, the low-priority control unit 123b temporarily suspends transmission of low-priority logs and stores them in the buffer 126. After that, when the normal state is restored, if there are low-priority logs stored in the buffer 126, the low-priority control unit 123b determines the destination of the low-priority logs.

[0045] The second pattern places a greater load than the first pattern by the amount of input processing into the buffer 126. The second pattern is a countermeasure that assumes that the congestion state will be resolved by, for example, scaling out the log processing unit 13, and the buffer 126 requires a capacity equivalent to the amount of data of low-priority logs that will be generated during the period until the congestion is resolved.

[0046] FIG. 9 is a diagram for explaining a third pattern of behavior according to the state of the low priority control unit 123b.

[0047] 9, the low-priority control unit 123b normally selects high-priority nodes as candidates for the destination of the low-priority log. Therefore, the candidates for the destination of the log from the transmission control unit 12 are only high-priority nodes, as shown on the left side of FIG.

[0048] When congestion occurs, the low-priority control unit 123b selects only low-priority nodes as candidates for the destination of the low-priority log. Therefore, the candidates for the destination of the log from the transmission control unit 12 are only high-priority nodes for the high-priority log and only low-priority nodes for the low-priority log, as shown on the right side of FIG.

[0049] It should be noted that the processing by the low priority node may be started after the processing by the high priority node is completed.

[0050] As described above, according to this embodiment, it is possible to reduce the possibility that a response to a log with a relatively high priority will be neglected.

[0051] The computer that functions as the transmission control unit 12 is an example of a log transmission control device.

[0052] Although the embodiments of the present invention have been described in detail above, the present invention is not limited to such specific embodiments, and various modifications and changes are possible within the scope of the gist of the present invention described in the claims.

[0053] REFERENCE SIGNS LIST 1 System monitoring platform 10 Computer 11 Log collection unit 12 Transmission control unit 13 Log processing unit 14 Log analysis unit 15 Resource monitoring unit 16 Congestion control unit 121 Reception unit 122 Distribution unit 123a High priority control unit 123b Low priority control unit 124a Transmission unit 124b Transmission unit 125 Status change unit 126 Buffer 100 Drive device 101 Recording medium 102 Auxiliary storage device 103 Memory device 104 Processor 105 Interface device B Bus

Claims

1. A log transmission control device comprising: a transmission control unit configured to control the transmission of logs collected from a monitored system to a node that performs a specified process on the logs; and the transmission control unit configured to, when congestion occurs in relation to the specified process, increase the number of nodes to which logs having relatively high priority are sent compared to when the congestion does not occur.

2. The log transmission control device according to claim 1, characterized in that a node that increases when congestion occurs in relation to the specified process is waiting to execute the specified process even when the congestion does not occur.

3. A log transmission control device comprising: a transmission control unit configured to control the transmission of logs collected from a monitored system to a node that executes a specified process, wherein the transmission control unit is configured to change the destination of logs having a relatively high priority among the logs to a node having a relatively high priority for allocation of computing resources when congestion occurs in the specified process.

4. A log transmission control method, comprising: a computer executing a transmission control procedure for controlling the transmission of logs collected from a monitored system to a node that performs a specified process for the logs; and the transmission control procedure, when congestion occurs in relation to the specified process, increases the number of nodes to which logs having relatively high priority are sent compared to a case in which the congestion does not occur.

5. A program for causing a computer to execute a transmission control procedure for controlling the transmission of logs collected from a monitored system to a node that executes a specified process, the transmission control procedure increasing the number of nodes to which logs with relatively high priority are sent when congestion occurs in relation to the specified process compared to when the congestion does not occur.

Images