Telemetry data collection system and method

Abstract

To provide a technology for collecting effective telemetry data from a distributed system.SOLUTION: The monitoring agent records the acquired telemetry data in a buffer and transmits the telemetry data corresponding to a predetermined condition to the monitoring manager, the monitoring manager specifies, based on the received telemetry data, a perpetuation condition indicating a monitoring agent to be perpetuated and a range of data to be perpetuated, and a transmission condition indicating a monitoring agent from which the telemetry data is to be additionally collected and the telemetry data to be additionally collected, and notifies the perpetuation monitoring agent of the perpetuation condition, the monitoring agent perpetuates the perpetuated data range according to the notified perpetuation condition, and the monitoring manager notifies the additional transmission monitoring agent of the transmission condition, the additional transmission monitoring agent transmits the additional transmission telemetry data in the buffer to the monitoring manager in accordance with the notified transmission condition.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present disclosure relates to techniques for collecting telemetry data from a system. [Background technology]

[0002] In recent years, the market for log management, which collects telemetry data from applications on various systems and records and manages it as logs, has been growing. Generally, collecting telemetry data consumes resources from the target system (hereinafter referred to as the "target system"), so the amount of telemetry data collected is often limited to prevent performance degradation and increased costs of operating the target system. For example, in systems such as financial systems where any impact on performance cannot be tolerated, minimal telemetry data is collected to minimize the impact on performance. Therefore, when an error occurs, it can be difficult to determine the cause of the error by analyzing limited telemetry data.

[0003] Patent Document 1 discloses a technology that enables error analysis while limiting the logs to be output. In the technology of Patent Document 1, the application side of the target system classifies logs into multiple levels, limits the log level of logs to be output under normal circumstances, and buffers detailed logs internally. When an error such as a transaction failure occurs, detailed logs are output from the buffer going back to the time the error occurred. This limits the amount of log data output under normal circumstances, making it possible to analyze using the buffered logs when an error occurs. Furthermore, the log to be output when an error occurs is only the data recorded in the buffer tens to hundreds of milliseconds before the process in which the error occurred, so the system resources required for log output can be kept to a certain level. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] US Patent US9891979B2 Summary of the Invention [Problem to be solved by the invention]

[0005] Services using microservices, in which multiple hosts work together to execute processing, are on the rise. Applications built using microservices consist of multiple sub-applications deployed on multiple hosts. When a distributed system such as a microservice is used as the target system, telemetry data acquired by the processes of multiple sub-applications may be required to analyze the cause of an error. However, the technology in Patent Document 1 does not take into consideration the acquisition of telemetry data across multiple processes in a distributed system. Furthermore, it does not take into consideration the potential impact on resources when attempting to acquire telemetry data from multiple processes. One objective of this disclosure is to provide techniques that enable efficient collection of telemetry data from distributed systems. [Means for solving the problem]

[0006] A telemetry data collection system according to one embodiment included in the present disclosure is a telemetry data collection system that collects telemetry data of an application consisting of a plurality of sub-applications, and includes a monitoring agent that acquires telemetry data from processes of the sub-applications, and a monitoring manager that receives the telemetry data from the monitoring agent, wherein the monitoring agent records the acquired telemetry data in a buffer and transmits telemetry data that meets a predetermined condition to the monitoring manager, and the monitoring manager executes a persistence monitoring process based on the telemetry data received from the monitoring agent. The monitoring manager identifies persistence conditions indicating an agent and a persistence data range indicating the range of data to be persisted from the telemetry data recorded in the buffer, and transmission conditions indicating an additional transmission monitoring agent, which is a monitoring agent that should collect additional telemetry data, and additional transmission telemetry data, which is the telemetry data to be collected.The monitoring manager notifies the persistence conditions to the persistence monitoring agent, and the persistence monitoring agent persists the persistence data range in accordance with the notified persistence conditions.The monitoring manager notifies the additional transmission monitoring agent of the transmission conditions, and the additional transmission monitoring agent transmits the additional transmission telemetry data in the buffer to the monitoring manager in accordance with the notified transmission conditions. [Effects of the Invention]

[0007] According to one aspect of the present disclosure, it is possible to collect effective telemetry data from distributed systems. [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 1 is an overall block diagram including a telemetry data collection system and a target system. [Figure 2] FIG. 10 is a sequence diagram showing a data collection sequence. [Figure 3]FIG. 10 is a sequence diagram of a persistence data identification sequence. [Figure 4] FIG. 10 is a sequence diagram of a sequential data extraction sequence. [Figure 5] FIG. 2 is a block diagram of a monitoring manager. [Figure 6] FIG. 10 is a diagram illustrating an example of a buffer flush condition table. [Figure 7] FIG. 10 is a diagram illustrating an example of a screen for setting a buffer flush condition. [Figure 8] FIG. 10 is a diagram showing an example of a sub-application agent mapping table. [Figure 9] FIG. 10 is a diagram illustrating an example of a sub-application logical configuration table. [Figure 10] FIG. 10 is a diagram illustrating an example of a buffer size table. [Figure 11] FIG. 10 is a diagram illustrating an example of a related telemetry table. [Figure 12] 10 is a flowchart of a persistent data identification process. [Figure 13] 10 is a flowchart of a sequential data extraction process. [Figure 14] 10 is a flowchart of a buffer size and transmission rate calculation process. DETAILED DESCRIPTION OF THE INVENTION

[0009] This embodiment will be described in detail with reference to the drawings.

[0010] In the drawings, parts having the same function may be given the same reference numeral, and their explanation may be omitted. Furthermore, in order to distinguish between parts having the same function, a reference numeral consisting of a main reference numeral given to parts having the same function and a sub-reference numeral added after a hyphen to distinguish between parts may be given. Furthermore, even when a reference numeral consisting of a main reference numeral and a sub-reference numeral is given, there are cases where it is not necessary to distinguish between parts or when it is not possible to distinguish between parts, and the main reference numeral alone is used for explanation.

[0011] The telemetry data collection system of this embodiment is a system that collects telemetry data from an application of a target system that is composed of multiple sub-application containers placed on multiple host computers (hereinafter also referred to as "hosts").

[0012] FIG. 1 is an overall block diagram including the telemetry data collection system and the target system. The target system is a system from which telemetry data is collected, and is a distributed system consisting of multiple hosts 800-1 to 800-N. Multiple sub-application containers 600-1 to 600-2 and a monitoring agent container 200-1 driven by a container runtime 700-1 are configured in the host 800-1. Sub-application containers 600-(M-1) to 600-M and a monitoring agent container 200-N driven by the container runtime 700-N are configured in the host 800-N. The applications of the target system are composed of the sub-application containers 600-1 to 600-M. The telemetry data collection system also includes a monitoring manager 100 and a plurality of monitoring agent containers 200-1 to 200-N.

[0013] The monitoring agent container 200-1 is a container that acquires telemetry data from the processes of the sub-application containers 600-1 to 600-2 on the host 800-1. The monitoring agent container 200-N is a container that acquires telemetry data from the processes of the sub-application containers 600-(M-1) to 600-M on the host 800-N.

[0014] The monitoring agent container 200-1 includes sub-application ring buffers 210-1 to 210-2 corresponding to the sub-application containers 600-1 to 600-2, respectively. The monitoring agent container 200-N includes sub-application ring buffers 210-(M-1) to 210-M corresponding to the sub-application containers 600-(M-1) to 600-M, respectively.

[0015] The monitoring manager 100 is a computer that receives telemetry data while controlling the transmission of telemetry data from the monitoring agent containers 200-1 to 200-N. The monitoring manager 100 and the monitoring agent container 200 cooperate as follows.

[0016] 2 is a sequence diagram showing a data collection sequence, which is a sequence in which the telemetry data collection system normally collects telemetry data from the target system.

[0017] Referring to Figure 2, first, telemetry data from each sub-application container 600-1, 600-2 is acquired by the monitoring agent container 200-1 (step 1010). The telemetry data includes trace data, log data, and metrics data. Trace data is data sampled from the container memory at predetermined times. Metrics data is data indicating index values ​​calculated based on the container's operations. Log data is data recording the container's operations. The monitoring agent container 200-1 records the telemetry data in the sub-application ring buffers 210-1 and 210-2 (step 1020).

[0018] Furthermore, the monitoring agent container 200-1 transmits trace data that satisfies a predetermined condition from the telemetry data to the trace data store 300 (step 1030). The trace data is saved in the trace data store 300 (step 1040).

[0019] Furthermore, the monitoring agent container 200-1 transmits log data that satisfies a predetermined condition to the log data store 500 (step 1050). The log data is saved in the log data store 500 (step 1060).

[0020] Furthermore, the monitoring agent container 200-1 transmits the metrics data that meets the predetermined conditions to the metrics data store 400 (step 1070). The metrics data is saved in the metrics data store 400 (step 1080).

[0021] As a result, the monitoring agent container 200-1 records the telemetry data obtained from the processes of the sub-application containers 600-1 and 600-2 in the sub-application ring buffers 210-1 to 210-2, respectively, and transmits only the telemetry data that meets specified conditions to a data store under the management of the monitoring manager 100.

[0022] Similarly, other monitoring agent containers 200 also record the telemetry data obtained from the processes of the sub-application container 600 in their respective sub-application ring buffers 210, and transmit only the telemetry data that meets certain conditions to a data store under the management of the monitoring manager 100. The processing of steps 1010 to 1080 is periodically repeated.

[0023] FIG. 3 is a sequence diagram of a persistence data identification sequence. The persistence data identification sequence is a sequence that identifies the data range to be persisted when it becomes necessary to send telemetry data stored in the sub-application ring buffer 210 of the monitoring agent container 200 to the monitoring manager 100, and requests the monitoring agent container 200 to persist that data area. Hereinafter, sending the telemetry data stored in the sub-application ring buffer 210 to the monitoring manager 100 may be referred to as a buffer flush.

[0024] Furthermore, "persistence" here means putting desired data in a buffer into a state where it will not be lost. In this embodiment, a ring buffer is used as an example, and therefore overwriting of the area in the ring buffer where data to be persisted is stored is prohibited. As another example, when using buffering that assigns a storage expiration date using TTL (Time to Live), data may be persisted by erasing the storage expiration date of the area where the data to be persisted is stored.

[0025] Referring to FIG. 3, when the monitoring manager 100 starts the process of identifying persistent data (step 1310), it first refers to the trace data stored in the trace data store 300 (step 1320) and acquires it (step 1330).

[0026] Furthermore, the monitoring manager 100 references (step 1340) and acquires (step 1350) the log data stored in the log data store 500. Furthermore, the monitoring manager 100 references (step 1360) and acquires (step 1370) the metrics data stored in the metrics data store 400.

[0027] Then, when the monitoring manager 100 determines that a buffer flush is necessary based on the telemetry data from all monitoring agent containers 200, it identifies a monitoring agent container (persistence monitoring agent) that should persist the telemetry data recorded in the sub-application ring buffer 210, a persistence condition that indicates the persistence data range that indicates the range of data that should be persisted among the telemetry data recorded in the sub-application ring buffer 210, and a transmission condition that indicates a monitoring agent container (additional transmission monitoring agent) that should collect additional telemetry data and the additional transmission telemetry data that is the additional telemetry data that should be collected. At this time, as an example, the monitoring manager 100 determines that a buffer flush is necessary if the referenced telemetry data meets a predetermined condition. Hereinafter, this condition may be referred to as a buffer flush condition.

[0028] Then, the monitoring manager 100 sends a persistence request including the identified persistence conditions to the monitoring agent container 200 that has become the persistence monitoring agent among the monitoring agent containers 200-1 to 200-N (step 1380). Upon receiving the notification, the monitoring agent container 200 persists the persistence data range in accordance with the notified persistence conditions.

[0029] 4 is a sequence diagram of a sequential data extraction sequence, which is a sequence for sequentially selecting necessary data from data persisted in the sub-application ring buffer 210 and flushing the buffer.

[0030] 4, when the monitoring manager 100 starts the process of sequential data extraction (step 1510), first, the monitoring manager 100 notifies the monitoring agent container 200 that has become an additional transmission monitoring agent according to the transmission conditions among the monitoring agent containers 200-1 to 200-N of the transmission conditions (steps 1510-1, 1510-N). In FIG. 4, it is assumed that the monitoring agent containers 200-1 and 200-N have become additional transmission monitoring agents.

[0031] The additional transmission monitoring agent that has received the notification transmits the additional transmission telemetry data in the sub-application ring buffer 210 to the monitoring manager 100 in accordance with the notified transmission conditions (steps 1520-1, 1520-N).

[0032] Of the additionally transmitted telemetry data, trace data is stored in the trace data store 300 (steps 1530-1, 1530-N). Also, log data is stored in the log data store 500 (steps 1540-1, 1540-N). Also, metrics data is stored in the metrics data store 400 (steps 1550-1, 1550-N).

[0033] Thereafter, the monitoring manager 100 references (step 1560) and acquires (step 1570) the trace data stored in the trace data store 300. Furthermore, the monitoring manager 100 references (step 1580) and acquires (step 1590) the log data stored in the log data store 500. Furthermore, the monitoring manager 100 references (step 1600) and acquires (step 1610) the metrics data stored in the metrics data store 400.

[0034] Then, based on the acquired telemetry data, the monitoring manager 100 identifies a monitoring agent container (additional transmission monitoring agent) from which additional telemetry data should be collected and transmission conditions indicating the additional transmission telemetry data that should be collected.

[0035] If there is additional telemetry data to be collected, the monitoring manager 100 sends a transmission condition to the additional transmission monitoring agent and repeats the process. If there is no additional telemetry data to be collected, the monitoring manager 100 sends a request to the monitoring agent container 200 to cancel the persistence (step 1620). Upon receiving the request, the monitoring agent container 200 cancels the persistence of the persistence data range.

[0036] As described above, according to the telemetry data collection system of this embodiment, the monitoring manager 100 controls the persistence and transmission of the telemetry data buffered in the monitoring agent container 200 based on the telemetry data received from the monitoring agent container 200, so that effective telemetry data can be collected from the target system, which is a distributed system.

[0037] FIG. 5 is a block diagram of the monitoring manager. The monitoring manager 100 is a computer equipped with a processor and memory (not shown), and can also be considered a telemetry data collection device. As shown in Figure 5, the monitoring manager 100 is executable by a telemetry data reception program 110, a persistent data identification program 120, a sequential data extraction program 130, and a buffer size / transmission rate calculation program 140. The monitoring manager 100 also stores a sub-application logical configuration table 150, a sub-application / agent mapping table 160, a buffer flush condition table 170, a buffer size table 180, and an associated telemetry table 190.

[0038] The telemetry collection device can be considered to logically be configured with a telemetry data reception unit, a persistent data identification unit, a sequential data extraction unit, and a buffer size / transmission rate calculation unit by the processor executing each of the telemetry data reception program 110, the persistent data identification program 120, the sequential data extraction program 130, and the buffer size / transmission rate calculation program 140. The details of the processing realized by the processor executing each program using each table will be described later.

[0039] 6 is a diagram showing an example of a buffer flush condition table 170. The buffer flush condition table 170 is a table that defines buffer flush conditions, and persistence conditions and transmission conditions when the buffer flush conditions are met.

[0040] 6, agent identifiers, conditions, persistence conditions, and transmission conditions are associated and registered in the buffer flush condition table 170. The agent identifier is an identifier that identifies the monitoring agent container 200. In the above conditions, the buffer flush conditions are determined by the data type.

[0041] It is determined that a buffer flush should be performed if telemetry data of the type of an entry in the buffer flush condition table 170 is received from a monitoring agent container 200 that matches the agent identifier in that entry. The persistence conditions and transmission conditions of that entry are used for the buffer flush.

[0042] Taking the first entry as an example, when telemetry data whose data type is "log data" is received from the monitoring agent container 200 whose agent identifier is "1", it is determined that a buffer flush should be performed. The persistence condition at that time is "if {timestamp} <= {timestamp_if_any_error} - 5sec && if{timestamp} <= {timestamp_if_any_error} + 5sec". As in this example, in the persistence condition, the data range can be expressed by the time range of the time when the telemetry data was acquired (i.e., the timestamp of the telemetry data). Furthermore, the transmission condition is "if {log_string} == \W*((?i)error(?-i))\W*". The transmission condition can be expressed by the telemetry identifier that identifies the telemetry data and / or the time range of the time when the telemetry data was acquired.

[0043] FIG. 7 is a diagram showing an example of a screen for setting buffer flush conditions. 7, a buffer flush condition setting screen 2300 is a GUI (Graphical User Interface) screen that accepts user input. The buffer flush condition setting screen 2300 has text boxes for "Sub-application name or process identifier," "Telemetry type," "Persistence condition," and "Transmission condition," as well as "Save" and "Cancel" buttons.

[0044] The sub-application container 600 can be specified by entering the sub-application name or process identifier in the "Sub-application name or process identifier" text box. The buffer flush condition table 170 is set with the agent identifier of the monitoring agent container 200 that is located on the same host 800 as the specified sub-application container 600.

[0045] By entering the type of telemetry data in the "Telemetry Type" text box, the conditions for determining that a buffer flush is necessary can be specified. The entered type of telemetry data is set as a condition in the buffer flush condition table 170.

[0046] A persistence condition can be input in the "persistence condition" text box. The input persistence condition is set in the buffer flush condition table 170.

[0047] The transmission conditions can be input in the "Transmission Condition" text box. The input transmission conditions are set in the buffer flush condition table 170.

[0048] The "Save" button is a button for confirming the buffer flush conditions entered in the text boxes. When the "Save" button is pressed, buffer flush conditions are generated according to the text information entered in each text box and set in the buffer flush condition table 170.

[0049] The "Cancel" button is used to cancel the creation of a buffer flush condition using the text entered in the text box. When the "Cancel" button is pressed, the information entered in the text box is cleared and the buffer flush condition is not created.

[0050] As shown in this example, persistence conditions and / or transmission conditions can be set using arithmetic expressions or regular expressions. Specifying persistence conditions and / or transmission conditions using arithmetic expressions or regular expressions enables effective telemetry data to be collected appropriately.

[0051] Furthermore, when text is entered into the "Condition" text box and the "Persistence Condition" text box on the buffer flush condition setting screen 2300, if there is a buffer flush condition among the set buffer flush conditions that has the same conditions and persistence conditions set, the text set as the transmission condition in that buffer flush condition may be displayed in the "Transmission Condition" text box as a recommended text input.

[0052] 8 shows an example of a sub-application agent mapping table 160. The sub-application agent mapping table 160 is a table that associates a sub-application container 600 with a monitoring agent container 200 that is located on the same host 800 as the sub-application container.

[0053] 8, process identifiers and agent identifiers are registered in association with each other in the sub-application / agent mapping table 160. The process identifier is an identifier for identifying the process of the sub-application. The agent identifier is an identifier for identifying the monitoring agent container 200.

[0054] The sub-application container 600 of the process corresponding to the process identifier in each entry of the sub-application-agent mapping table 160 and the monitoring agent container 200 corresponding to the agent identifier are located on the same host 800 .

[0055] Taking the first entry as an example, it shows that the sub-application container 600 of a process with the process identifier "PROCESS1" and the monitoring agent container 200 with the agent identifier "AGENT1" are located on the same host 800.

[0056] 9 is a diagram showing an example of a sub-application logical configuration table. The sub-application logical configuration table 150 is a table that associates sub-applications with sub-applications related to them. Here, "related" means, for example, that there is a dependency relationship. The sub-application logical configuration table 150 is used to identify sub-applications for which telemetry data is persisted and additional data is collected.

[0057] 9, process identifiers and associated process identifiers are registered in association with each other in the sub-application logical configuration table 150. Both the process identifier and the associated process identifier are identifiers that identify the processes of the sub-application container 600.

[0058] It is shown that the sub-application of the process corresponding to the process identifier in each entry of the sub-application logical configuration table 150 is related to the sub-application of the process corresponding to the related process identifier.

[0059] Taking the first entry as an example, it shows that the sub-application of the process with the process identifier "PROCESS1" is related to the sub-application of the process with the process identifier "PROCESS2".

[0060] 10 is a diagram showing an example of the buffer size table 180. The buffer size table 180 is a table in which the buffer size and usage status of the sub-application ring buffer 210 provided in each monitoring agent container 200 are recorded.

[0061] Referring to Figure 10, the buffer size table 180 records an agent identifier, time, maximum usage, transmission rate, and buffer size in association with each other. The agent identifier is an identifier that identifies the monitoring agent container 200. The time indicates the time when the maximum usage of the entry was confirmed. The maximum usage indicates the maximum usage value of the sub-application ring buffer 210 confirmed at the time indicated in the time column. The transmission rate is the data speed at which the monitoring agent container 200 transmits the telemetry data in the sub-application ring buffer 210. The buffer size indicates the amount of telemetry data that can be held in the sub-application ring buffer 210.

[0062] Each entry in the buffer size table 180 is set with the maximum usage confirmed at the time indicated in the time column for the monitoring agent container 200 corresponding to the agent identifier, as well as the transmission rate and buffer size of that monitoring agent container 200, and these are updated as appropriate.

[0063] Taking the first entry as an example, it shows that the maximum usage confirmed for the monitoring agent container 200 with agent identifier "1" at time "2024-02-28 14:21:56" was "1.5MB", the transmission rate was "100KBS", and the buffer size was "2MB".

[0064] 11 is a diagram showing an example of a related telemetry table. The related telemetry table 190 is a table that associates telemetry data with telemetry data related to it. Here, "related" means, for example, that the telemetry data represents a correlation between events. The related telemetry table 190 is used to identify additional telemetry data that should be transmitted.

[0065] 11, a telemetry identifier and a related telemetry identifier are associated and registered in the related telemetry table 190. Both the telemetry identifier and the related telemetry identifier are identifiers that identify telemetry data.

[0066] It is shown that the telemetry data corresponding to the telemetry identifier in each entry of the related telemetry table 190 is related to the telemetry data corresponding to the related telemetry identifier.

[0067] Taking the first entry as an example, it indicates that the telemetry data with the telemetry identifier "LOG_1237293JD" and the telemetry data with the telemetry identifier "METRICS_53243pD" are related to each other.

[0068] Here, the telemetry data with the telemetry identifier "LOG_1237293JD" is log data output from function 1 of sub-application "APP1." Also, the telemetry data with the telemetry identifier "METRICS_53243pD" is metric data called "cpu usage" of sub-application "app1."

[0069] Therefore, the first entry indicates that the log data output from function 1 of sub-application "APP1" is associated with the metrics data "cpu usage" of sub-application "app1".

[0070] 12 is a flowchart of the persistent data identification process. The persistent data identification process is realized by the processor in the monitoring manager 100 executing the persistent data identification program 120. The persistent data identification process is started when the target system and the telemetry data collection system are started. The persistent data identification process is a process that identifies the data range to be persisted when it becomes necessary to send telemetry data stored in the sub-application ring buffer 210 of the monitoring agent container 200 to the monitoring manager 100, and requests the monitoring agent container 200 to persist that data area.

[0071] 12, the monitoring manager 100 waits until it receives telemetry data (step 4010). The telemetry data is extracted from the trace data store 300, the metrics data store 400, and the log data store 500 by the processing of the telemetry data reception program 110, so it is sufficient to monitor the extracted telemetry data.

[0072] When the telemetry data is received, the monitoring manager 100 determines whether or not there is an entry in the buffer flush condition table 170 whose condition is satisfied by the received telemetry data (step 4020). If there is no entry in the buffer flush condition table 170 whose condition is satisfied by the received telemetry data, the monitoring manager 100 returns to step 4010.

[0073] If there is an entry in the buffer flush condition table 170 where the received telemetry data satisfies the condition, the monitoring manager 100 identifies the agent identifier included in the telemetry data acquired in step 4010 (step 4030).

[0074] Next, the monitoring manager 100 refers to the sub-application-agent mapping table 160 and identifies the process identifiers of all entries in the table whose agent identifier is the same as the agent identifier identified in step 4030 (step 4040).

[0075] Next, the monitoring manager 100 combines the sub-application agent mapping table 160 and the sub-application logical configuration table 150 using the process identifier as a key, and by referencing the combined table, identifies the agent identifiers of all entries whose associated process identifier is the process identifier identified in step 4040 (step 4050).

[0076] Next, the monitoring manager 100 starts the sequential data extraction program 130 if it is not running, and starts the buffer size and transmission rate calculation program 140 if it is not running (step 4060).

[0077] Next, the monitoring manager 100 refers to the buffer flush condition table 170 and identifies the persistence condition corresponding to the agent identifier identified in step 4050 (step 4070).

[0078] Next, the monitoring manager 100 issues an incident identifier and requests persistence by sending the persistence conditions and the incident identifier to the monitoring agent container 200 with the agent identifier identified in step 4050 (step 4080). Issuing an incident identifier here means that when a new incident occurs, a new incident identifier is assigned to that incident. The monitoring manager 100 then returns to step 4010 and repeats the process.

[0079] 13 is a flowchart of the sequential data extraction process. The sequential data extraction process is realized by the processor in the monitoring manager 100 executing the sequential data extraction program 130. The sequential data extraction process is started by the processing of step 4060 described above. The sequential data extraction process is a process of sequentially selecting necessary data from the data persisted in the sub-application ring buffer 210 and buffer-flushing the data.

[0080] Referring to FIG. 13, the monitoring manager 100 waits for a predetermined time limit to receive telemetry data including the incident identifier issued in step 4080 (step 4210).

[0081] If the telemetry data is received within a predetermined time, the monitoring manager 100 identifies the telemetry identifier included in the received telemetry data (step 4220).

[0082] Next, the monitoring manager 100 refers to the associated telemetry table 190 and identifies the associated telemetry identifiers of all entries that have the same telemetry identifier as the telemetry identifier identified in step 4220 (step 4230).

[0083] Next, the monitoring manager 100 identifies all of the associated telemetry identifiers identified in step 4230 that are not included in the transmitted identifier list (step 4240). The transmitted identifier list is list information that records the telemetry identifiers of the telemetry that transmitted the telemetry data. The transmitted identifier list is initially empty, and the number of telemetry identifiers increases as the processing progresses.

[0084] Next, the monitoring manager 100 determines whether or not there is one or more associated telemetry identifiers that are not included in the transmitted identifier list among the associated telemetry identifiers recognized in step 4230 (step 4250). If there is no associated telemetry identifier that is not included in the transmitted identifier list among the associated telemetry identifiers recognized in step 4230, the process returns to step 4210.

[0085] If there is at least one associated telemetry identifier that is not included in the transmitted identifier list among the associated telemetry identifiers identified in step 4230, the monitoring manager 100 then sends the associated telemetry identifier identified in step 4240 and the incident identifier issued in step 4070 to the monitoring agent container 200 identified in step 4050 (step 4260).The monitoring manager 100 then adds the pair of the sent associated telemetry identifier and incident identifier to the transmitted identifier list.

[0086] The related telemetry identifier and incident identifier sent by the monitoring manager 100 are information indicating the additional telemetry data to be sent (additional transmission telemetry data). Upon receiving the related telemetry identifier and incident identifier, the monitoring agent container 200 sends the telemetry data persisted in the sub-application ring buffer 210 to the monitoring manager 100 in accordance with the information. After step 4260 , the monitoring manager 100 returns to step 4250 .

[0087] In step 4210, if the telemetry data is not received within a predetermined time, the monitoring manager 100 determines that collection of the required telemetry data has been completed, clears the transmitted identifier list to empty, and terminates processing (step 4270).

[0088] 14 is a flowchart of the buffer size and transmission rate calculation process. The buffer size and transmission rate calculation process is realized by the processor in the monitoring manager 100 executing the buffer size and transmission rate calculation program 140. The buffer size and transmission rate calculation process is started by the processing of step 4060 described above. The buffer size and transmission rate calculation process is a process for appropriately controlling the buffer size and transmission rate of the sub-application ring buffer 210 in the monitoring agent container 200.

[0089] Referring to Figure 14, the monitoring manager 100 receives information from each monitoring agent container 200 indicating the usage of the sub-application ring buffer 210 and the surplus computing resources (CPU / Memory / IO) allocated to the monitoring agent container 200 (step 4410).

[0090] Next, the monitoring manager 100 identifies (step 4420) an entry in the buffer size table 180 whose agent identifier is the same as the agent identifier of the monitoring agent container 200 received in step 4410. If no such entry exists, the monitoring manager 100 adds a new entry.

[0091] Next, the monitoring manager 100 determines whether the difference between the time of the entry identified in step 4420 and the current time is greater than or equal to a predefined value, or whether the maximum usage of the entry is less than the usage identified in step 4410 (step 4430).

[0092] If the difference between the time of the entry identified in step 4420 and the current time is not greater than a predetermined value, and the maximum usage amount of the entry is not smaller than the usage amount identified in step 4410, the process returns to step 4410.

[0093] On the other hand, if the difference between the time of the entry identified in step 4420 and the current time is equal to or greater than a predetermined value, or the maximum usage of that entry is smaller than the usage identified in step 4410, the monitoring manager 100 identifies an entry in the buffer size table 180 whose agent identifier is the same as the agent identifier of the monitoring agent container 200 received in step 4410 (step 4435).If no such entry exists, the monitoring manager 100 adds a new entry.

[0094] Next, the monitoring manager 100 saves the current time and the usage amount determined in step 4410 as the time and maximum usage amount of the entry determined in step 4420 (step 4440).

[0095] Next, the monitoring manager 100 determines whether the absolute value of the difference between the maximum usage of the entry determined in step 4420 multiplied by a coefficient α and the buffer size of the same entry is greater than a predetermined value (step 4450). The coefficient α is a value by which the maximum usage is multiplied to allow for some margin in the buffer size, and is, for example, a value greater than or equal to 1.

[0096] If the absolute value of the difference between the maximum usage amount of the entry grasped in step 4420 multiplied by the coefficient α and the buffer size of the same entry is not greater than the predetermined value, the process returns to step 4410.

[0097] On the other hand, if the absolute value of the difference between the maximum usage of the entry determined in step 4420 multiplied by the coefficient α and the buffer size of the same entry is greater than a predetermined specified value, the monitoring manager 100 determines whether the absolute value of the difference between the surplus amount of computing resources determined in step 4410 and a predefined value is greater than or equal to the predetermined specified value (step 4455).

[0098] If the absolute value of the difference between the surplus amount of computing resources determined in step 4410 and a predefined value is greater than or equal to a predetermined value, the monitoring manager 100 saves the value obtained by multiplying the transmission rate by a coefficient γ (γ<1) if the surplus amount of computing resources determined in step 4410 is smaller than the specified value, or the value obtained by multiplying the transmission rate by a coefficient γ' (γ'>1) if the surplus amount of computing resources determined in step 4410 is greater than the specified value, as the transmission rate of the entry determined in step 4420 (step 4460).

[0099] Next, the monitoring manager 100 sends a notification to the monitoring agent container 200 identified in step 4410 requesting a change in the transmission rate (step 4470), and ends the process.

[0100] On the other hand, in step 4455, if the absolute value of the difference between the surplus amount of computing resources determined in step 4410 and the predefined value is not greater than a predetermined defined value, the monitoring manager 100 saves the value obtained by multiplying the maximum usage amount of the entry determined in step 4420 by the coefficient β as the buffer size of the sub-application ring buffer (step 4480).

[0101] Next, the monitoring manager 100 notifies the monitoring agent container 200 identified in step 4410 of a request to change the buffer size (step 4490), and ends the process.

[0102] The above-described embodiment is an example for explaining the present invention, and is not intended to limit the scope of the present invention to this embodiment. Those skilled in the art can implement the present invention in various other modes without departing from the scope of the present invention. The above-described embodiment also includes the following features, but the features included in the above-described embodiment are not limited to those listed below.

[0103] (Item 1) A telemetry data collection system for collecting telemetry data of an application consisting of a plurality of sub-applications includes a monitoring agent that acquires telemetry data from processes of the sub-applications, and a monitoring manager that receives the telemetry data from the monitoring agent, wherein the monitoring agent records the acquired telemetry data in a buffer and transmits the telemetry data that meets a predetermined condition to the monitoring manager, and the monitoring manager, based on the telemetry data received from the monitoring agent, selects a persistence monitoring agent that is a monitoring agent that should persist the telemetry data recorded in the buffer and a monitoring manager that should persist the telemetry data recorded in the buffer. The monitoring manager identifies persistence conditions indicating a persistence data range that indicates the range of telemetry data that should be persisted, and transmission conditions indicating additional transmission monitoring agents that are monitoring agents that should collect additional telemetry data and additional transmission telemetry data that is the telemetry data that should be collected, and notifies the persistence conditions to the persistence monitoring agents, which persist the persistence data range in accordance with the notified persistence conditions, and notifies the additional transmission monitoring agents of the transmission conditions, which the additional transmission monitoring agents transmit the additional transmission telemetry data in the buffer to the monitoring manager in accordance with the notified transmission conditions. This allows the monitoring manager to control the persistence and transmission of the telemetry data buffered in the monitoring agents based on the telemetry data received from the monitoring agents, making it possible to collect effective telemetry data from a distributed system.

[0104] (Item 2) In the telemetry data collection system described in item 1, when the monitoring manager receives telemetry data transmitted from the additional transmission monitoring agent in accordance with the transmission conditions, the monitoring manager again identifies new transmission conditions indicating the new additional transmission monitoring agent and the additional transmission telemetry data based on the received telemetry data, and notifies the new additional transmission monitoring agent of the new transmission conditions. This recursively repeats the process of the monitoring manager controlling the transmission of further telemetry data in the monitoring agent based on the telemetry data received from the monitoring agent, thereby enabling effective collection of telemetry data.

[0105] (Item 3) In the telemetry data collection system described in item 1, the monitoring manager indicates the data range in the persistence condition by the time range of the time when the telemetry data was acquired. This allows the data range to be specified by the time range of the timestamp, making it easy to collect telemetry data in an appropriate data range.

[0106] (Item 4) In the telemetry data collection system described in item 1, the monitoring manager expresses the transmission conditions by a telemetry identifier that identifies the telemetry data and / or a time range of the time when the telemetry data was acquired. This makes it possible to easily collect appropriate telemetry data, since the transmission conditions are specified by the identifier that identifies the telemetry data and / or the time range of the time stamp.

[0107] (Item 5) In the telemetry data collection system described in item 1, when a state in which telemetry data according to the transmission conditions is not received continues for a predetermined threshold time, the monitoring manager requests the persistence monitoring agent to cancel persistence, and the persistence monitoring agent cancels persistence of the persistence data range. This keeps the telemetry data in the buffer persistent until collection of the telemetry data is completed, ensuring the collection of valid telemetry data.

[0108] (Item 6) In the telemetry data collection system described in item 1, the monitoring manager acquires information on the buffer usage and information on surplus computing resources from the monitoring agents, controls the buffer size of the monitoring agents based on the buffer usage, and controls the transmission rate for transmitting telemetry data from the monitoring agents based on the surplus computing resources. This makes it possible to appropriately control the buffer size and transmission rate of the monitoring agents and effectively collect telemetry data.

[0109] (Item 7) In the telemetry data collection system described in item 1, the monitoring manager sets the transmission conditions using an arithmetic expression or a regular expression to determine the target application and whether the telemetry data should be transmitted. This allows the transmission conditions to be specified using an arithmetic expression or a regular expression, making it possible to appropriately collect effective telemetry data.

[0110] (Item 8) In the telemetry data collection system described in item 7, the monitoring manager provides a graphical user interface for accepting the transmission conditions by text input on a screen, thereby enabling the user to easily set the transmission conditions from the GUI screen.

[0111] (Item 9) In the telemetry data collection system described in item 8, the monitoring manager accepts, by text input on the screen, the conditions for determining whether a buffer flush is necessary, the persistence conditions, and the transmission conditions, and when the conditions and the persistence conditions are input, the monitoring manager displays, as text input, a recommended transmission condition that is set in association with the conditions and the persistence conditions in the set buffer flush conditions. This allows the user to easily set the transmission conditions using the recommendations on the GUI screen. [Explanation of symbols]

[0112] 100...Monitoring manager, 110...Telemetry data reception program, 120...Persistence data identification program, 130...Sequential data extraction program, 140...Transmission rate calculation program, 150...Sub-application logical configuration table, 160...Agent mapping table, 170...Buffer flush condition table, 180...Buffer size table, 190...Related telemetry table, 200...Monitoring agent container, 210...Sub-application ring buffer, 300...Trace data store, 400...Metrics data store, 500...Log data store, 600...Sub-application container, 700...Container runtime, 800...Host, 2300...Buffer flush condition setting screen

Claims

1. A telemetry data collection system that collects telemetry data of an application that is composed of multiple sub-applications, a monitoring agent that obtains telemetry data from the processes of the sub-application; a monitoring manager that receives telemetry data from the monitoring agents; The monitoring agent records the acquired telemetry data in a buffer, and transmits the telemetry data that meets a predetermined condition to the monitoring manager; The monitoring manager, based on the telemetry data received from the monitoring agent, identifies a persistence condition indicating a persistence monitoring agent that is a monitoring agent that should persist the telemetry data recorded in the buffer and a persistence data range that indicates the range of data that should be persisted among the telemetry data recorded in the buffer, and a transmission condition indicating an additional transmission monitoring agent that is a monitoring agent that should collect additional telemetry data and additional transmission telemetry data that is the telemetry data that should be additionally collected; the monitoring manager notifies the persistence monitoring agent of the persistence condition; the persistence monitoring agent persists the persistence data range in accordance with the notified persistence conditions; the monitoring manager notifies the additional transmission monitoring agent of the transmission condition; the additional transmission monitoring agent transmits the additional transmission telemetry data in the buffer to the monitoring manager in accordance with the notified transmission condition; Telemetry data collection system.

2. The monitoring manager When receiving telemetry data transmitted from the additional transmission monitoring agent in accordance with the transmission conditions, re-identifying new transmission conditions indicating a new additional transmission monitoring agent and additional transmission telemetry data based on the received telemetry data; notifying the newly added transmission monitoring agent of the new transmission condition; The telemetry data collection system of claim 1 .

3. The monitoring manager represents the data range in the persistence condition by a time range of the time when the telemetry data was acquired. The telemetry data collection system of claim 1 .

4. The monitoring manager expresses the transmission condition by a telemetry identifier that identifies the telemetry data and / or a time range of the time when the telemetry data was acquired. The telemetry data collection system of claim 1 .

5. the monitoring manager requests the persistence monitoring agent to cancel persistence when a state in which telemetry data according to the transmission condition is not received continues for a predetermined threshold time; the persistence monitoring agent cancels the persistence of the persistence data range; The telemetry data collection system of claim 1 .

6. The monitoring manager acquiring information on the usage amount of the buffer and information on surplus computing resources from the monitoring agent; Controlling the buffer size of the monitoring agent based on the usage of the buffer; controlling a transmission rate at which telemetry data is transmitted from the monitoring agent based on the surplus of the computing resources; The telemetry data collection system of claim 1 .

7. The monitoring manager sets the transmission conditions using an arithmetic expression or a regular expression to determine the target application and whether the telemetry data should be transmitted. The telemetry data collection system of claim 1 .

8. the monitoring manager provides a graphical user interface for accepting the transmission conditions by text input on a screen; 8. The telemetry data collection system of claim 7.

9. The monitoring manager accepts, by text input on the screen, the conditions for determining that a buffer flush is necessary, the persistence conditions, and the transmission conditions, and when the conditions and the persistence conditions are input, displays them as text input recommending transmission conditions that have been set in association with the conditions and the persistence conditions in the set buffer flush conditions.

9. The telemetry data collection system of claim 8.

10. A telemetry data collection device for an application configured with a plurality of sub-applications, which acquires telemetry data from processes of the sub-applications, records the acquired telemetry data in a buffer, and collects the telemetry data from a monitoring agent that transmits telemetry data that meets a predetermined condition, a persistence data specification unit that specifies, based on the telemetry data received from the monitoring agent, a persistence condition indicating a persistence monitoring agent that is a monitoring agent that should persist the telemetry data recorded in the buffer and a persistence data range that indicates the range of data that should be persisted among the telemetry data recorded in the buffer, and a transmission condition indicating an additional transmission monitoring agent that is a monitoring agent that should collect additional telemetry data and additional transmission telemetry data that is the telemetry data that should be collected, and notifies the persistence condition to the persistence monitoring agent; a sequential data extraction unit that notifies the additional transmission monitoring agent of the transmission condition and receives the additional transmission telemetry data in the buffer from the additional transmission monitoring agent in accordance with the transmission condition; A telemetry data collection device having:

11. A telemetry data collection method for collecting telemetry data of an application consisting of a plurality of sub-applications by a monitoring agent that acquires telemetry data from processes of the sub-applications, and a monitoring manager that receives the telemetry data from the monitoring agent, comprising: The monitoring agent records the acquired telemetry data in a buffer, and transmits the telemetry data that meets a predetermined condition to the monitoring manager; The monitoring manager, based on the telemetry data received from the monitoring agent, identifies a persistence condition indicating a persistence monitoring agent that is a monitoring agent that should persist the telemetry data recorded in the buffer and a persistence data range that indicates the range of data that should be persisted among the telemetry data recorded in the buffer, and a transmission condition indicating an additional transmission monitoring agent that is a monitoring agent that should collect additional telemetry data and additional transmission telemetry data that is the telemetry data that should be additionally collected; the monitoring manager notifies the persistence monitoring agent of the persistence condition; the persistence monitoring agent persists the persistence data range in accordance with the notified persistence conditions; the monitoring manager notifies the additional transmission monitoring agent of the transmission condition; the additional transmission monitoring agent transmits the additional transmission telemetry data in the buffer to the monitoring manager in accordance with the notified transmission condition; Telemetry data collection methods.

Images