Monitoring systems, monitoring methods, and programs

Abstract

To detect system anomalies more effectively. [Solution] The monitoring system obtains a plurality of first values ​​corresponding to each of a plurality of target time points included in the monitoring period, each representing a representative value of the replication delay from the primary database to the secondary database during a first period including the corresponding target time point. It also obtains a second value corresponding to each of the plurality of target time points, each representing a representative value of the replication delay during a second period that includes the corresponding target time point and is longer than the first period. If the number of the plurality of target time points in which the corresponding first value is greater than the corresponding second value satisfies the anomaly detection condition, the system outputs an alert regarding replication.

Description

Technical Field

[0001] The present invention relates to a monitoring system, a monitoring method, and a program.

Background Art

[0002] There is a system that synchronizes data between multiple databases (also referred to as replication) and uses the synchronized databases. When performing replication from a primary database to a secondary database, a delay in data writing due to replication occurs. If this delay becomes large, there is a risk of problems in providing services. Furthermore, in database synchronization, the consistency of data may be affected due to reading of data before synchronization or the like.

[0003] In order to quickly address such problems, there is a technology for monitoring the operating status of a system.

Summary of the Invention

Problems to be Solved by the Invention

[0004] In monitoring database replication, many factors related to the operating environment (for example, application configuration and network) affect performance. Then, for example, it is not easy to detect abnormalities with high precision, and there is a risk that abnormalities in replication cannot be adequately addressed.

[0005] An object of the present disclosure is to provide a technology for more appropriately detecting system abnormalities.

Means for Solving the Problems

[0006] (1) A monitoring system that obtains a plurality of first values ​​corresponding to each of a plurality of target time points included in the monitoring period, each representing a typical value of the replication delay from the primary database to the secondary database during a first period including the corresponding target time point; obtains a second value corresponding to each of the plurality of target time points, each representing a typical value of the replication delay during a second period that includes the corresponding target time point and is longer than the first period; and outputs a replication alert when the number of the plurality of target time points in which the corresponding first value is counted to be greater than the corresponding second value satisfies the anomaly detection condition.

[0007] (2) A monitoring system in which, in (1), the plurality of first values ​​each represent a moving average of the replication delay during a first period including the corresponding target time, and the plurality of second values ​​each represent a moving average of the replication delay during a second period including the corresponding target time.

[0008] (3) In (1) or (2), the replication delay is the time from when the information is written to the primary database until when the information is written to the secondary database, a monitoring system.

[0009] (4) In any of (1) to (3), a monitoring system in which each of the multiple target time points is closer to the end of the corresponding second period than to the start of the second period.

[0010] (5) A monitoring system that outputs a replication alert when, in any of (1) to (4), the counted number relative to the number of target time points is greater than a threshold corresponding to the number of target time points.

[0011] (6)(5) The threshold is determined by multiplying the number of target time points by a predetermined ratio, a monitoring system.

[0012] (7) A monitoring system that outputs an alert regarding replication to the administrator when the anomaly detection conditions described in (1) to (6) are met.

[0013] (8) A monitoring method comprising: obtaining a plurality of first values ​​corresponding to each of a plurality of target time points included in the monitoring period, each representing a representative value of the replication delay from the primary database to the secondary database during a first period including the corresponding target time point; obtaining a second value corresponding to each of the plurality of target time points, each representing a representative value of the replication delay during a second period including the corresponding target time point and longer than the first period; and outputting a replication alert when the number of the plurality of target time points in which the corresponding first value is counted to be greater than the corresponding second value satisfies an anomaly detection condition.

[0014] (9) A program to cause a computer to execute a process that obtains a plurality of first values ​​corresponding to each of a plurality of target time points included in the monitoring period, each representing a typical value of the replication delay from the primary database to the secondary database during a first period including the corresponding target time point; obtains a second value corresponding to each of the plurality of target time points, each representing a typical value of the replication delay during a second period longer than the first period including the corresponding target time point; and outputs a replication alert when the number of the plurality of target time points in which the corresponding first value is counted to be greater than the corresponding second value satisfies the anomaly detection condition. [Effects of the Invention]

[0015] This invention makes it possible to more appropriately detect system abnormalities. [Brief explanation of the drawing]

[0016] [Figure 1]This figure shows elements related to an information processing system according to an embodiment of the present invention. [Figure 2] This is a diagram illustrating database replication. [Figure 3] This is a block diagram showing the functions that an information processing system can perform. [Figure 4] This is a flowchart illustrating an example of a process for collecting monitoring data. [Figure 5] This figure shows an example of data stored in a metrics database. [Figure 6] This is a flowchart illustrating an example of a process for detecting anomalies. [Figure 7] This figure shows an example of how the moving average of a delay changes over time. [Modes for carrying out the invention]

[0017] Embodiments of the present invention will be described below with reference to the drawings. For components denoted by the same reference numerals, redundant descriptions will be omitted.

[0018] Figure 1 shows elements related to an information processing system according to an embodiment of the present invention. The information processing system includes a primary database server 1, a secondary database server 2, one or more monitoring servers 3, and one or more application servers 4. The primary database server 1, secondary database server 2, monitoring server 3, and application server 4 are so-called server computers. They communicate with each other via a network.

[0019] The primary database server 1 and the secondary database server 2 provide database services for storing various data. Hereinafter, when referring to them without distinction, they will simply be described as "database servers". Replication processing is executed between the primary database server 1 and the secondary database server 2. As a result, the data of the secondary database server 2 is synchronized with the primary database server 1. In the example of FIG. 1, the primary database server 1 can write to and read from the database, while the secondary database server 2 can only read. The information processing system may include a plurality of primary database servers 1 that cooperate with each other to provide database services, and a plurality of secondary database servers 2 that cooperate with each other to provide database services.

[0020] The monitoring server 3 includes one or more processors 31, one or more storages 32, and one or more communication units 33. The primary database server 1, the secondary database server 2, and the application server 4 also include one or more processors 31, one or more storages 32, and one or more communication units 33. These may be implemented on one or more virtual servers or container platforms.

[0021] The processor 31 operates according to a program (also referred to as instruction code) stored in the storage 32. The processor 31 also controls the communication unit 33. The processor 31 includes, for example, a CPU (Central Processing Unit), and may further include a GPU (Graphic Processing Unit) and an NPU (Neural Processing Unit). Note that the above program may be provided via the Internet or the like, or may be provided by being stored in a computer-readable storage medium such as a flash memory or a DVD-ROM.

[0022] The storage 32 is composed of memory elements such as RAM and flash memory, and external storage devices such as hard disk drives (HDDs) and solid state drives (SSDs). The storage 32 stores the above program. Also, the storage 32 stores information and calculation results input from the processor 31 and the communication unit 33.

[0023] The communication unit 33 is a communication interface for communicating with other devices, such as a network interface card. The communication unit 33 is composed of, for example, an integrated circuit, an antenna, a communication terminal, etc. that realize a wireless LAN or a wired LAN. Based on the control of the processor 31, the communication unit 33 inputs information received from other devices to the processor 31 and the storage 32 via a network, and transmits information to other devices.

[0024] Note that the hardware configurations of the monitoring server 3 and other servers are not limited to the above examples. For example, the monitoring server 3 may include a device for reading a computer-readable information storage medium (e.g., an optical disk drive or a memory card slot) and a device for inputting / outputting data with an external device (e.g., a USB port). The external device may be an input device or an output device.

[0025] The replication process between the primary database server 1 and the secondary database server 2 will be further described. FIG. 2 is a diagram for explaining database replication. FIG. 2 explains the replication process in MySQL (registered trademark).

[0026] [[ID=​​​​More specifically, when data is written to the primary database, the primary database server 1 outputs a log (corresponding to the Binary Logs in Figure 2) indicating the transaction that includes the data write, through a log output process (corresponding to the Binary dump thread in Figure 2). The transaction log includes, for example, a timestamp indicating the time the data was written in the primary database, the item to be written, the data written, and a transaction ID that identifies this transaction. The secondary database server 2 then receives the log (corresponding to the IO thread in Figure 2) through a log reception process and outputs the log (corresponding to the Relay Logs in Figure 2) to its internal storage 32. The secondary database server 2 then writes the data to the secondary database through a log-based write process (corresponding to the SQL thread in Figure 2).

[0028] Generally, replication is performed asynchronously, resulting in a delay between when data is written to the primary database and when it is written to the secondary database. Here, the time it takes to write data is, more precisely, the time it takes for a process called data commit to be executed. Committing in the primary database occurs, for example, immediately after the output of a transaction log (binary log) or immediately after the transmission of that log, while committing in the secondary database occurs during the write process based on the log.

[0029] Figure 2 and the above description illustrate an example of so-called log-distribution type replication. For example, instead of logs, other information such as more logical update information per table or SQL may be sent and received. The database does not have to be a relational database; any database that performs asynchronous replication is acceptable. The primary and secondary databases may be NoSQL databases such as mongoDB® or distributed file management systems such as HADOOP®.

[0030] The following describes how to monitor replication. Figure 3 is a block diagram showing the functions implemented by the information processing system. The primary database server 1 functionally includes a database management system 51 and a monitoring data transmission unit 52. The secondary database server 2 functionally includes a database management system 53 and a monitoring data transmission unit 54. The database management system 51 and the monitoring data transmission unit 52 are implemented by a processor 31 included in the primary database server 1 executing a program (instruction code) stored in the storage 32. The database management system 53 and the monitoring data transmission unit 54 are implemented by a processor 31 included in the secondary database server 2 executing a program (instruction code) stored in the storage 32.

[0031] The database management system 51 provides services for the primary database. When the database management system 51 receives a data write request from the application server 4, it writes the data to the primary database and performs processing to send replication information (for example, the Binary Logs in Figure 2) to the secondary database server 2 (for example, processing for the Binary dump thread in Figure 2, and the transmission processing corresponding to the IO thread).

[0032] The database management system 53 provides services for the secondary database. The database management system 53 receives replication information from the primary database server 1 and writes data written to the primary database to the secondary database. The database management systems 51 and 53 may be implemented by programs for relational databases, for example. The database management systems 51 and 53 may also be implemented by programs for NoSQL databases or distributed file management systems.

[0033] The monitoring data transmission units 52 and 54 are so-called monitoring agents. The monitoring data transmission unit 52 collects metrics from the database management system 51 and sends monitoring data containing those metrics to the monitoring server 3. The monitoring data transmission unit 54 collects metrics from the database management system 53 and sends monitoring data containing those metrics to the monitoring server 3. Metrics are one or more indicators that show the status of a server or service, and include information indicating replication delay. Replication delay is the time from when data is written to the primary database until that data is written to the secondary database. If the database management systems 51 and 53 allow the collection of metrics from external sources, the monitoring data transmission units 52 and 54 may be located on the monitoring server 3.

[0034] Multiple monitoring servers 3 implement a monitoring data acquisition unit 61, a monitoring data manipulation unit 62, an anomaly detection unit 63, and a metrics database 65.

[0035] The monitoring data acquisition unit 61 receives monitoring data from the monitoring data transmission units 52 and 54, and stores the metrics contained in the monitoring data, along with the time when the metrics were collected, in the metrics database 65. The metrics database 65 is mainly composed of storage 32 and stores the collected metrics.

[0036] The monitoring data manipulation unit 62 extracts metrics that meet the specified conditions from the metrics database 65, processes the extracted metrics (for example, by aggregating them), and outputs the processed results.

[0037] With respect to replication, the monitoring data manipulation unit 62 performs the following processing: The monitoring data manipulation unit 62 calculates a plurality of first values ​​that correspond to each of the plurality of target time points included in the monitoring period, and each of these first values ​​represents a representative value of the replication delay in the first period including the corresponding target time point. Here, the monitoring period represents the time subject to aggregation related to monitoring, and the time subject to aggregation is the plurality of target time points (e.g., 10 target time points) that are included in that monitoring period (e.g., 10 minutes) and exist at predetermined intervals (e.g., 1 minute). The representative value may be a moving average (e.g., a simple moving average) of the replication delay in the first period (1 minute) including the corresponding target time point. The target time point is closer to the end of the first period including it than to the start, for example, the end of the first period.

[0038] The monitoring data manipulation unit 62 also calculates a number of second values ​​corresponding to each of the multiple target time points, each representing a representative value of the replication delay in the second period that includes the corresponding target time point. Here, the second period is longer than the first period (for example, 5 minutes), and this representative value may be a moving average of the replication delay in the second period (5 minutes) that includes the corresponding target time point. The target time point is closer to the end of the second period that includes it than to the start, for example, at the end of that second period. The lengths of the first and second periods are predetermined.

[0039] The monitoring data manipulation unit 62 also calculates the number of times, among multiple target time points, when the corresponding first value is greater than the corresponding second value.

[0040] The monitoring data acquisition unit 61 and the monitoring data manipulation unit 62 may be implemented using a known monitoring tool such as Prometheus. When Prometheus is used, the monitoring data transmission units 52 and 54 are also called exporters.

[0041] The anomaly detection unit 63 outputs an alert when the processed metrics meet the anomaly detection conditions. The anomaly detection unit 63 outputs a replication-related alert when the number of counts where the corresponding first value is greater than the corresponding second value meets the anomaly detection conditions. The anomaly detection conditions may be that the counted number is greater than a threshold, which may be determined by multiplying the number of target time points included in the monitoring period by a predetermined percentage (e.g., 70%). The anomaly detection unit 63 may be implemented by having a script executed by a well-known tool such as Grafana® or AlertManager, or by having a script executed by another monitoring tool.

[0042] The following section further describes the processes performed by the monitoring server 3. Figure 4 is a flowchart showing an example of the process for collecting monitoring data. This process is executed each time monitoring data is received from the database server. The process shown in Figure 4 may be executed at regular intervals.

[0043] First, the monitoring data acquisition unit 61 acquires monitoring data sent from the database server (S101). Then, the monitoring data acquisition unit 61 writes the metrics included in the monitoring data, along with the time when the metrics were acquired, to the metrics database 65 (S102).

[0044] Figure 5 shows an example of data stored in the metrics database 65. In this example, the database server sends monitoring data each time it writes data to the database, and this monitoring data includes a timestamp and information indicating the replication delay as a metric. The information indicating the replication delay is stored in association with the timestamp. The information indicating the replication delay may be the replication delay time calculated by the database management system 53. The timestamp may be the time the replication delay was obtained from the database management system 53.

[0045] Here, the replication delay time may be the difference between the time it took to write the data included in the transaction to the secondary database and the time it took to write the same data included in the same transaction to the primary database. If the log sent from the primary database to the secondary database includes the time it took to write the data to the primary database, the database management system 53 can easily calculate the replication delay.

[0046] Information indicating replication delay may include the time it took to write data included in a transaction to the secondary database, and the time it took to write the same data included in the same transaction to the primary database.

[0047] Here, the secondary database server 2 may send monitoring data including a timestamp, the transaction ID of the transaction, the write time of the transaction, and the replication delay time, and the monitoring data acquisition unit 61 may acquire this monitoring data and store it as metrics in the metrics database 65. Furthermore, the primary database server 1 may send monitoring data including a timestamp, the transaction ID of the transaction, and the write time of the transaction, and the monitoring data acquisition unit 61 may acquire this monitoring data and store it as metrics in the metrics database 65.

[0048] In this case, the replication delay can be calculated by taking into account data that has not yet been written to the secondary database. Specifically, the monitoring data acquisition unit 61 may periodically perform the following processing (at intervals of less than or equal to the first period, for example, every 30 seconds) to calculate the replication delay and store it in the metrics database 65.

[0049] First, the monitoring data acquisition unit 61 obtains the transaction ID of the latest write to the secondary database from the metrics database 65. Next, the monitoring data acquisition unit 61 obtains the write time of the next write after the write identified by that transaction ID for the primary database from the metrics database 65, and calculates the difference between that write time and the current time. If the calculated difference is greater than the latest delay sent from the secondary database server 2 and stored in the metrics database 65, the monitoring data acquisition unit 61 stores that difference in the metrics database 65 as the replication delay at the current time. This makes it possible to detect delays even when replication is almost stopped due to network anomalies, for example.

[0050] Next, we will describe the process of detecting anomalies based on the information stored in the metrics database 65. Figure 6 is a flowchart showing an example of the process of detecting anomalies. The flow shown in Figure 6 is executed at predetermined repetition intervals (for example, every 10 minutes). The process shown in Figure 6 (especially S201 to S206) may be executed, for example, by an anomaly detection unit 63 executing a set script (program), outputting an aggregation command to the monitoring data operation unit 62, and the monitoring data operation unit 62 performing aggregation processing based on that command.

[0051] In steps S201 to S206, the monitoring data manipulation unit 62 calculates a first value and a second value for each of the multiple target time points within the monitoring period, and then performs an aggregation process to count the number of times among the multiple target time points in which the corresponding first value is greater than the corresponding second value. Before the process in S201, the monitoring period is set to a predetermined length of time up to the start time of the process in Figure 6, and the multiple target time points included within it are determined. The monitoring period may also be defined by the number of target time points included within it. Alternatively, instead of the start time of the process, a time offset by a predetermined amount of time within the first period may be used.

[0052] In the aggregation process, the monitoring data manipulation unit 62 first determines the first target time point within the monitoring period as the target time point for processing (S201). Multiple target time points may be arranged in chronological order or by other criteria.

[0053] The monitoring data manipulation unit 62 obtains the average of the replication delay values ​​for the target time point in the first period as the first value (S202). In the example in Figure 6, the first period is 1 minute up to the target time point. The monitoring data manipulation unit 62 calculates the average of the replication delays stored in the metrics database 65, which are associated with the timestamps belonging to the first period. This average corresponds to a moving average.

[0054] The monitoring data manipulation unit 62 also obtains the average of the replication delay values ​​over a second period for the target time point of the processing target as the second value (S203). In the example in Figure 6, the second period is 5 minutes up to the target time point. The monitoring data manipulation unit 62 calculates the average of the replication delays stored in the metrics database 65, which are associated with the timestamps belonging to the second period. This average corresponds to a moving average.

[0055] The monitoring data manipulation unit 62 determines whether the first value is greater than the second value for the target time point of the processing target (S204). If the first value is greater than the second value (Y in S204), the monitoring data manipulation unit 62 increments the counter by 1 (S205). If the first value is less than or equal to the second value (N in S204), S205 is skipped.

[0056] If aggregation (processing in S202-S204) has not been performed for all target time points (N in S206), the monitoring data manipulation unit 62 determines the next target time point from among the multiple target time points to be processed (S207), and repeats the processing from S202 onwards.

[0057] If the data has been aggregated for all target time points (Y in S206), the anomaly detection unit 63 determines whether the counter value has exceeded a threshold (S208). This threshold may be a predetermined percentage multiplied by the number of target time points. The predetermined percentage is greater than 50%, for example, 70%.

[0058] If the counter value exceeds the threshold in S208 (Y in S208), the anomaly detection unit 63 outputs an alert indicating a replication anomaly to the administrator (S209). The anomaly detection unit 63 may send the alert by email, by a chat service such as Slack®, by notification by SMS / telephone, or by push notification to a smartphone. The anomaly detection unit 63 may also output the alert on the screen of a display device.

[0059] If the counter value does not exceed the threshold (N in S208), the process shown in Figure 6 is terminated. The anomaly detection unit 63 may also determine in S208 whether the value obtained by dividing the counter value by the number of target time points exceeds a predetermined percentage. If it exceeds the predetermined percentage, the process in S209 is executed.

[0060] Figure 7 shows an example of the time evolution of the moving average of delay. In the graph shown in Figure 7, the vertical axis represents delay (s), and the horizontal axis represents the elapsed time since the start of monitoring. The elapsed time corresponds to the target time point. The values ​​of the markers connected by solid lines show the simple moving average of the delay over 1 minute, and the values ​​of the markers connected by dashed lines show the simple moving average of the delay over 5 minutes.

[0061] In Figure 7, an arrow pointing upwards to the right is shown in the section from approximately 37 minutes to 113 minutes of elapsed time. In this section, the lag is on an upward trend. When the lag is on an upward trend, the 1-minute moving average increases more quickly and is more likely to exceed the 5-minute moving average. In particular, if the target time point is in the latter half of the moving average period, the 1-minute moving average is more likely to exceed the 5-minute moving average. In the example in Figure 7, a replication anomaly can be detected within 10 minutes of a significant increase in lag.

[0062] The information processing system according to this embodiment can detect an increase in delay earlier than methods that determine when the absolute value of measured values ​​such as delay exceeds a threshold. By detecting anomalies early, administrators can respond quickly. Furthermore, anomalies are continuously detected while the latency in replication is increasing. As a result, administrators are continuously notified of alerts, and there is no risk of them forgetting to take action.

[0063] Furthermore, in this embodiment, the threshold is related to the percentage of the 1-minute moving average exceeding the 5-minute moving average, and is not related to the absolute value of the moving average. Experimental determination of the threshold is unnecessary. Therefore, the monitoring server 3 according to this embodiment can be easily applied even when there are significant differences in individual environments such as the database server configuration and network. For example, it can be applied even when the type of database executed by the database management systems 51 and 53 is not a relational database (e.g., NoSQL), and it can also be applied when network latency differs significantly.

[0064] Furthermore, because it can detect increases in delays with high accuracy, it can prevent the burden on administrators caused by false positives and delays in response due to misidentification of false positives. [Explanation of Symbols]

[0065] 1 Primary database server, 2 Secondary database server, 3 Monitoring server, 4 Application server, 31 Processor, 32 Storage, 33 Communication unit, 51, 53 Database management system, 52, 54 Monitoring data transmission unit, 61 Monitoring data acquisition unit, 62 Monitoring data manipulation unit, 63 Anomaly detection unit, 65 Metrics database.

Claims

1. Obtain a plurality of first values ​​corresponding to each of the plurality of target time points included in the monitoring period, each representing a representative value of the replication delay from the primary database to the secondary database during the first period including the corresponding target time point. A second value is obtained that corresponds to each of the multiple target time points and represents a representative value of the replication delay during a second period that is longer than the first period and includes the corresponding target time point. If, among the multiple target time points, the number of instances where the corresponding first value is greater than the corresponding second value satisfies the anomaly detection condition, an alert regarding replication is output. A monitoring system.

2. A monitoring system according to claim 1, Each of the aforementioned plurality of first values ​​represents the moving average of the replication delay during the first period, which includes the corresponding target time point. The plurality of second values ​​each represent the moving average of the replication delay during the second period, which includes the corresponding target time point. Monitoring system.

3. A monitoring system according to claim 1, The replication delay is the time from when the information is written to the primary database until the information is written to the secondary database. Monitoring system.

4. A monitoring system according to claim 1, Each of the aforementioned multiple target time points is closer to the end than the start of the corresponding second period. Monitoring system.

5. A monitoring system according to claim 1, If the counted number for the number of target time points is greater than a threshold corresponding to the number of target time points, an alert regarding replication is output. A monitoring system.

6. A monitoring system according to claim 5, The threshold is determined by multiplying the number of target time points by a predetermined ratio. Monitoring system.

7. A monitoring system according to claim 1, When the aforementioned anomaly detection conditions are met, an alert regarding the replication is sent to the administrator. Monitoring system.

8. The steps include obtaining a plurality of first values, each corresponding to a plurality of target time points included in the monitoring period, which represent a typical value of the replication delay from the primary database to the secondary database during the first period including the corresponding target time point, A step of obtaining a second value that corresponds to each of the multiple target time points and indicates a representative value of the replication delay during a second period that includes the corresponding target time point and is longer than the first period, The steps include outputting a replication alert when, among the multiple target time points, the number of instances where the corresponding first value is greater than the corresponding second value satisfies the anomaly detection condition, A monitoring method that includes this.

9. Obtain a plurality of first values ​​corresponding to each of the plurality of target time points included in the monitoring period, each representing a representative value of the replication delay from the primary database to the secondary database during the first period including the corresponding target time point. A second value is obtained that corresponds to each of the multiple target time points and represents a representative value of the replication delay during a second period that is longer than the first period and includes the corresponding target time point. If, among the multiple target time points, the number of instances where the corresponding first value is greater than the corresponding second value satisfies the anomaly detection condition, an alert regarding replication is output. A program that causes a computer to perform a process.

Images