Monitoring system, monitoring method and program

Abstract

Better detection of system anomalies. [Solution] The monitoring system acquires a plurality of first values ​​corresponding to each of a plurality of target points included in a monitoring period, each of which indicates a representative value of the replication delay from the primary database to the secondary database during a first period including the corresponding target points; acquires second values ​​corresponding to each of the plurality of target points and indicating a representative value of the replication delay during a second period including the corresponding target points and longer than the first period; and outputs an alert regarding replication when the number of the plurality of target points where the corresponding first value is counted as being greater than the corresponding second value satisfies an abnormality detection condition.

Description

[Technical Field]

[0001] The present invention relates to a monitoring system, a monitoring method, and a program. [Background technology]

[0002] There are systems that synchronize data between multiple databases (also known as replication) and use the synchronized databases. When replicating from a primary database to a secondary database, a delay occurs in writing data due to replication. If this delay becomes too large, it could cause problems in providing services. Furthermore, when synchronizing databases, there is a risk that data consistency may be affected due to factors such as reading data before synchronization.

[0003] In order to deal with such problems early, there are techniques for monitoring the operational status of a system. Summary of the Invention [Problem to be solved by the invention]

[0004] When monitoring database replication, many factors related to the operating environment (such as application configuration and network) affect performance. This means that it is not easy to detect anomalies with high accuracy, and there is a risk that replication anomalies cannot be adequately addressed.

[0005] An object of the present disclosure is to provide a technology for more appropriately detecting system abnormalities. [Means for solving the problem]

[0006] (1) A monitoring system that acquires a plurality of first values ​​corresponding to a plurality of target points included in a monitoring period, each of which indicates a representative value of the replication delay from a primary database to a secondary database during a first period including the corresponding target points; acquires a plurality of second values ​​corresponding to the plurality of target points and indicating a representative value of the replication delay during a second period including the corresponding target points and longer than the first period; and outputs an alert regarding replication when the number of the plurality of target points whose corresponding first value is counted as being greater than the corresponding second value satisfies an abnormality detection condition.

[0007] (2) In (1), the plurality of first values ​​each indicate a moving average of the replication delay during the first period including a corresponding target time point, and the plurality of second values ​​each indicate a moving average of the replication delay during the second period including a corresponding target time point.

[0008] (3) In the monitoring system according to (1) or (2), the replication delay is the time between when information is written to the primary database and when the information is written to the secondary database.

[0009] (4) In any one of (1) to (3), a monitoring system wherein each of the plurality of target time points is closer to the end than the start of the corresponding second period.

[0010] (5) In any of (1) to (4), a monitoring system outputs an alert regarding replication when the counted number for the number of target time points is greater than a threshold value corresponding to the number of target time points.

[0011] (6) In (5), the threshold value is calculated by multiplying the number of the plurality of target time points by a predetermined ratio.

[0012] (7) A monitoring system according to any one of (1) to (6), wherein when the abnormality detection condition is satisfied, an alert regarding the replication is output to an administrator.

[0013] (8) A monitoring method including the steps of: acquiring a plurality of first values ​​corresponding to a plurality of target points included in a monitoring period, each of the first values ​​indicating a representative value of the replication delay from the primary database to the secondary database during a first period including the corresponding target points; acquiring second values ​​corresponding to the plurality of target points and indicating a representative value of the replication delay during a second period including the corresponding target points and longer than the first period; and outputting an alert regarding replication when the number of the plurality of target points whose corresponding first value is counted as being greater than the corresponding second value satisfies an abnormality detection condition.

[0014] (9) A program for causing a computer to execute a process of acquiring a plurality of first values ​​corresponding to each of a plurality of target points included in a monitoring period, each of which indicates a representative value of the replication delay from the primary database to the secondary database during a first period including the corresponding target points; acquiring second values ​​corresponding to each of the plurality of target points and indicating a representative value of the replication delay during a second period including the corresponding target points and longer than the first period; and outputting an alert regarding replication when the number of the plurality of target points for which the corresponding first value is counted as being greater than the corresponding second value satisfies an abnormality detection condition. [Effects of the Invention]

[0015] The present invention allows for more appropriate detection of system abnormalities. [Brief explanation of the drawings]

[0016] [Figure 1]FIG. 1 is a diagram illustrating elements related to an information processing system according to an embodiment of the present invention. [Figure 2] FIG. 1 is a diagram illustrating database replication. [Figure 3] FIG. 2 is a block diagram showing functions realized by the information processing system. [Figure 4] FIG. 10 is a flow diagram illustrating an example of a process for collecting monitoring data. [Figure 5] FIG. 2 is a diagram illustrating an example of data stored in a metrics database. [Figure 6] FIG. 10 is a flowchart illustrating an example of a process for detecting an abnormality. [Figure 7] FIG. 10 is a diagram illustrating an example of a time change in the moving average of delay. DETAILED DESCRIPTION OF THE INVENTION

[0017] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. Duplicate descriptions of components with the same reference numerals will be omitted.

[0018] 1 is a diagram showing elements related to an information processing system according to an embodiment of the present invention. The information processing system includes a primary database server 1, a secondary database server 2, one or more monitoring servers 3, and one or more application servers 4. The primary database server 1, the secondary database server 2, the monitoring server 3, and the application servers 4 are so-called server computers. These communicate with each other via a network.

[0019] The primary database server 1 and secondary database server 2 provide database services that store various types of data. Hereinafter, when there is no need to distinguish between them, they will simply be referred to as "database servers." Replication processing is performed between the primary database server 1 and the secondary database server 2. This synchronizes the data in the secondary database server 2 with the primary database server 1. In the example of Figure 1, the primary database server 1 can write to and read from the database, while the secondary database server 2 can only read from it. An information processing system may include multiple primary database servers 1 that work together to provide database services, and multiple secondary database servers 2 that work together to provide database services.

[0020] The monitoring server 3 includes one or more processors 31, one or more storages 32, and one or more communication units 33. The primary database server 1, the secondary database server 2, and the application server 4 also include one or more processors 31, one or more storages 32, and one or more communication units 33. These may be implemented on one or more virtual server or container platforms.

[0021] The processor 31 operates according to a program (also referred to as an instruction code) stored in the storage 32. The processor 31 also controls the communication unit 33. The processor 31 includes, for example, a CPU (Central Processing Unit), and may further include a GPU (Graphic Processing Unit) and an NPU (Neural Processing Unit). The program may be provided via the Internet or the like, or may be provided by being stored in a computer-readable storage medium such as a flash memory or a DVD-ROM.

[0022] The storage 32 is composed of memory elements such as RAM and flash memory, and external storage devices such as a hard disk drive (HDD) and a solid state drive (SSD). The storage 32 stores the above programs. The storage 32 also stores information input from the processor 31 and the communication unit 33 and calculation results.

[0023] The communication unit 33 is a communication interface, such as a network interface card, that communicates with other devices. The communication unit 33 is configured with an integrated circuit, an antenna, a communication terminal, etc. that realizes a wireless LAN or a wired LAN. Based on the control of the processor 31, the communication unit 33 inputs information received from other devices via a network to the processor 31 or the storage 32, and transmits the information to other devices.

[0024] The hardware configuration of the monitoring server 3 and other servers is not limited to the above example. For example, the monitoring server 3 may include a device for reading a computer-readable information storage medium (e.g., an optical disk drive or a memory card slot) or a device for inputting and outputting data to and from an external device (e.g., a USB port). The external device may be an input device or an output device.

[0025] The following further describes the replication process between the primary database server 1 and the secondary database server 2. Figure 2 is a diagram illustrating database replication. Figure 2 explains the replication process in MySQL (registered trademark).

[0026] In order to synchronize the data in the two database servers, when data is written to the database (primary database) of primary database server 1 at the request of application server 4, secondary database server 2 writes the same data to its own database (secondary database).

[0027] More specifically, when data is written to the primary database, the primary database server 1 outputs a log (corresponding to the Binary Logs in FIG. 2) indicating the transaction that included the writing of that data through log output processing (corresponding to the Binary dump thread in FIG. 2). The log indicating the transaction includes, for example, a timestamp indicating the time the data was written in the primary database, the item to be written, the written data, and a transaction ID that identifies this transaction. The secondary database server 2 then receives the log through log reception processing (corresponding to the IO thread in FIG. 2) and outputs the log (corresponding to the Relay Logs in FIG. 2) to its internal storage 32. The secondary database server 2 writes the data to the secondary database through a write processing based on the log (corresponding to the SQL thread in FIG. 2).

[0028] Replication is generally performed asynchronously, which means there is a delay between when data is written to the primary database and when it is written to the secondary database. Strictly speaking, the time it takes to write data is the time it takes to execute a process called a data commit. A commit in a primary database occurs, for example, immediately after a binary log indicating the transaction is output or immediately after the log is sent, while a commit in a secondary database occurs during the write process based on the log.

[0029] Figure 2 and the above description are examples of so-called log-shipping replication. For example, other information, such as more logical update information for each table or SQL, may be sent and received instead of logs. Furthermore, the databases are not limited to relational databases as long as they perform asynchronous replication. The primary and secondary databases may be, for example, NoSQL databases such as MongoDB (registered trademark) or distributed file management systems such as HADOOP (registered trademark).

[0030] A method for monitoring replication will be described below. Figure 3 is a block diagram showing functions realized by the information processing system. The primary database server 1 functionally includes a database management system 51 and a monitoring data transmission unit 52. The secondary database server 2 functionally includes a database management system 53 and a monitoring data transmission unit 54. The database management system 51 and the monitoring data transmission unit 52 are realized by the processor 31 included in the primary database server 1 executing a program (instruction code) stored in the storage 32. The database management system 53 and the monitoring data transmission unit 54 are realized by the processor 31 included in the secondary database server 2 executing a program (instruction code) stored in the storage 32.

[0031] The database management system 51 provides services for the primary database. When the database management system 51 receives a request to write data from the application server 4, it writes the data to the primary database and performs processing (for example, processing of the Binary dump thread in FIG. 2 and transmission processing corresponding to the IO thread) to send replication information (for example, Binary Logs in FIG. 2) to the secondary database server 2.

[0032] The database management system 53 provides secondary database services. The database management system 53 receives replication information from the primary database server 1 and writes data written to the primary database to the secondary database. The database management systems 51 and 53 may be realized by, for example, a relational database program. The database management systems 51 and 53 may also be realized by a NoSQL database or a distributed file management system program.

[0033] The monitoring data transmitters 52 and 54 are so-called monitoring agents. The monitoring data transmitter 52 collects metrics in the database management system 51 and transmits monitoring data including the metrics to the monitoring server 3. The monitoring data transmitter 54 collects metrics in the database management system 53 and transmits monitoring data including the metrics to the monitoring server 3. Metrics are one or more indicators that indicate the status of a server or service, and include information indicating replication delay. Replication delay is the time between when data is written to the primary database and when that data is written to the secondary database. If the database management systems 51 and 53 enable the collection of metrics from outside, the monitoring data transmitters 52 and 54 may be located on the monitoring server 3.

[0034] The plurality of monitoring servers 3 implement a monitoring data acquisition unit 61 , a monitoring data operation unit 62 , an abnormality detection unit 63 , and a metrics database 65 .

[0035] The monitoring data acquisition unit 61 receives monitoring data from the monitoring data transmission units 52 and 54, and stores the metrics included in the monitoring data together with the time the metrics were collected in the metrics database 65. The metrics database 65 is mainly composed of the storage 32, and stores the collected metrics.

[0036] The monitoring data operation unit 62 extracts metrics that satisfy the conditions from the metrics database 65, processes (for example, aggregates) the extracted metrics, and outputs the processed results.

[0037] With regard to replication, the monitoring data operation unit 62 performs the following processing. The monitoring data operation unit 62 calculates a plurality of first values ​​corresponding to a plurality of target time points included in the monitoring period, each of which indicates a representative value of the replication delay for the first period including the corresponding target time point. Here, the monitoring period indicates the time period to be aggregated for monitoring, and the aggregation targets a plurality of target time points (e.g., 10 target time points) included in the monitoring period (e.g., 10 minutes) and present at a predetermined interval (e.g., 1 minute). The representative value may be a moving average (e.g., a simple moving average) of the replication delay for the first period (1 minute) including the corresponding target time point. The target time point may be closer to the end of the first period than the start of the first period including the target time point, for example, the end of the first period.

[0038] The monitoring data manipulation unit 62 also calculates a plurality of second values ​​corresponding to the plurality of target time points and each representing a representative value of the replication delay during a second period including the corresponding target time point. Here, the second period is longer than the first period (e.g., 5 minutes), and the representative value may be a moving average of the replication delay during the second period (5 minutes) including the corresponding target time point. The target time point may be closer to the end of the second period than the start thereof, for example, the end of the second period. The lengths of the first period and the second period are each predetermined.

[0039] Furthermore, the monitoring data operating unit 62 calculates the number of times at which the corresponding first value is counted as being greater than the corresponding second value, among the plurality of target time points.

[0040] The monitoring data acquisition unit 61 and the monitoring data operation unit 62 may be realized by a known monitoring tool such as Prometheus. When Prometheus is used, the monitoring data transmission units 52 and 54 are also called exporters.

[0041] The anomaly detection unit 63 outputs an alert when the processed metrics satisfy the anomaly detection condition. The anomaly detection unit 63 outputs an alert regarding replication when the counted number of times when the corresponding first value is greater than the corresponding second value satisfies the anomaly detection condition. The anomaly detection condition may be that the counted number is greater than a threshold value, and the threshold value may be calculated by multiplying the number of target time points included in the monitoring period by a predetermined percentage (e.g., 70%). The anomaly detection unit 63 may be implemented by having a known tool such as Grafana (registered trademark) or AlertManager execute a script, or may be implemented by having another monitoring tool execute a script.

[0042] The following further describes the processing executed by the monitoring server 3. Figure 4 is a flow diagram showing an example of the processing for collecting monitoring data. This processing is executed every time monitoring data is received from the database server. The processing shown in Figure 4 may be executed at regular intervals.

[0043] First, the monitoring data acquisition unit 61 acquires monitoring data transmitted from the database server (S101), and then writes the metrics included in the monitoring data into the metrics database 65 together with the time when the metrics were acquired (S102).

[0044] FIG. 5 is a diagram showing an example of data stored in the metrics database 65. In the example shown in this figure, a database server sends monitoring data each time it writes data to a database, and the monitoring data includes a timestamp and information indicating replication delay as a metric. The information indicating replication delay is stored in association with the timestamp. The information indicating replication delay may be the time of replication delay calculated by the database management system 53. The timestamp may be the time when the replication delay is obtained from the database management system 53.

[0045] Here, the replication delay may be the difference between the time when data included in a transaction in the secondary database is written and the time when the data included in the same transaction is written to the primary database. If the log sent from the primary database to the secondary database includes the time when the data was written to the primary database, the database management system 53 can easily calculate the replication delay.

[0046] The information indicating the replication delay may be the time when data included in a transaction in the secondary database is written and the time when data included in the same transaction is written to the primary database.

[0047] Here, the secondary database server 2 may transmit a timestamp, the transaction ID of the transaction, the write time of the transaction, and the replication delay time as monitoring data, and the monitoring data acquisition unit 61 may acquire this monitoring data and store it as metrics in the metrics database 65. Furthermore, the primary database server 1 may transmit a timestamp, the transaction ID of the transaction, and the write time of the transaction as monitoring data, and the monitoring data acquisition unit 61 may acquire this monitoring data and store it as metrics in the metrics database 65.

[0048] In this case, the replication delay can be calculated taking into account data that has not yet been written to the secondary database. Specifically, the monitoring data acquisition unit 61 may execute the following process periodically (at a cycle equal to or shorter than the first period, for example, every 30 seconds) to calculate the replication delay and store it in the metrics database 65.

[0049] First, the monitoring data acquisition unit 61 acquires the transaction ID of the latest write to the secondary database from the metrics database 65. Next, the monitoring data acquisition unit 61 acquires from the metrics database 65 the write time of the write following the write identified by that transaction ID for the primary database, and calculates the difference between that write time and the current time. If the calculated difference is greater than the latest delay sent from the secondary database server 2 and stored in the metrics database 65, the monitoring data acquisition unit 61 stores that difference in the metrics database 65 as the replication delay at the current time. This makes it possible to detect delays even when replication has almost stopped due to, for example, a network abnormality.

[0050] Next, a process for detecting an abnormality based on information stored in the metrics database 65 will be described. FIG. 6 is a flow diagram showing an example of the process for detecting an abnormality. The flow shown in FIG. 6 is executed at predetermined repetition intervals (for example, every 10 minutes). The process shown in FIG. 6 (particularly S201 to S206) may be executed, for example, by the abnormality detection unit 63 executing a set script (program) outputting a command related to aggregation to the monitoring data operation unit 62, and the monitoring data operation unit 62 performing the process related to aggregation based on the command.

[0051] In S201 to S206, the monitoring data operation unit 62 calculates a first value and a second value for each of a plurality of target time points within the monitoring period, and further performs a counting process to count the number of target time points for which the corresponding first value is counted as being greater than the corresponding second value. Before the process of S201, the monitoring period is set to a period of a predetermined length up to the start time of the process of FIG. 6, and the plurality of target time points included therein are determined. The monitoring period may be defined by the number of target time points included therein. Note that instead of the start time of the process, a time offset by a predetermined time within the first period may be used.

[0052] In the aggregation process, the monitoring data operation unit 62 first determines the first target time point within the monitoring period as the target time point to be processed (S201). Note that the multiple target time points may be arranged in chronological order or by other criteria.

[0053] The monitoring data operation unit 62 acquires the average of the replication delay values ​​in a first period for the target time point to be processed as a first value (S202). In the example of FIG. 6, the first period is one minute up to the target time point. The monitoring data operation unit 62 calculates the average of the replication delays stored in the metrics database 65 in association with the timestamps belonging to the first period. This average corresponds to a moving average.

[0054] The monitoring data operation unit 62 also acquires the average of the replication delay values ​​in the second period for the target time point being processed as a second value (S203). In the example of FIG. 6, the second period is the five minutes up to the target time point. The monitoring data operation unit 62 calculates the average of the replication delays stored in the metrics database 65 in association with the timestamps belonging to the second period. This average corresponds to a moving average.

[0055] The monitoring data operation unit 62 determines whether the first value is greater than the second value for the target time point to be processed (S204). If the first value is greater than the second value (Y in S204), the monitoring data operation unit 62 increments the counter by 1 (S205). If the first value is equal to or less than the second value (N in S204), S205 is skipped.

[0056] If aggregation (processing of S202 to S204) has not been performed for all target time points (N in S206), the monitoring data operation unit 62 determines the next target time point among the multiple target time points as the target time point to be processed (S207), and repeats the processing from S202 onwards.

[0057] If the counting has been completed for all target time points (Y in S206), the anomaly detection unit 63 determines whether the counter value has exceeded a threshold value (S208). This threshold value may be a number obtained by multiplying the number of target time points by a predetermined percentage. The predetermined percentage is greater than 50%, for example, 70%.

[0058] If the counter value exceeds the threshold in S208 (Y in S208), the abnormality detection unit 63 outputs an alert indicating a replication abnormality to the administrator (S209). The abnormality detection unit 63 may send the alert by email, by a chat service such as Slack (registered trademark), by SMS / phone call, or by push notification to a smartphone. The abnormality detection unit 63 may output the alert on the screen of a display device.

[0059] If the counter value does not exceed the threshold value (N in S208), the process shown in Fig. 6 ends. Note that the anomaly detection unit 63 may determine in S208 whether the value obtained by dividing the counter value by the number of target time points exceeds a predetermined ratio. If the predetermined ratio is exceeded, the process of S209 is executed.

[0060] Figure 7 is a diagram showing an example of the change over time in the moving average of delay. In the graph shown in Figure 7, the vertical axis represents delay (s) and the horizontal axis represents the elapsed time from the start of monitoring. The elapsed time corresponds to the target time point. The values ​​of the markers connected by a solid line represent the simple moving average of delay over one minute, and the values ​​of the markers connected by a dashed line represent the simple moving average of delay over five minutes.

[0061] In Figure 7, an arrow pointing to the upper right is drawn in the section from approximately 37 minutes to approximately 113 minutes elapsed. In this section, the lag is on an upward trend. When the lag is on an upward trend, the 1-minute moving average increases more quickly and is more likely to exceed the 5-minute moving average. In particular, if the target time falls in the latter half of the moving average period, the 1-minute moving average is more likely to exceed the 5-minute moving average. In the example in Figure 7, a replication anomaly can be detected within 10 minutes of a significant increase in lag.

[0062] The information processing system according to this embodiment can detect an increase in delay earlier than a method that determines whether the absolute value of a measurement value, such as delay, exceeds a threshold. Early detection of anomalies allows the administrator to respond quickly. Furthermore, while replication delays are increasing, anomalies are continuously detected. This allows the administrator to receive continuous alerts, eliminating the risk of the administrator forgetting to respond.

[0063] Furthermore, in this embodiment, the threshold value relates to the rate at which the one-minute moving average exceeds the five-minute moving average, and is not related to the absolute value of the moving average. There is no need to experimentally determine the threshold value. Therefore, the monitoring server 3 according to this embodiment can be easily applied even when there are significant differences in individual environments, such as the configuration of the database server or the network. For example, it can be applied even when the type of database executed by the database management systems 51 and 53 is not a relational database (for example, in the case of NoSQL), and it can also be applied when network latency differs significantly.

[0064] In addition, since it can detect increased delays with high accuracy, it can prevent the burden on administrators caused by false positives and delays in response due to mistaking a detection for a false positive. [Explanation of symbols]

[0065] 1 Primary database server, 2 Secondary database server, 3 Monitoring server, 4 Application server, 31 Processor, 32 Storage, 33 Communication unit, 51, 53 Database management system, 52, 54 Monitoring data transmission unit, 61 Monitoring data acquisition unit, 62 Monitoring data operation unit, 63 Anomaly detection unit, 65 Metrics database.

Claims

1. acquiring a plurality of first values ​​corresponding to a plurality of target time points included in a monitoring period, each of the first values ​​indicating a representative value of a replication delay from the primary database to the secondary database during a first period including the corresponding target time points; acquiring second values ​​that correspond to the plurality of target time points, respectively, and indicate a representative value of the replication delay in a second period that includes the corresponding target time points and is longer than the first period; outputting an alert regarding replication when the number of times at which the corresponding first value is counted as being greater than the corresponding second value among the plurality of target time points satisfies an abnormality detection condition; Surveillance system.

2. 2. The monitoring system of claim 1, each of the plurality of first values ​​represents a moving average of the replication lag during the first time period including a corresponding target time point; each of the plurality of second values ​​indicates a moving average of the replication delay during the second period including a corresponding target time point; Surveillance system.

3. 2. The monitoring system of claim 1, The replication lag is the time between when information is written to the primary database and when the information is written to the secondary database. Surveillance system.

4. 2. The monitoring system of claim 1, Each of the plurality of target time points is closer to the end of the corresponding second period than to the start of the corresponding second period. Surveillance system.

5. 2. The monitoring system of claim 1, outputting an alert regarding replication when the counted number for the number of the plurality of target time points is greater than a threshold value according to the number of target time points; Surveillance system.

6. 6. The monitoring system of claim 5, The threshold value is obtained by multiplying the number of the plurality of target time points by a predetermined ratio. Surveillance system.

7. 2. The monitoring system of claim 1, outputting an alert regarding the replication to an administrator when the abnormality detection condition is satisfied; Surveillance system.

8. A monitoring system including a monitoring data manipulation unit and an anomaly detection unit, the monitoring data operation unit acquiring a plurality of first values ​​corresponding to a plurality of target time points included in a monitoring period, each of the first values ​​indicating a representative value of a replication delay from the primary database to the secondary database during a first period including the corresponding target time points; the monitoring data operation unit acquiring second values ​​that correspond to the plurality of target time points, respectively, and indicate representative values ​​of the replication delay in a second period that includes the corresponding target time points and is longer than the first period; the anomaly detection unit outputs an alert regarding replication when the number of times at which the corresponding first value is greater than the corresponding second value among the plurality of target time points satisfies an anomaly detection condition; Monitoring methods including:

9. acquiring a plurality of first values ​​corresponding to a plurality of target time points included in a monitoring period, each of the first values ​​indicating a representative value of a replication delay from the primary database to the secondary database during a first period including the corresponding target time points; acquiring second values ​​that correspond to the plurality of target time points, respectively, and indicate a representative value of the replication delay in a second period that includes the corresponding target time points and is longer than the first period; outputting an alert regarding replication when the number of times at which the corresponding first value is counted as being greater than the corresponding second value among the plurality of target time points satisfies an abnormality detection condition; A program that causes a computer to execute a process.

Images