Data volume anomaly detection method, device, medium and program product

Abstract

The application provides a data volume anomaly detection method and device, a medium and a program product. The method comprises the following steps: obtaining an initial time sequence, the initial time sequence comprising data volumes of at least one target data table in a database at different time points; calculating a plurality of change rates of the initial time sequence according to a preset time interval; calculating logarithmic values of each change rate according to a preset base; combining the logarithmic values into a first time sequence according to a time sequence corresponding to the change rate; judging whether each data in the first time sequence falls into an abnormal interval; if yes, determining that the first time sequence has an abnormal outlier; and outputting corresponding alarm information according to an abnormal type corresponding to the abnormal outlier. The technical problem that existing data anomaly monitoring cannot distinguish abnormality at continuous time points and cannot be applied to trend, periodic and seasonal time sequences is solved.

Description

Technical Field

[0001] This application relates to the field of financial technology (Fintech), and in particular to a method, apparatus, medium, and program product for detecting abnormal data volume. Background Technology

[0002] With the development of computer technology, more and more technologies are being applied in the financial field, and the traditional financial industry is gradually transforming into financial technology (Fintech).

[0003] Currently, the detection of outliers or data anomalies is very important for many internet systems in the financial sector. Timely detection of outliers and immediate notification of relevant personnel for follow-up can prevent or mitigate unnecessary losses. Therefore, data anomaly monitoring is an important component of various internet system operation and maintenance tools.

[0004] Existing data anomaly monitoring technologies have gone to two extremes. One is that the monitoring logic is too simple. It compares the changes in values ​​of two data points before and after. If the change exceeds a threshold, it is considered to be an anomaly. This method can only identify anomalies at a single point in time and cannot identify anomalies at multiple consecutive points in time. The other is algorithms such as the extreme value student distribution outlier test algorithm or other improved algorithms. These algorithms are complex, have low monitoring efficiency, and cannot be applied to the monitoring of trend, periodic, and seasonal time series.

[0005] Therefore, existing data anomaly monitoring methods have technical problems such as being unable to identify anomalies occurring at several consecutive time points and being unsuitable for trend-based, periodic, and seasonal time series. Summary of the Invention

[0006] This application provides a method, device, medium, and program product for detecting data anomalies, in order to solve the technical problems of existing data anomaly monitoring that cannot identify anomalies occurring at several consecutive time points and cannot be applied to trending, periodic, or seasonal time series.

[0007] Firstly, this application provides a method for detecting abnormal data volume, comprising:

[0008] Obtain the initial time series, which includes the amount of data in at least one target data table in the database at different points in time.

[0009] Calculate multiple rates of change of the initial time series based on preset time intervals;

[0010] Calculate the logarithm of each rate of change based on the preset base;

[0011] The logarithmic values ​​are combined into a first time series based on the time sequence corresponding to the rate of change.

[0012] Determine whether each data point in the first time series falls into an abnormal interval;

[0013] If so, then it is determined that there are outliers in the first time series;

[0014] Based on the anomaly type corresponding to the outlier value, output the corresponding alarm information.

[0015] In one possible design, the preset base corresponds to the predicted distribution pattern followed by the rate of change, and the preset base includes the natural constant e.

[0016] In one possible design, determining whether each data point in the first time series falls within an outlier range includes:

[0017] Calculate multiple moving averages corresponding to the first time series based on the moving average period, and combine the moving averages into the second time series.

[0018] Determine whether each moving average in the second time series falls into an abnormal range.

[0019] In one possible design, the anomaly type includes at least one of the following: single-point impact anomaly, periodic fluctuation anomaly, and trend fluctuation anomaly.

[0020] Correspondingly, the corresponding alarm information is output, including:

[0021] When only one type of exception occurs, the exception type will be recorded in a timed or immediate notification message.

[0022] If two or more anomalies occur, immediately send an alarm message to the operations and maintenance personnel.

[0023] In one possible design, the preset time interval corresponding to a single-point impact anomaly is at least one time unit, and the moving average period is at least twice the preset time interval. The time units include: minutes, hours, days, weeks, months, quarters, and years.

[0024] In one possible design, the preset time interval corresponding to the periodic fluctuation anomaly is greater than or equal to two time units, the moving average period is greater than the preset time interval, and the time units include: minutes, hours, days, weeks, months, quarters, and years.

[0025] In one possible design, the preset time interval corresponding to the abnormal trend fluctuation is at least one time unit, and the moving average period is greater than the preset time interval and is at least 7 to 10 time units, including: minutes, hours, days, weeks, months, quarters, and years.

[0026] Secondly, this application provides a data volume anomaly detection device, comprising:

[0027] The acquisition module is used to acquire the initial time series, which includes the amount of data in at least one target data table in the database at different time points.

[0028] Processing module, used for:

[0029] Calculate multiple rates of change of the initial time series based on preset time intervals;

[0030] Calculate the logarithm of each rate of change based on the preset base;

[0031] The logarithmic values ​​are combined into a first time series based on the time sequence corresponding to the rate of change.

[0032] Determine whether each data point in the first time series falls into an abnormal interval;

[0033] If so, then it is determined that there are outliers in the first time series;

[0034] Based on the anomaly type corresponding to the outlier value, output the corresponding alarm information.

[0035] In one possible design, the preset base corresponds to the predicted distribution pattern followed by the rate of change, and the preset base includes the natural constant e.

[0036] In one possible design, the processing module is used for:

[0037] Calculate multiple moving averages corresponding to the first time series based on the moving average period, and combine the moving averages into the second time series.

[0038] Determine whether each moving average in the second time series falls into an abnormal range.

[0039] In one possible design, the anomaly type includes at least one of the following: single-point impact anomaly, periodic fluctuation anomaly, and trend fluctuation anomaly.

[0040] Correspondingly, the processing module is used for:

[0041] When only one type of exception occurs, the exception type will be recorded in a timed or immediate notification message.

[0042] If two or more anomalies occur, immediately send an alarm message to the operations and maintenance personnel.

[0043] In one possible design, the preset time interval corresponding to a single-point impact anomaly is at least one time unit, and the moving average period is at least twice the preset time interval. The time units include: minutes, hours, days, weeks, months, quarters, and years.

[0044] In one possible design, the preset time interval corresponding to the periodic fluctuation anomaly is greater than or equal to two time units, the moving average period is greater than the preset time interval, and the time units include: minutes, hours, days, weeks, months, quarters, and years.

[0045] In one possible design, the preset time interval corresponding to the abnormal trend fluctuation is at least one time unit, and the moving average period is greater than the preset time interval and is at least 7 to 10 time units, including: minutes, hours, days, weeks, months, quarters, and years.

[0046] Thirdly, this application provides an electronic device, comprising:

[0047] Memory, used to store program instructions;

[0048] A processor for calling and executing program instructions in the memory, performing any of the possible methods provided in the first aspect.

[0049] Fourthly, this application provides a storage medium storing a computer program for executing any of the possible data volume anomaly detection methods provided in the first aspect.

[0050] Fifthly, this application also provides a computer program product, including a computer program that, when executed by a processor, implements any of the possible data volume anomaly detection methods provided in the first aspect.

[0051] This application provides a data volume anomaly detection method, apparatus, medium, and program product. It involves acquiring an initial time series, which includes the data volume of at least one target data table in a database at different time points; calculating multiple rates of change for the initial time series according to preset time intervals; calculating the logarithm of each rate of change based on a preset base; combining the logarithms into a first time series according to the time sequence corresponding to the rates of change; determining whether each data point in the first time series falls within an abnormal range; if so, determining that the first time series contains outliers; and outputting corresponding alarm information based on the anomaly type corresponding to the outliers. This method achieves periodic and trend verification, and not only determines whether the data at the current time point is outlier, but also detects all outliers in the entire time series. It solves the technical problems of existing data anomaly monitoring systems, which cannot identify anomalies occurring at several consecutive time points and are not applicable to trending, periodic, or seasonal time series. Attached Figure Description

[0052] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0053] Figures 1a-1b A schematic diagram illustrating the detection effect of an existing data volume anomaly detection method provided in this application;

[0054] Figure 2 A flowchart illustrating a data volume anomaly detection method provided in an embodiment of this application;

[0055] Figure 3 A schematic diagram of the time series structure provided in the embodiments of this application;

[0056] Figure 4 A statistical curve of a periodically changing time series provided for this application;

[0057] Figure 5 A time series curve of trend change provided for an embodiment of this application;

[0058] Figure 6 A flowchart illustrating another data volume anomaly detection method provided for the implementation of this application;

[0059] Figure 7 A graph of an initial time series provided in an embodiment of this application;

[0060] Figure 8 Provided for the embodiments of this application Figure 7 The curve of the first time series corresponding to the initial time series shown;

[0061] Figure 9 A graph of the second time series provided in the embodiments of this application;

[0062] Figure 10 This is a schematic diagram of the structure of a data volume anomaly detection device provided in an embodiment of this application;

[0063] Figure 11 This is a schematic diagram of the structure of an electronic device provided in this application.

[0064] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0065] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort, including but not limited to combinations of multiple embodiments, are within the scope of protection of this application.

[0066] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a particular order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented, for example, in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0067] The following is an explanation of the technical terms used in this application:

[0068] Time series: Observations of a variable in chronological order, such as the dynamic values ​​of memory usage at times t1, t2, ..., tn, where t1...tn... <t2<…<tn。

[0069] Database: A repository for organizing, storing, and managing data according to a specific data structure.

[0070] A table is an object used to store data in a database. It is a structured collection of data and forms the foundation of the entire database system.

[0071] Point-in-time value: The numerical value corresponding to a specific point in time t in a time series.

[0072] Outliers: Points in a time series that exhibit unusual fluctuations.

[0073] N(μ,σ 2 ): This indicates that the expected value is μ and the variance is σ. 2 It follows a normal distribution.

[0074] Log-normal distribution: If the natural logarithm LN(X) of X follows a normal distribution, then X follows a log-normal distribution.

[0075] LN(X) or ln(x): represents the natural logarithm function.

[0076] Extreme Studentized Deviate (ESD) Test: An algorithm proposed by Twitter for detecting outliers in extreme student distributions.

[0077] Mean μ: In probability theory, it represents the average value of a sample or the population.

[0078] Standard deviation σ: In probability theory, it represents the degree of deviation of an individual from the overall mean.

[0079] Interval: A preset time interval used to identify the interval between two points in time, such as an interval of 7 days, interval=7.

[0080] Period: The moving average period is used to identify the interval from which the moving average is taken. For example, if a five-day moving average is taken, then period = 5.

[0081] P_Threshold: Preset probability threshold, used to identify the marginal probability.

[0082] Alpha_value: Used to identify the probability quantile corresponding to P_Threshold.

[0083] Detecting outliers or abnormal fluctuations in data is particularly important for many IT (Internet Technology) systems. Timely detection of outliers and prompt notification to relevant personnel for follow-up can significantly mitigate unnecessary losses. Therefore, data anomaly monitoring is a fundamental and crucial component of various IT system maintenance tools. Furthermore, the vast majority of data anomaly monitoring scenarios involve examining time series data, such as monitoring the time-series usage of server CPUs (central processing units) or memory.

[0084] Many IT systems need to monitor fluctuations in the amount of data in some tables in the database, usually by time series, which can be statistically analyzed by minute, hour, or day. If there are abnormal fluctuations, an alarm will be issued and the operation and maintenance or relevant personnel will follow up and handle it.

[0085] The existing data anomaly detection solutions mainly fall into the following two categories:

[0086] Option 1: A simple and direct way to check is to compare the value at the current time point with the value at the previous time point to obtain the change in value between the two time points. If the change in value exceeds a preset threshold, it is considered an abnormal data point.

[0087] Option 2: Extreme Studentize Deviate (ESD) Test: The algorithm proposed by Titter assumes that the data values ​​at each point in the time series generally follow a normal distribution, and then the t-distribution can be used to detect one outlier.

[0088] As for Option 1, its detection method is too simple and crude. To be precise, it can only detect anomalies at a certain point in time (when the data at the previous point in time was normal).

[0089] Figures 1a-1b This diagram illustrates the detection effect of an existing data volume anomaly detection method provided in this application. Figure 1a As shown, the data at the previous time point is D1, and the data at the next time point is D2. Scheme 1 above can detect this anomaly. However, as... Figure 1b As shown, if the data D1 at the previous time point is already an outlier, the data D2 at the next time point is also an outlier, and the data D3 at the next time point returns to normal, but because Scheme 1 compares the changes of two adjacent data points, data D1 can be identified as an outlier, but data D2 cannot be identified as an outlier, and data D3, which should be a normal value, is misjudged as an outlier.

[0090] Scheme 2 only detects one outlier. Although there are some improved algorithms in the existing technology that can detect multiple outliers, these methods are very complex, which reduces the detection efficiency. Moreover, they still cannot avoid the defects of Scheme 1 because they assume that the data values ​​in the time series follow a normal distribution, which makes it impossible to detect anomalies in time series with trends, periods, or seasonality.

[0091] To solve the above problems, the inventive concept of this application is as follows:

[0092] After analyzing the solutions of the two existing technologies mentioned above, the inventors of this application found that Solution 1 can only solve a single scenario and has a high probability of error, while Solution 2's basic assumptions do not conform to reality and ignore the trends and periodicity of data, requiring the introduction of other methods to address these issues, thus increasing monitoring costs. Furthermore, if supervised machine learning algorithms are used, human labeling is required, and external data, such as subjective judgments on whether it is a weekend or a holiday, must be introduced. The model parameters need to be continuously iterated, resulting in extremely high computational complexity.

[0093] This application uses the rate of change of table data at two points in time as the basic calculation unit, that is, the rate of change across time periods. Combining the differences between the basic principles of moving average, normal distribution and skewed distribution, it cleverly solves the problems of trend and periodicity, and the algorithm has low time complexity.

[0094] This application proposes a set of operation and maintenance tools solutions that can perform real-time outlier or abnormal value detection on various time-series data. It mainly monitors the time-series fluctuation changes of table data volume in the production database and includes three parts: a real-time data collector, an outlier fluctuation detector, and a real-time alarm platform.

[0095] The real-time data acquisition device is used to collect data from one or more tables in the database in real time and combine them into a time series. The outlier fluctuation detector is used to detect whether there are abnormal outliers in the time series. If so, it issues an alert to the user through a real-time alarm platform. The data volume anomaly detection method of this application has the following characteristics:

[0096] (1) Periodicity test can detect abnormal fluctuations in time series with periodicity.

[0097] (2) Trend verification can detect abnormal fluctuations in time series with trends.

[0098] (3) It can not only determine whether there is an outlier at the current time point, but also detect all outliers in the entire time series.

[0099] (4) Simple and efficient, the algorithm is simple, and the statistical time complexity is O(n^2). 2 The time complexity for judging a single time point is O(1), and the time complexity for judging all time points is O(n).

[0100] (5) It can monitor and report changes in the amount of data in a specified table in the database in real time.

[0101] The data volume anomaly detection method provided in this application is described in detail below:

[0102] Figure 2 This is a flowchart illustrating a data volume anomaly detection method provided in an embodiment of this application. Figure 2 As shown, this data volume anomaly detection method is used to monitor the time-series changes in the number of data tables or the amount of data in a database. The specific steps include:

[0103] S201. Obtain the initial time series.

[0104] In this step, the initial time series includes the amount of data in at least one target data table in the database at different points in time.

[0105] In this embodiment, the initial time series is obtained through a real-time data acquisition device. At least one data table in the database can be pre-specified as the target data table, and the data volume of the target data table can be acquired periodically, or the data volume of the target data table can be acquired in real time in other non-timed manner, and then these data volumes are combined into the initial time series according to the time sequence.

[0106] Specifically, the process begins by establishing a connection to the database. For example, using Java, a connection pool can be created via JDBC (Java Database Connectivity), which can adapt to various database types. Then, in response to time-series data collection commands, the data volume of the target table is read from the connection pool on demand or periodically. For instance, the data collection SQL statement `select count(*) from table` is executed every hour. The amount of data read from the target table, the name of the target table, and the current collection time are then recorded in a predefined result table. An example structure of this result table is shown in Table 1.

[0107] Table name numerical values time Tbl1 12313 2021 / 01 / 01 21:00:00 Tbl1 13231 2021 / 01 / 01 22:00:00 … … …

[0108] Table 1

[0109] Then, read the initial time series output from the real-time data acquisition unit: X(X1, X2, X3, ..., X...). n-2 X n-1 X n ).

[0110] Figure 3 This is a schematic diagram of the time series structure provided for an embodiment of this application. For example... Figure 3 As shown, any value X in the time series X i It is a key-value pair, such as: (data value i, time i). Figure 3 The arrows in the text indicate the chronological order.

[0111] S202. Calculate multiple rates of change of the initial time series according to the preset time interval.

[0112] In this step, the preset time interval can be entered in advance by the operations and maintenance personnel, such as: eginterval=1 (day). That is, the rate of change of a point in time relative to the previous interval points, such as the rate of change relative to the previous day or 7 days ago.

[0113] It should be noted that the preset time interval is an integer multiple of the preset time unit, which can be set to: minutes, hours, days, weeks, months, quarters, years, etc.

[0114] It should also be noted that the rate of change used in this application is the ratio of the data values ​​at two different time points, i.e., X. n / X n-interval For example, when interval = 1, the rate of change is x2 / x1, not the change in existing technology, which is the difference between the data values ​​at two different time points, i.e., X. n -Xn-interval .

[0115] This application allows adjustment of the periodicity to be extracted from the time series by adjusting the size of the preset time interval and / or the unit of the corresponding time unit, or in other words, adjusting the preset time interval based on the periodicity of the time series. For example, actual data may exhibit monthly, weekly, or even half-day periodicity, meaning the data changes regularly according to these cycles. Therefore, the size of the preset time interval and / or the unit of the corresponding time unit can be determined according to the actual periodicity.

[0116] Figure 4 This application provides a statistical curve of a periodically changing time series. For example... Figure 4 As shown, curve 1 represents the change in the amount of data in the target table of the database over time, i.e., the initial time series. The unit of data volume is millions, and the unit of time is days. Figure 4 As shown above, curve 1 exhibits periodic fluctuations over a weekly period, i.e., the preset time interval = 7. Curve 2 represents the rate of change calculated from the initial time series over the preset time interval as a function of time. It can be seen that curve 2 smooths out the periodic fluctuations. Therefore, using the rate of change solves the technical problem in existing technologies where outliers cannot be accurately identified in periodically changing time series. This breaks away from the conventional approach of relying solely on the magnitude of differences, and more accurately identifies abnormal fluctuations in data volume.

[0117] Figure 5 This is a time series graph illustrating a trend change, provided as an embodiment of this application. For example... Figure 5 As shown, curve 3 represents the change in the amount of data in the target table of the database over time, i.e., the initial time series. The unit of data volume is millions, and the unit of time is days. Figure 5 As can be seen above, curve 3 exhibits a trend of fluctuating growth. For this type of time series, both existing solutions 1 and 2 will misjudge the data. Furthermore, the assumption in solution 2 that the entire time series follows a normal distribution is clearly inapplicable. Curve 4 is the rate of change curve of the initial time series proposed in this application, and it can be seen that curve 4 eliminates the interference of trend fluctuations.

[0118] S203. Calculate the logarithmic value of each rate of change based on the preset base, and combine the logarithmic values ​​into a first time series according to the time sequence corresponding to the rate of change.

[0119] In this step, unlike the existing Extreme Studentize Deviate (ESD) Test which assumes the time series follows a normal distribution, this application uses the rate of change to eliminate the interference of periodic or trend fluctuations. At this point, the object we are detecting changes from the initial time series to the sequence formed by the rate of change. The inventors of this application have discovered that the rate of change does not follow a normal distribution in various practical application scenarios; instead, it is closer to a skewed distribution. That is, the rate of change is generally concentrated in a certain area, or skewed towards one side of the mean, unlike the normal distribution which is perfectly symmetrically distributed on both sides of the mean, or an unbiased distribution. Therefore, taking the logarithm of the rate of change can eliminate the skewed effect of the rate of change, or in other words, eliminate the non-normal distribution characteristic caused by the heteroscedasticity of the rate of change, or its non-linear regression characteristics.

[0120] The first time series formed in this way solves the technical problem that existing technologies do not consider the skewed characteristics of the rate of change, which makes the identification of abnormal outliers inaccurate.

[0121] S204. Determine whether each data point in the first time series falls into an abnormal interval.

[0122] In this step, the abnormal interval corresponds to the probability distribution followed by the first time series. For example, if the first time series follows a log-normal distribution, the probability quantile can be determined by setting the corresponding probability threshold.

[0123] In this embodiment, if yes, then S205 is executed; otherwise, S201 is returned.

[0124] Specifically, the pre-trained skewed distribution model analyzes and divides the skewed distribution of the rate of change of the data volume into regions. This can be understood as the probability that the rate of change appears in a certain region of the rate of change distribution map is greater than or equal to a preset probability threshold.

[0125] In this embodiment, the preset probability threshold is divided into an upper threshold and a lower threshold. The upper threshold and the lower threshold each correspond to a certain value on the coordinate axis of the rate of change distribution graph, i.e., a probability quantile. Therefore, this step specifically includes:

[0126] Determine the probability quantiles in the skewed distribution model based on a preset probability threshold;

[0127] Determine the abnormal interval based on probability quantiles;

[0128] Determine whether each data point in the first time series falls into an abnormal interval;

[0129] If so, then the data is identified as an outlier.

[0130] S205. Output the corresponding alarm information based on the anomaly type corresponding to the outlier value.

[0131] In this step, the anomaly types include at least one of the following: single-point impact anomaly, periodic fluctuation anomaly, and trend fluctuation anomaly. Correspondingly, the corresponding alarm information is output, including:

[0132] When only one type of exception occurs, the exception type will be recorded in a timed or immediate notification message.

[0133] If two or more anomalies occur, immediately send an alarm message to the operations and maintenance personnel.

[0134] In this embodiment, the anomaly level is divided into four levels:

[0135] No abnormalities: N / A, meaning no action is required and no alarms are issued.

[0136] Notification level: INFO, only used for recording notification information; whether to send it to alarm routes is set by the operations and maintenance personnel.

[0137] Warning level: WARNING. This indicates a high probability of data anomalies, requiring attention from engine maintenance personnel and necessitating an alert.

[0138] Error level: ERROR, which indicates that the data is abnormal and an alarm must be sent.

[0139] The specific allocation method for sending alarms for the above four levels is shown in Table 2:

[0140] Numerical shock Periodic Trend Alarm Level normal normal normal N / A abnormal normal normal INFO normal abnormal normal INFO normal normal abnormal INFO abnormal abnormal normal WARNING normal abnormal abnormal WARNING abnormal normal abnormal WARNING abnormal abnormal abnormal ERROR

[0141] Table 2

[0142] This application provides a data volume anomaly detection method. The method involves acquiring an initial time series, which includes the data volume of at least one target data table in a database at different time points; calculating multiple rates of change for the initial time series based on preset time intervals; calculating the logarithm of each rate of change based on a preset base; combining the logarithms into a first time series according to the time sequence corresponding to the rates of change; determining whether each data point in the first time series falls into an abnormal range; if so, determining that the first time series contains outliers; and outputting corresponding alarm information based on the anomaly type corresponding to the outliers. This method achieves periodic and trend verification, and not only determines whether the data at the current time point is outlier, but also detects all outliers in the entire time series. It solves the technical problem that existing data anomaly monitoring methods cannot identify anomalies occurring at several consecutive time points and are not applicable to trend-based, periodic, or seasonal time series.

[0143] Figure 6A flowchart illustrating another data volume anomaly detection method provided for the implementation of this application. (For example...) Figure 6 As shown, the specific steps of this data volume anomaly detection method include:

[0144] S601. Obtain the initial time series.

[0145] In this step, the initial time series includes the amount of data in at least one target data table in the database at different points in time.

[0146] In this embodiment, for ease of understanding, it is assumed that the amount of data (in millions) of the target data table collected by the initial time series X at each sampling time is as follows:

[0147] X=(122.2,123.3,123.1,124.5,123.7,122.9,122.1,121.7,122.6,123 .1, 122.5, 123.4, 123, 124.6, 123.8, 122.6, 122.3, 121.8, 122.7, 123.2, 122.2, 123.3, 123.1, 124.5, 123.7, 122.9, 122.1, 121.7, 122.6, 123.1, 122.2, 123.3, 123.1, 124.5, 123.7, 122.9, 122.1, 121.7, 122.6, 129.1).

[0148] Figure 7 A graph of an initial time series provided for an embodiment of this application. For example... Figure 7 As shown, the initial time series exhibits both periodicity and a trend within a period.

[0149] S602. Calculate multiple rates of change of the initial time series based on preset time intervals.

[0150] In this step, the rates of change of the initial time series are obtained according to formula (1), which is shown below:

[0151]

[0152] Where interval is the preset time interval.

[0153] S603. Calculate the logarithm of each rate of change based on the preset base.

[0154] In this step, the base is assumed to be greater than zero, for example, the natural constant e, or 10. The purpose of the logarithmic value is to eliminate heteroscedasticity between different rates of change. This is because the inventors of this application have found that the sequences corresponding to the rates of change do not conform to the symmetry characteristics of the normal distribution in practical applications, but are concentrated in a certain region, exhibiting asymmetric bias. The logarithmic operation is intended to utilize this bias characteristic, laying the foundation for subsequently using the log-normal distribution to identify outliers. The log-normal distribution is a type of skewed distribution.

[0155] S604. Determine the first time series based on the time sequence corresponding to each logarithmic value.

[0156] For steps S603 and S604, specifically, we can assume that the preset base is a (a>0), then any data X′ in the first time series X' i Formula (2) can be used to calculate:

[0157] X′ i =log a B i (2)

[0158] Among them, B i The change rate is denoted as .

[0159] It should be noted that the preset base value can be different depending on the skewed distribution model, or it can be set to a different value depending on the object being detected.

[0160] For ease of understanding, let's assume that the natural constant e is taken as the preset base, then the first time series X'(X' k ,X' k+1 ,…,X' n-2 ,X' n-1 ,X' n ),in,

[0161] It is worth noting that since the rate of change cannot be calculated for the first intervals of the initial time series X, k here is counted starting from the interval of the initial time series X.

[0162] Figure 8 Provided for the embodiments of this application Figure 7 The graph shows the curve of the first time series corresponding to the initial time series. (Example:) Figure 8 As shown, the logarithmic rate of change effectively filters out the periodicity and trend fluctuations of the initial time series. The logarithm is used because the rate of change is more consistent with a skewed distribution.

[0163] S605. Determine the probability quantiles in the skewed distribution model based on the preset probability thresholds, and determine the abnormal intervals based on the probability quantiles.

[0164] In this embodiment, the skewed distribution model includes: the log-normal distribution model.

[0165] S606. Using the moving average algorithm, determine the second time series based on the moving average period and the first time series.

[0166] The purpose of this step is to smooth out the normal fluctuations in the first time series, so that outliers can be more clearly identified.

[0167] Specifically, a new logarithmic moving average sequence X”, i.e., the second time series, can be output through the MoveAverageTimeSeriesConverter. The moving average sequence X” is calculated according to the moving average period, where:

[0168] X” i It is X' i The average of the previous consecutive periods. X” i =∑ i i-period+1 X' j Generate a new sequence X ((X) m ,X” m+1 ,………X” n-2 ,X” n-1 ,X” n ).

[0169] It is worth noting that since the moving average cannot be calculated for the first period of X', m here starts counting from the period of the first time series X'.

[0170] S607. Determine whether each mean in the second time series falls into the abnormal interval.

[0171] In this step, if yes, the data is determined to be an outlier and S608 is executed; otherwise, the next value is evaluated. If no outlier is found after the second time series is completed, the process returns to S601.

[0172] Figure 9 A graph of the second time series provided in the embodiments of this application. Figure 9 As shown, the logarithmic rate of change is mostly within ±0.005, and anything outside this range is considered an outlier. Therefore, the last data point in the graph is an outlier.

[0173] S608. Output the corresponding alarm information according to the anomaly type corresponding to the outlier value.

[0174] In this step, the anomaly types include at least one of the following: single-point impact anomaly, periodic fluctuation anomaly, and trend fluctuation anomaly.

[0175] In this embodiment, (1) the preset time interval corresponding to the single-point impact anomaly is at least one time unit, and the moving average period is at least twice the preset time interval;

[0176] (2) The preset time interval corresponding to the periodic fluctuation anomaly is greater than or equal to two time units, and the moving average period is greater than the preset time interval.

[0177] (3) The preset time interval corresponding to the abnormal trend fluctuation is at least one time unit, the moving average period is greater than the preset time interval, and is at least 7 to 10 time units.

[0178] It should be noted that time units include: minutes, hours, days, weeks, months, quarters, and years.

[0179] Specifically, in order to more accurately identify data volume anomalies in the target data table, it is necessary to comprehensively consider the periodicity and trend of the data. Therefore, it is necessary to perform scenario detection on time series data according to periodicity, trend, etc., so outlier fluctuations are divided into three scenarios:

[0180] Single-point shock anomaly: This means that the data at the current time point fluctuates sharply compared to the data at the previous time point.

[0181] Abnormal periodic fluctuations: Abnormal fluctuations that occur outside of a periodicity within a certain time period.

[0182] Abnormal trend fluctuations: In a trending (growing or falling) time series, there are abnormal fluctuation points that clearly deviate from the trend.

[0183] In this embodiment, the calculation instruction module in the real-time alarm platform sends the calculation parameter instructions corresponding to the calculation parameters of each of the above scenarios to the fluctuation detector in JSON format. The fluctuation detector then identifies different anomaly types.

[0184] For example, in a single-point anomaly impact scenario (Single Detection): the change in the time point value in this scenario can be compared with the previous one, that is, the calculation instruction is: interval = 1. eg: {"P_Threshold": "0.99", "period": "5", "interval": "1"}.

[0185] Periodic fluctuation anomaly scenario (Frequency Detection): For abnormal fluctuation points in periodic data, the interval should reflect the periodic frequency. For example, if the data has a weekly cycle, then interval = 7 days, and the shift interval should be greater than the interval. The calculation command is: e.g., {"P_Threshold":"0.99","period":"14","interval":"7"}.

[0186] Trend-based fluctuation anomaly scenario (Trend Detect): At the point of abnormal fluctuation with a trend (upward or downward trend), the trend parameter is the moving average period. For example, if the trend has been upward for the past month, the calculation instruction is: moving average period = 30 days, e.g.: {"P_Threshold": "0.99", "period": "50", "interval": "1"}.

[0187] Correspondingly, the corresponding alarm information is output, including:

[0188] When only one type of exception occurs, the exception type will be recorded in a timed or immediate notification message.

[0189] If two or more anomalies occur, immediately send an alarm message to the operations and maintenance personnel.

[0190] This application provides a data volume anomaly detection method. The method involves acquiring an initial time series, which includes the data volume of at least one target data table in a database at different time points; calculating multiple rates of change for the initial time series based on preset time intervals; calculating the logarithm of each rate of change based on a preset base; combining the logarithms into a first time series according to the time sequence corresponding to the rates of change; determining whether each data point in the first time series falls into an abnormal range; if so, determining that the first time series contains outliers; and outputting corresponding alarm information based on the anomaly type corresponding to the outliers. This method achieves periodic and trend verification, and not only determines whether the data at the current time point is outlier, but also detects all outliers in the entire time series. It solves the technical problem that existing data anomaly monitoring methods cannot identify anomalies occurring at several consecutive time points and are not applicable to trend-based, periodic, or seasonal time series.

[0191] Figure 10 This is a schematic diagram of a data volume anomaly detection device provided in an embodiment of this application. The data volume anomaly detection device 1000 can be implemented through software, hardware, or a combination of both.

[0192] like Figure 10 As shown, the data volume anomaly detection device 1000 includes:

[0193] The acquisition module 1001 is used to acquire an initial time series, which includes the amount of data in at least one target data table in the database at different time points.

[0194] Processing module 1002 is used for:

[0195] Calculate multiple rates of change of the initial time series based on preset time intervals;

[0196] Calculate the logarithm of each rate of change based on the preset base;

[0197] The logarithmic values ​​are combined into a first time series based on the time sequence corresponding to the rate of change.

[0198] Determine whether each data point in the first time series falls into an abnormal interval;

[0199] If so, then it is determined that there are outliers in the first time series;

[0200] Based on the anomaly type corresponding to the outlier value, output the corresponding alarm information.

[0201] In one possible design, the preset base corresponds to the predicted distribution pattern followed by the rate of change, and the preset base includes the natural constant e.

[0202] In one possible design, the processing module 1002 is used for:

[0203] Calculate multiple moving averages corresponding to the first time series based on the moving average period, and combine the moving averages into the second time series.

[0204] Determine whether each moving average in the second time series falls into an abnormal range.

[0205] In one possible design, the anomaly type includes at least one of the following: single-point impact anomaly, periodic fluctuation anomaly, and trend fluctuation anomaly.

[0206] Correspondingly, processing module 1002 is used for:

[0207] When only one type of exception occurs, the exception type will be recorded in a timed or immediate notification message.

[0208] If two or more anomalies occur, immediately send an alarm message to the operations and maintenance personnel.

[0209] In one possible design, the preset time interval corresponding to a single-point impact anomaly is at least one time unit, and the moving average period is at least twice the preset time interval. The time units include: minutes, hours, days, weeks, months, quarters, and years.

[0210] In one possible design, the preset time interval corresponding to the periodic fluctuation anomaly is greater than or equal to two time units, the moving average period is greater than the preset time interval, and the time units include: minutes, hours, days, weeks, months, quarters, and years.

[0211] In one possible design, the preset time interval corresponding to the abnormal trend fluctuation is at least one time unit, and the moving average period is greater than the preset time interval and is at least 7 to 10 time units, including: minutes, hours, days, weeks, months, quarters, and years.

[0212] It is worth noting that, Figure 10 The apparatus provided in the illustrated embodiments can execute the methods provided in any of the above method embodiments. Their specific implementation principles, technical features, explanations of technical terms, and technical effects are similar and will not be repeated here.

[0213] Figure 11 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Figure 11 As shown, the electronic device 1100 may include at least one processor 1101 and a memory 1102. Figure 11 The example shown is an electronic device using a processor.

[0214] The memory 1102 is used to store programs. Specifically, the program may include program code, which includes computer operation instructions.

[0215] The memory 1102 may include high-speed RAM memory, and may also include non-volatile memory, such as at least one disk storage.

[0216] The processor 1101 is used to execute computer execution instructions stored in the memory 1102 to implement the methods described in the above embodiments.

[0217] The processor 1101 may be a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of this application.

[0218] Optionally, the memory 1102 can be either standalone or integrated with the processor 1101. When the memory 1102 is a device independent of the processor 1101, the electronic device 1100 may further include:

[0219] Bus 1103 is used to connect the processor 1101 and the memory 1102. The bus can be an industry standard architecture (ISA) bus, a peripheral component interconnect (PCI) bus, or an extended industry standard architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc., but this does not mean there is only one bus or one type of bus.

[0220] Optionally, in a specific implementation, if the memory 1102 and the processor 1101 are integrated on a single chip, the memory 1102 and the processor 1101 can communicate through an internal interface.

[0221] This application also provides a computer-readable storage medium, which may include various media capable of storing program code, such as a USB flash drive, a portable hard drive, a read-only memory (ROM), a random access memory (RAM), a disk, or an optical disk. Specifically, the computer-readable storage medium stores program instructions, which are used in the methods described in the above-mentioned method embodiments.

[0222] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the methods described in the above-described method embodiments.

[0223] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this application are indicated by the claims.

[0224] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.

Claims

1. A method for detecting abnormal data volume, characterized in that, include: Obtain an initial time series, which includes: the amount of data in at least one target data table in the database at different points in time; Multiple rates of change of the initial time series are calculated based on preset time intervals, and the formula for the rate of change is: Where interval is the preset time interval; The rate of change; Calculate the logarithm of each rate of change based on a preset base; The logarithmic values ​​are combined into a first time series according to the time sequence corresponding to the rate of change; Determining whether each data point in the first time series falls into an abnormal interval includes: calculating multiple moving averages corresponding to the first time series based on the moving average period, and combining each moving average into a second time series; Determine whether each of the moving averages in the second time series falls into the abnormal interval; If so, then it is determined that there are outliers in the first time series; Based on the anomaly type corresponding to the outlier value, output the corresponding alarm information.

2. The data volume anomaly detection method according to claim 1, characterized in that, The preset base corresponds to the predicted distribution pattern followed by the rate of change, and the preset base includes the natural constant e.

3. The data volume anomaly detection method according to claim 2, characterized in that, The anomaly types include at least one of the following: single-point impact anomaly, periodic fluctuation anomaly, and trend fluctuation anomaly. Correspondingly, the output alarm information includes: When only one type of exception occurs, the exception type is recorded in a timed or immediate notification message; If two or more anomalies occur, immediately send the aforementioned alarm information to the operations and maintenance personnel.

4. The data volume anomaly detection method according to claim 3, characterized in that, The preset time interval corresponding to the single-point impact anomaly is at least one time unit, and the moving average period is at least twice the preset time interval. The time unit includes: minute, hour, day, week, month, quarter, and year.

5. The data volume anomaly detection method according to claim 3, characterized in that, The preset time interval corresponding to the periodic fluctuation anomaly is greater than or equal to two time units, the moving average period is greater than the preset time interval, and the time units include: minutes, hours, days, weeks, months, quarters, and years.

6. The data volume anomaly detection method according to claim 3, characterized in that, The preset time interval corresponding to the trend fluctuation anomaly is at least one time unit, and the moving average period is greater than the preset time interval and is at least 7 to 10 time units. The time units include: minutes, hours, days, weeks, months, quarters, and years.

7. A data volume anomaly detection device, characterized in that, include: The acquisition module is used to acquire an initial time series, which includes the amount of data in at least one target data table in the database at different time points. Processing module, used for: Multiple rates of change of the initial time series are calculated based on preset time intervals, and the formula for the rate of change is: Where interval is the preset time interval; The rate of change; Calculate the logarithm of each rate of change based on a preset base; The logarithmic values ​​are combined into a first time series according to the time sequence corresponding to the rate of change; Determining whether each data point in the first time series falls into an abnormal interval includes: calculating multiple moving averages corresponding to the first time series based on the moving average period, and combining each moving average into a second time series; Determine whether each of the moving averages in the second time series falls into the abnormal interval; If so, then it is determined that there are outliers in the first time series; Based on the anomaly type corresponding to the outlier value, output the corresponding alarm information.

8. An electronic device, characterized in that, include: processor; as well as, Memory for storing the computer program of the processor; The processor is configured to execute the data volume anomaly detection method according to any one of claims 1 to 6 by executing the computer program.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the data volume anomaly detection method according to any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the data volume anomaly detection method according to any one of claims 1 to 6.

Images