Method and device for identifying outliers of sparse time series data

Abstract

The present application relates to a method and device for identifying outliers of sparse time series data. The method comprises: slicing sparse time series data of a monitoring object into segments in time series; judging whether the time series segment data conforms to normal distribution; generating different alarm threshold values for time series segment data judged to conform to normal distribution and time series segment data judged not to conform to normal distribution using different alarm threshold generation algorithms; calculating peak periods of the time series segment data, correcting the calculated peak periods according to the time zone in which the monitoring object is located, and obtaining peak period data and non-peak period data; generating alarm rules based on the different alarm threshold values, adjusting the alarm rules by setting different monitoring indicators for the peak period data and the non-peak period data to generate adjusted alarm rules; and triggering an alarm when actual data of the monitoring object conforms to the adjusted alarm rules.

Description

Technical Field

[0001] This invention relates to computing technology, and more specifically to a method and apparatus for outlier identification of sparse time-series data. Background Technology

[0002] Outlier identification is a crucial step in data monitoring and anomaly detection. It has wide applications in data analysis and mining scenarios such as intelligent operation and maintenance and risk management. It can help relevant personnel to promptly identify and handle anomalies, ensuring system stability and security.

[0003] Currently, outlier identification methods can be divided into three categories: The first category is based on manual rules, setting fixed thresholds based on expert experience. These methods are simple to set but highly subjective, making fine-tuning and optimization difficult. The second category is based on machine learning methods, such as Independent Forest, K-Means, and DBSCAN algorithms. These methods are suitable for high-dimensional data but require significant computational resources. The third category is based on statistical methods, such as year-on-year and month-on-month comparison methods. These algorithms have low computational complexity, require no anomaly labeling or model training, and need low-dimensional data with high interpretability, but are not suitable for high-dimensional data. Each of these three categories has its advantages and disadvantages and is suitable for outlier identification in different scenarios.

[0004] For time series data, in scenarios with high data density, the time series signals are relatively stable, with small fluctuations and obvious regularities, making outlier identification algorithms perform well. However, in sparse data scenarios, the time series signals fluctuate more significantly, and the data regularity is weaker, making outlier identification more difficult. Existing methods mostly focus on outlier identification in data-intensive scenarios, while paying less attention to outlier identification algorithms for sparse data scenarios. However, sparse time series signals are abundant in real-world scenarios, and some sparse data, although occurring infrequently, are highly important. Therefore, we aim to propose methods that can effectively identify outliers in sparse data. Summary of the Invention

[0005] In view of the above problems, the present invention aims to propose an outlier identification method and device for sparse time series data with high accuracy in outlier identification.

[0006] The outlier identification method for sparse time-series data of the present invention is characterized by comprising:

[0007] The time series slicing step involves slicing the sparse time series data of the monitored object into segments to obtain time series segment data.

[0008] The data distribution judgment step determines whether the time series segment data conforms to a normal distribution.

[0009] The alarm value generation step involves using different alarm threshold generation algorithms to generate different alarm thresholds for time series data segments that are judged to conform to a normal distribution and those that are judged not to conform to a normal distribution.

[0010] The peak period calculation step involves calculating the peak period of the time series data segment and correcting the calculated peak period according to the time zone of the monitored object to obtain peak period data and non-peak period data.

[0011] The alarm rule generation step involves generating alarm rules based on the different alarm thresholds, and adjusting the alarm rules by setting different monitoring indicators for peak and off-peak data to generate adjusted alarm rules; and

[0012] The alarm maintenance procedure is to trigger an alarm when the actual data of the monitored object matches the adjusted alarm rules.

[0013] Optionally, in the time series slicing step, the length of the segment is determined based on the sparsity of the sparse time series data.

[0014] Optionally, in the data distribution judgment step, the KS nonparametric test is used to determine whether the data distribution of the time series segment data conforms to a normal distribution.

[0015] Optionally, in the alarm value generation step, for data that is determined to conform to a normal distribution, the 3σ model is used to calculate the alarm threshold line; for data distributions that are determined not to conform to a normal distribution, the quantile discrimination method is used to calculate the alarm threshold line.

[0016] Optionally, in the alarm rule generation step, alarm rules are generated by fitting the outer contour of the alarm threshold line formed by the alarm threshold.

[0017] Optionally, the peak period calculation step includes:

[0018] Set specified thresholds for peak periods;

[0019] For the data distribution of the time series segment data, the length of the peak period and the starting point of the peak period are calculated by determining whether the time series segment data exceeds a predetermined threshold of the peak period; and

[0020] The calculated peak period and the portion of the time zone where the monitored object is located are selected as the corrected peak period, and the time series data segment is divided into peak period data and non-peak period data.

[0021] Optionally, the outlier identification device for sparse time-series data is characterized by comprising:

[0022] The real-time signal acquisition module is used to acquire real-time timing signals.

[0023] The data processing module is used to preprocess real-time time-series signals to obtain the data.

[0024] The data calculation module is used to determine whether the data distribution of the time series segment data conforms to a normal distribution. For time series segment data determined to conform to a normal distribution and those determined not to conform, different alarm threshold generation algorithms are used to generate different alarm thresholds. The module calculates the peak period of the time series segment data and corrects the calculated peak period according to the time zone of the monitored object to obtain peak period data and non-peak period data. Alarm rules are generated based on the different alarm thresholds, and the alarm rules are adjusted by setting different monitoring indicators for peak and non-peak data to generate adjusted alarm rules.

[0025] The operation and maintenance processing module is used to trigger alarms when the actual data of the monitored object meets the adjusted alarm rules.

[0026] Optionally, the data processing module includes:

[0027] The data format standardization submodule preprocesses timing signals;

[0028] The data content definition submodule allows you to set different content filtering rules for different monitored objects to remove data that does not need to be monitored.

[0029] The data signal segmentation module slices the time-series signal to obtain time-series data segments; and

[0030] The data signal retrieval submodule retrieves real-time or historical time-series data.

[0031] Optionally, the data calculation module includes:

[0032] The data distribution judgment submodule uses the KS nonparametric test to determine whether the time series segment data follows a normal distribution, and uses different outlier identification algorithms for data with different distributions;

[0033] The outlier identification algorithm application submodule should use the alarm threshold line generated by the 3σ model if the outlier conforms to a normal distribution; otherwise, it should use the alarm threshold line generated by the 99th quantile model if the outlier does not conform to a normal distribution.

[0034] The data peak period calculation submodule determines whether the time series data segment is in a peak period and corrects for the high period based on the time zone to obtain the final peak period; and

[0035] The alarm rule generation submodule generates alarm rules based on alarm threshold lines. Different monitoring indicators are used for peak and off-peak periods to adjust the generated alarm rules, resulting in adjusted alarm rules.

[0036] Optionally, in the operation and maintenance processing module, when an alarm is triggered, the alarm information is pushed after further classifying the importance of the alarm based on the alarm object type, abnormal indicator type, and alarm frequency.

[0037] A computer-readable medium of one aspect of the present invention stores a computer program thereon, characterized in that the computer program, when executed by a processor, implements the outlier identification method for sparse time-series data.

[0038] A computer device according to one aspect of the present invention includes a storage module, a processor, and a computer program stored on the storage module and executable on the processor, characterized in that the processor executes the computer program to implement the outlier identification method for sparse time-series data. Attached Figure Description

[0039] Figure 1 This is a summary flowchart of the outlier identification method for sparse time-series data according to the present invention.

[0040] Figure 2 This diagram illustrates the fitting process for the alarm threshold line.

[0041] Figure 3 A diagram showing the calculation results for ordinary monitored objects during peak periods.

[0042] Figure 4 A schematic diagram showing the calculation results of a large time zone monitoring object during peak periods.

[0043] Figure 5 This is a schematic diagram of the outlier identification device for sparse time-series data according to the present invention.

[0044] Figure 6 This is a structural block diagram of the computer device of the present invention. Detailed Implementation

[0045] The following are some embodiments of the present invention, intended to provide a basic understanding of the invention. They are not intended to identify key or decisive elements of the invention or to limit the scope of protection sought.

[0046] For purposes of brevity and illustrativeness, the principles of the invention are described herein primarily with reference to exemplary embodiments thereof. However, those skilled in the art will readily recognize that the same principles can be equivalently applied to outlier identification methods and devices for all types of sparse time-series data, and that these same principles can be implemented therein, with any such variations not departing from the true spirit and scope of this patent application.

[0047] The outlier identification method and device for sparse time-series data of the present invention are applicable to unsupervised outlier identification problems in sparse time-series data scenarios. The main reason we use a statistical model instead of a machine learning model in this invention is that sparse data lacks obvious regularity, making outlier detection more difficult. Machine learning models often have many preconditions and are not suitable for outlier mining in sparse data. Therefore, a statistical model, which has fewer preconditions and a wider range of applications, is used for outlier discovery.

[0048] Figure 1 This is a summary flowchart of the outlier identification method for sparse time-series data according to the present invention.

[0049] like Figure 1 As shown, the outlier identification method for sparse time-series data of the present invention includes the following steps:

[0050] Step S100: Slice the sparse data on the time series;

[0051] Step S200: Determine the data distribution of the time series;

[0052] Step S300: Generate different alarm thresholds using different alarm threshold generation algorithms based on the data distribution;

[0053] Step S400: Generate alarm rules by fitting the outer contour formed by the alarm threshold;

[0054] Step S500: Calculate the peak periods of the data distribution and correct the calculated peak periods according to the actual situation of the monitored objects; and

[0055] Step S600: Adjust the alarm rules by setting different monitoring indicators for peak and off-peak data and generate the adjusted alarm rules. When the actual data meets the adjusted alarm rules, an alarm is triggered.

[0056] The outlier identification method for sparse time-series data of the present invention can be applied to financial transaction log data, that is, financial transaction log data is processed as "sparse data" in step S100. As a specific example, the outlier identification method for sparse time-series data of the present invention can be used for anomaly identification in financial transaction log data with time periodicity.

[0057] The specific implementation methods for these steps are described below.

[0058] 1. Step S100: Slice the sparse data on the time series.

[0059] Obtain the historical signals of the previous N days of the original time series, segment each signal into M time series segments, and obtain a total of N*M time series segments.

[0060] For sparse time-series digital signals, the first N historical segments (N≥90) are typically selected as a reference for generating threshold rules. The length of the signal segmentation period is determined based on the sparsity of the actual data. When the signal is relatively sparse, a longer segmentation period is considered, resulting in a smaller number of time segments M; when the signal is strong, a shorter segmentation period is considered, resulting in a larger number of time segments M. If the strength of the time-series signal is below a certain threshold, then manually determined rules based on expert experience are considered.

[0061] 2. Step S200: Determine the data distribution of the time series.

[0062] The nonparametric KS test is used to determine whether the historical data in all time series segments follow a normal distribution.

[0063] Specifically, the Kolmogorov-Smirnov test, or KS nonparametric test for short, is a nonparametric test commonly used to determine whether a sample conforms to a pre-given distribution, or whether two samples have different probability distributions. In this invention, it is used to test whether the sample follows a normal distribution. The specific operating steps are as follows:

[0064] 1) For input data X = {x1, x2, ..., x...} n Calculate its mean μ and standard deviation σ, and make the following assumptions:

[0065] Null hypothesis H0: X follows a distribution N(μ, σ) 2 );

[0066] Alternative hypothesis H1: X does not conform to the above distribution.

[0067] The KS statistic is: D = max(σ k Its rejection region is D > D.n，α

[0068] in:

[0069]

[0070] 2) Sort the data X, calculate the cumulative piecewise function of X, and perform a log transformation on the coordinate axes; calculate the theoretical distribution N(μ, σ). 2 The cumulative piecewise function of ).

[0071] 3) Calculate the maximum value of the gap between the actual distribution and the theoretical distribution.

[0072] 4) Compare with the critical value of the distribution to determine whether it falls into the rejection region. If it falls into the rejection region, then the data X does not follow a normal distribution; otherwise, it follows a normal distribution.

[0073] 3. Step S300: Generate different alarm thresholds using different alarm threshold generation algorithms based on the data distribution.

[0074] If the time series data follows a normal distribution, the 3σ model is used to calculate the alarm threshold. When the data exceeds three times the standard deviation, it can be considered an outlier. The calculation formula for the 3σ model is shown in formula (1):

[0075]

[0076] In practice, the above algorithm can be adjusted according to the monitoring indicators. For some monitoring indicators with low volatility or insensitivity to volatility, the alarm threshold can be appropriately reduced, and the 3σ model can be adjusted to a 2-2.5σ model.

[0077] If the data does not follow a normal distribution, the quantile method is used. Data exceeding the 99th quantile can be considered outliers; otherwise, they are considered normal. The formula for calculating the 99th quantile is shown in formula (2):

[0078]

[0079] In practice, the above algorithm can be adjusted according to the monitoring indicators. For some monitoring indicators with low volatility or insensitivity to volatility, the alarm threshold can be appropriately reduced, and the 99th percentile can be adjusted to the 95th to 99th percentile.

[0080] 4. Step S400: Generate alarm rules by fitting the outer contour of the alarm threshold line.

[0081] Since the alarm threshold line generated by the above algorithm is a curve with large fluctuations, it is necessary to process the alarm threshold line to generate threshold alarm rules in actual alarm rule generation. In this invention, the corresponding alarm rules (i.e., initial alarm rules) are generated by fitting the outer contour of the alarm threshold line. The specific operation steps are as follows:

[0082] First, input: X = {x1, x2, ..., x} n}

[0083] The above input values ​​are processed as follows:

[0084] Step 1: Smooth the original alarm threshold line, select the maximum value of the K points before and after it as the threshold of that point, and then use the simple moving average smoothing algorithm to perform secondary smoothing.

[0085] Step 2: For the smoothed alarm threshold line, merge points with the same threshold into a small band. When there is a band length less than δ, continue to step 3.

[0086] Step 3: For different sub-bands, if the value of this sub-band is less different from the absolute value of the threshold line integral of the previous sub-band, then merge it into the previous sub-band; otherwise, merge the value of the previous sub-band into this sub-band. If the difference between the two sub-bands is less than a certain threshold δ, then take the average value of the integrals of the two sub-bands as the merged value.

[0087] Step 4: Determine if there is a band length less than δ. If so, return to step 2; otherwise, output the final result.

[0088] The final output is: X′={x1′,x2′,...,x n ′}.

[0089] Figure 2 This diagram illustrates the fitting process for the alarm threshold line.

[0090] like Figure 2 As shown, for sparse time-series data, the generated alarm threshold line is as follows: Figure 2 As shown in line A, the horizontal axis represents the time series, and the vertical axis represents the values ​​of the monitored indicators. The alarm fitting line generated by fitting the alarm threshold line shown in line A (a broken line with peaks and troughs) is shown in line B (a straight line). In practical scenarios, the corresponding alarm rules can be configured based on the values ​​of the orange line.

[0091] 5. Step S500: Calculate the peak periods of data distribution and correct the calculated peak periods according to the actual situation of the monitored objects.

[0092] Calculate the peak period length and peak period starting point. For example, select points above the 60th percentile as candidate peak periods, and exclude points below the 60th percentile. Due to significant time differences between some monitored objects and Beijing time zones, their data peak periods span midnight. Considering the continuity of time zones, peak periods in some areas may be concentrated in the middle of the night and early morning. Their peak periods are located at the beginning and end of the time series data, while the middle portion is not a peak period. Therefore, it is necessary to calculate the maximum value sequence segment of the circular array. The specific steps are as follows:

[0093] First, input: X = {x1, x2, ..., x} n}

[0094] The above input values ​​are processed as follows:

[0095] Step 1: Determine the input data X = {x1, x2, ..., x...} n We determine whether each point in the matrix is ​​higher than the 60th quantile to obtain X0 = {f(x1), f(x2), ..., f(x...}. n )},f(x n () represents the value of each point;

[0096]

[0097] Step 2: Calculate the peak period length and peak period start point, as shown below:

[0098] Step 2-1: Current longest time period curMax, historical longest time period sumMax = 0, -inf

[0099] The current shortest time interval curMin, the historical shortest time interval sumMin = 0, inf

[0100] Total value per period = 0

[0101] The start point of the current longest time period, curMax_start, and the end point of the current longest time period, curMax_end, are both 0 and 0.

[0102] The longest historical time period starts at sumMax_start, and ends at sumMax_end = 0.

[0103] The start point of the current shortest time interval, curMin_start, and the end point of the current shortest time interval, curMin_end, are both 0 and 0.

[0104] The start point of the shortest historical time interval, sumMin_start, and the end point of the shortest historical time interval, sumMin_end = 0, 0

[0105] Step 2-2: Extract the element x and its index from sequence X0.

[0106] Steps 2-3: curMax = max(x + curMax, x)

[0107] curMax_end = index

[0108] If x + curMax < x,

[0109] Then curMax_start = index

[0110] sumMax = max(sumMax, curMax)

[0111] If sumMax < curMax,

[0112] Then sumMax_start, sumMax_end=curMax_start, curMax_end

[0113] curMin = min(x + curMin, x)

[0114] curMin_end = index

[0115] If x + curMin > x

[0116] Then curMin_start = index

[0117] sumMin = min(sumMin, curMin)

[0118] If sumMin <curMax,

[0119] Then sumMin_start, sumMin_end = curMin_start, curMin_end

[0120] total + = x

[0121] Step 2-4: If there are still elements in sequence X0, proceed to S2-2; otherwise, proceed to S2-5.

[0122] Steps 2-5: If sumMax > total - sumMin and sumMax > 0

[0123] Then start, end, length = sumMax_start, sumMax_end, sumMax

[0124] Otherwise start, end, length = sumMin_start, sumMin_end, total-sumMin.

[0125] The final output is: the start position of the longest sequence segment (start), the end position (end), and the length of the sequence.

[0126] The calculated peak period is corrected based on the actual time zone of the monitored object. Generally, the peak period of the monitored object is considered to be between 8:00 and 24:00 local time. The time when the calculated peak period overlaps with the peak time zone is selected as the final peak period result.

[0127] In addition, there are differences between the peak period calculation results for ordinary monitored objects and the peak period calculation results for monitored objects with large time differences. Figure 3 A diagram showing the calculation results for ordinary monitored objects during peak periods. Figure 4 A schematic diagram showing the calculation results of a large time zone monitoring object during peak periods.

[0128] like Figure 3 As shown, in the calculation of peak periods for ordinary monitored objects, the curve represented by line D indicates the change of data frequency over time. Line D1 is the data peak period calculated by the algorithm of the present invention. Line D2 represents the approximate range of data peak periods estimated based on the time zone of the monitored object. Line D3 is the part where lines D1 and D2 overlap, representing the final peak period.

[0129] like Figure 4 As shown, for monitoring objects with a large time difference from Beijing time and whose data peak period spans midnight, the curve represented by line E indicates the change in data frequency over time. Line E1 is the data peak period calculated by the algorithm of the present invention, line E2 represents the approximate range of the data peak period estimated based on the time zone of the monitoring object, and line E3 is the part where lines E1 and E2 overlap. It can be seen that the present invention is also applicable to monitoring objects with a large time difference and whose data peak period spans midnight.

[0130] 6. Step S600: Set different monitoring indicators for peak and off-peak data to adjust alarm rules. Trigger an alarm when the actual data meets the alarm rules.

[0131] In this invention, it is preferable to set different monitoring indicators for peak and off-peak periods. This is because the data sparsity is lower and the data is more stable during peak periods, allowing for additional sparsity-sensitive monitoring indicators. For off-peak data, no sparsity-sensitive monitoring indicators are set. Then, the corresponding method in step S400 is used to generate alarm rules, resulting in adjusted alarm rules (the steps of generating alarm rules and adjusting them to generate adjusted alarm rules can be completed in a single "alarm rule generation step"). An alarm is triggered when the actual data matches the adjusted alarm rules.

[0132] The specific methods for setting sparsity-sensitive monitoring indicators include setting both sparsity-sensitive and non-sparseness-sensitive indicators. Sparsity-sensitive monitoring indicators are those whose values ​​fluctuate significantly in sparse time-series scenarios but less so in data-intensive scenarios, thus reflecting data sparsity to some extent. Examples include: length of time without signal and negative fluctuation ratio. Non-sparseness-sensitive monitoring indicators are those whose values ​​do not change much in either sparse or data-intensive scenarios.

[0133] The outlier identification method for sparse time-series data of the present invention has been described above. The outlier identification device for sparse time-series data of the present invention will be described below.

[0134] Figure 5 This is a schematic diagram of the outlier identification device for sparse time-series data according to the present invention.

[0135] like Figure 5 As shown, the outlier identification device for sparse time-series data of the present invention includes:

[0136] The real-time signal acquisition module 100 is used to acquire real-time timing signals;

[0137] The data processing module 200 is used to preprocess real-time time-series signals, such as processing the data format, removing certain non-compliant data, and slicing the time-series signals.

[0138] The data calculation module 300 is used for outlier identification in sparse time series data. Specifically, it includes: analyzing and calculating historical signals; using different alarm threshold generation algorithms based on the distribution of historical data; processing the alarm thresholds to obtain alarm rules; identifying peak and off-peak periods; applying different monitoring indicators to peak and off-peak periods; and generating alarm rules.

[0139] The operation and maintenance processing module 400 is used to trigger alarms when the relevant indicator values ​​of the actual signal are higher than the threshold line of the alarm rules. It can also further classify the importance of alarms according to the type of alarm object, the type of abnormal indicator, the alarm frequency, etc., and automatically push them to the relevant operation and maintenance personnel.

[0140] The real-time signal acquisition module 100 is responsible for receiving real-time data from various upstream business systems and transmitting the real-time data flow to the data processing module 200. Here, as an example, the "real-time data from various upstream business systems" mentioned above could be financial transaction log data.

[0141] The data processing module 200 preprocesses the data from the real-time signal acquisition module to ensure that the data format meets the requirements of the data calculation module 300.

[0142] The data calculation module 300 analyzes historical data to obtain the characteristics of abnormal data, the threshold for judging abnormal data, and the range of peak periods for the data. Based on whether the data's time interval falls within the peak period range, different monitoring indicators are applied to the data. Sparse-sensitive monitoring indicators are used within the peak period range, while non-sparse-sensitive monitoring indicators are not used within the off-peak period range. Finally, alarm rules are generated. When the relevant indicator value of the actual signal reaches the threshold line set in the alarm rule, the operation and maintenance processing module 400 generates an alarm and pushes it to the operation and maintenance personnel for further processing.

[0143] Specifically, the data processing module 200 includes:

[0144] The data format standardization submodule 210 preprocesses the raw time-series signals, including data cleaning, data auditing, and data verification, to ensure that the data conforms to business specifications and that the data quality meets the standards for analysis and use.

[0145] The data content definition submodule 220 sets different content filtering rules for different monitoring objects, and selectively removes certain data and content that do not need to be monitored to achieve a personalized monitoring solution.

[0146] The data signal segmentation module 230 segments the daily data signal into second- or minute-level segments to obtain time-series data fragments; and

[0147] The data signal retrieval submodule 240 retrieves real-time or historical time-series data existing in the system database, facilitating statistical analysis of the data.

[0148] Specifically, the data calculation module 300 includes:

[0149] The data distribution judgment submodule 310 uses the KS nonparametric test to determine whether the data follows a normal distribution and uses different outlier identification algorithms for data with different distributions.

[0150] The outlier identification algorithm application submodule 320 states that if the data distribution of the monitored object follows a normal distribution, the alarm threshold line generated by the 3σ model should be used; otherwise, the alarm threshold line generated by the 99th quantile model should be used. For data with a small daily average data volume, statistical rules are no longer applicable, and manual rules set by expert experience should be used for monitoring.

[0151] The alarm rule generation submodule 330 generates alarm rules based on the alarm threshold line, and adjusts the generated alarm rules according to different monitoring indicators for peak and off-peak periods based on the data peak period calculation submodule 340, and generates the adjusted alarm rules.

[0152] The data peak period calculation submodule 340 determines whether the data is in a peak period. For sparse data, the regularity of peak period data is more obvious, and the identification of outliers is more accurate. It further uses the actual time zone of the monitored object for correction, selects the time when the calculated peak period overlaps with the peak time zone as the final peak period result, and divides it into peak period and non-peak period.

[0153] In the operation and maintenance processing module 400, an alarm is triggered when the relevant indicator value of the actual signal exceeds the threshold line of the alarm rule. The alarm importance is further categorized based on the alarm object type, abnormal indicator type, and alarm frequency. For example, monitoring objects on the key monitoring object list have higher alarm importance; certain monitoring indicators reflecting major operation and maintenance problems have higher importance; and alarms of a certain type that occur frequently have higher importance. Operation and maintenance personnel promptly implement operation and maintenance operations based on the received alarm information, prioritizing alarms of higher importance. When the monitoring indicator value no longer meets the alarm rule, an alarm recovery notification is triggered, and the alarm cycle ends.

[0154] Figure 6 This is a structural block diagram of the computer device of the present invention.

[0155] like Figure 6 As shown, embodiments of this disclosure also provide a computer device 500, including a memory 510 and a processor 520, wherein the memory 510 is configured to store computation instructions, and the processor 520 is configured to execute the outlier identification method for sparse time-series data of the present invention as provided in any of the foregoing embodiments when running the computer instructions.

[0156] The memory 510 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory, electrically erasable programmable read-only memory, erasable programmable read-only memory, programmable read-only memory, read-only memory, magnetic memory, flash memory, magnetic disk or optical disk, etc.

[0157] The processor 520 may be a central processing unit, a field-programmable logic array, a microcontroller, a digital signal processor, or an application-specific integrated circuit (ASIC), or other logic devices with data processing and / or program execution capabilities. One or more processors may be configured to simultaneously execute the stereo equalization adjustment method described above as a parallel computing processor group, or to execute some steps of the stereo equalization adjustment method with some processors, and other steps of the stereo equalization adjustment method with some processors. Computer instructions include one or more processor operations defined by the instruction set architecture corresponding to the processor, and these computer instructions may be logically contained and represented by one or more computer programs.

[0158] The computer device 500 can also connect to various input devices (such as user interface, keyboard, etc.), various output devices (such as speakers, etc.), and displays to enable interaction between the computer device and other products or users, which will not be elaborated here.

[0159] As described above, the innovation of this application is mainly reflected in three aspects: The outlier identification method and device for sparse time-series data of this invention consider the sparsity of time-series data from multiple perspectives, including the overall sparsity of the data, the sparsity of different time periods, and sparsity-sensitive monitoring indicators, resulting in high accuracy in outlier identification of sparse time-series data. Furthermore, it considers the different monitoring objects, such as the differences between normally distributed and non-normally distributed monitoring objects, and between ordinary monitoring objects and large time-difference monitoring objects, demonstrating strong compatibility in practical scenarios. Moreover, this invention can further transform complex algorithm results into simple monitoring rules. Compared to schemes that perform real-time calculations on all data, this invention adopts a "single calculation, periodic update" approach for sparse data, greatly improving computer processing efficiency while ensuring the effectiveness of the algorithm results, making it suitable for real-time monitoring scenarios.

[0160] The above examples primarily illustrate the outlier identification method and apparatus for sparse time-series data according to the present invention. Although only some specific embodiments of the present invention have been described, those skilled in the art should understand that the present invention can be implemented in many other forms without departing from its spirit and scope. Therefore, the examples and embodiments shown are considered illustrative rather than restrictive, and the present invention may cover various modifications and substitutions without departing from the spirit and scope of the invention as defined by the appended claims.

Claims

1. A method for outlier identification in sparse time-series data, characterized in that, include: The time series slicing step involves slicing the sparse time series data of the monitored object into segments to obtain time series segment data. The data distribution judgment step determines whether the time series segment data conforms to a normal distribution. The alarm value generation step involves using different alarm threshold generation algorithms to generate different alarm thresholds for time series data segments that are judged to conform to a normal distribution and those that are judged not to conform to a normal distribution. The peak period calculation step involves calculating the peak period of the time series data segment and correcting the calculated peak period according to the time zone of the monitored object to obtain peak period data and non-peak period data. The alarm rule generation step involves generating alarm rules based on the different alarm thresholds, and adjusting the alarm rules by setting different monitoring indicators for peak and off-peak data to generate adjusted alarm rules. as well as The alarm maintenance procedure involves triggering an alarm when the actual data of the monitored object matches the adjusted alarm rules. The peak period calculation steps include: Set specified thresholds for peak periods; For the data distribution of the time series segment data, the length of the peak period and the starting point of the peak period are calculated by determining whether the time series segment data exceeds a predetermined threshold of the peak period; and The portion of the calculated peak period that overlaps with the time zone of the monitored object is selected as the corrected peak period. The time series data segment is then divided into peak period data and non-peak period data. Specifically, sparsity-sensitive monitoring indicators are set for peak period data, while sparsity-sensitive monitoring indicators are not set for off-peak period data. The sparsity-sensitive monitoring indicators refer to indicators that have large numerical fluctuations in sparse time-series scenarios but small numerical fluctuations in data-intensive scenarios.

2. The outlier identification method for sparse time-series data as described in claim 1, characterized in that, In the time series slicing step, the length of the segment is determined based on the sparsity of the sparse time series data.

3. The outlier identification method for sparse time-series data as described in claim 1, characterized in that, In the data distribution judgment step, the KS nonparametric test is used to determine whether the data distribution of the time series segment data conforms to a normal distribution.

4. The outlier identification method for sparse time-series data as described in claim 3, characterized in that, In the alarm value generation step, for data that is determined to conform to a normal distribution, the 3σ model is used to calculate the alarm threshold line; for data that is determined not to conform to a normal distribution, the quantile discrimination method is used to calculate the alarm threshold line.

5. The outlier identification method for sparse time-series data as described in claim 3, characterized in that, In the alarm rule generation step, alarm rules are generated by fitting the outer contour of the alarm threshold line formed by the alarm threshold.

6. An outlier identification device for sparse time-series data, characterized in that, include: The real-time signal acquisition module is used to acquire real-time timing signals. The data processing module is used to preprocess real-time time-series signals to obtain the data. The data calculation module is used to slice the sparse time-series data of the monitored object into segments on the time series to obtain time-series segment data. It determines whether the data distribution of the time-series segment data conforms to a normal distribution. For time-series segment data that is determined to conform to a normal distribution and time-series segment data that is determined not to conform to a normal distribution, different alarm threshold generation algorithms are used to generate different alarm thresholds. The peak period of the time-series segment data is calculated and corrected according to the time zone of the monitored object to obtain peak period data and non-peak period data. Alarm rules are generated based on the different alarm thresholds. The alarm rules are adjusted by setting different monitoring indicators for peak period data and non-peak period data to generate adjusted alarm rules. as well as The operation and maintenance processing module is used to trigger alarms when the actual data of the monitored object matches the adjusted alarm rules. Specifically, the data calculation module sets a predetermined threshold for peak periods. For the data distribution of the time series segment data, the length and starting point of the peak period are calculated by determining whether the time series segment data exceeds the predetermined threshold. The portion of the calculated peak period that overlaps with the time zone of the monitored object is selected as the corrected peak period. The time series segment data is divided into peak period data and non-peak period data. For the peak period data, a sparsity-sensitive monitoring indicator is set, while for the non-peak period data, no sparsity-sensitive monitoring indicator is set. The sparsity-sensitive monitoring indicator refers to an indicator whose value fluctuates greatly in sparse time series scenarios but less in data-intensive scenarios.

7. The outlier identification device for sparse time-series data as described in claim 6, characterized in that, The data processing module includes: The data format standardization submodule preprocesses timing signals; The data content definition submodule allows you to set different content filtering rules for different monitored objects to remove data that does not need to be monitored. The data signal segmentation module slices the time-series signal to obtain time-series data segments; and The data signal retrieval submodule retrieves real-time or historical time-series data.

8. The outlier identification device for sparse time-series data as described in claim 6, characterized in that, The data calculation module includes: The data distribution judgment submodule uses the KS nonparametric test to determine whether the time series segment data follows a normal distribution, and uses different outlier identification algorithms for data with different distributions; The outlier identification algorithm application submodule should use the alarm threshold line generated by the 3σ model if the outlier conforms to a normal distribution; otherwise, it should use the alarm threshold line generated by the 99th quantile model if the outlier does not conform to a normal distribution. The data peak period calculation submodule determines whether the time series data segment is in a peak period and corrects for the high period based on the time zone to obtain the final peak period; and The alarm rule generation submodule generates alarm rules based on alarm threshold lines. Different monitoring indicators are used for peak and off-peak periods to adjust the generated alarm rules, resulting in adjusted alarm rules.

9. The outlier identification device for sparse time-series data as described in claim 6, characterized in that, In the operation and maintenance processing module, when an alarm is triggered, the alarm information is pushed after further classifying the importance of the alarm based on the alarm object type, abnormal indicator type, and alarm frequency.

10. A computer-readable medium having a computer program stored thereon, characterized in that, When executed by a processor, the computer program implements the outlier identification method for sparse time-series data as described in any one of claims 1 to 5.

11. A computer device, comprising a storage module, a processor, and a computer program stored on the storage module and executable on the processor, characterized in that, When the processor executes the computer program, it implements the outlier identification method for sparse time-series data as described in any one of claims 1 to 5.

Images