Time series data outlier detection method, device, computer equipment and storage medium
Through sliding window technology, the sliding tree is generated and the abnormality rate is calculated, and the outliers are identified, which solves the problem of outliers masking in growth trend-based time series data, and realizes effective detection in Poisson distribution data, which is suitable for commercial sales data analysis.
Patent Information
- Application Number
- CN202411129459.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-16
- Publication Date
- 2024-11-22
- Estimated Expiration
- 2044-08-16
AI Technical Summary
The prior art cannot effectively detect outliers in growth trend-based timing data, especially in Poisson distributed timing data, resulting in outliers being masked, and multi-dimensional detection methods are not suitable for one-dimensional timing data.
The sliding window technology is used to generate the minimum and maximum value sliding trees, calculate the coefficient of variation and full window rate of the off-average sliding tree, determine the sliding anomaly rate, identify and merge the abnormal sliding trees until all sequences are traversed to identify adjacent outliers.
It effectively solves the problem of outlier occlusion, and does not need to strictly comply with Gaussian distribution or other distributions. It performs well in commercial sales timing data detection, and is suitable for data visual analysis and intelligent detection, and has market promotion value.
Smart Images

Figure CN118656760B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of data processing technology, and in particular to a method, device, computer equipment and storage medium for detecting outliers in time series data. Background Art
[0002] In the field of time series data outlier detection, there is no professional detection method for time series data with growth trends and conforming to the Poisson distribution. In actual scenarios, the "outlier masking problem" caused by time series data with growing sales volume results in the inability to detect outliers in the early stages of the time series data. Or if sales fluctuate slightly over a period of time and suddenly increase in leaps and bounds, but the growth does not reach the outlier level of the overall time series data, it will also be masked by the detection algorithm. Statistics ,difference , IQR and Grubbs cannot effectively solve the "outlier masking problem". The classic methods LOF, iForest, DBSCAN and machine learning methods SVM, Knn, K-means are multidimensional detection methods and are not suitable for outlier detection of one-dimensional time series data. Summary of the invention
[0003] Based on this, it is necessary to provide a method, device, computer equipment and storage medium for detecting outliers in time series data in response to the above technical problems.
[0004] A method for detecting outliers in time series data, the method comprising:
[0005] Get the time series data of product sales growth.
[0006] According to the time series data and the preset sliding window size, the minimum sliding tree and the maximum sliding tree of each sliding window are generated.
[0007] According to the minimum sliding tree and the maximum sliding tree of each sliding window, the mean difference sliding tree and the maximum mean difference value of the corresponding sliding window are determined.
[0008] Calculate the coefficient of variation for each sliding tree from the mean.
[0009] The full window rate of the sliding window is determined according to the maximum mean difference value and the preset sliding window size.
[0010] According to the coefficient of variation of each sliding tree from the mean difference and the full window rate of the corresponding sliding window, the sliding anomaly rate is determined, and the abnormal sliding tree and the outliers therein are determined according to the sliding anomaly rate.
[0011] When the maximum mean difference value identified within the first round of sliding window is not less than 1 time, and the coefficient of variation is greater than the preset variation threshold, the abnormal sliding trees are merged into a first abnormal sliding tree, and the abnormal values in the first round of abnormal sliding tree are updated to the sliding window average value, and the second round of sliding window calculation is entered until all sequences corresponding to the first abnormal sliding trees are traversed to obtain adjacent abnormal values.
[0012] A time series data outlier detection device, the device comprising:
[0013] The time series data acquisition module is used to obtain the time series data of commodity sales growth.
[0014] The mean difference sliding tree generation module is used to generate the minimum sliding tree and the maximum sliding tree of each sliding window according to the time series data and the preset sliding window size; according to the minimum sliding tree and the maximum sliding tree of each sliding window, the mean difference sliding tree and the maximum mean difference value of the corresponding sliding window are determined.
[0015] The abnormal sliding tree identification module is used to calculate the coefficient of variation of each mean difference sliding tree; determine the full window rate of the sliding window according to the maximum mean difference value and the preset sliding window size; determine the sliding anomaly rate according to the coefficient of variation of each mean difference sliding tree and the full window rate of the corresponding sliding window, and determine the abnormal sliding tree and the abnormal value therein according to the sliding anomaly rate.
[0016] The adjacent outlier detection module is used to merge the abnormal sliding trees into a first abnormal sliding tree when the maximum mean difference value identified within the first round of sliding window is not less than 1 time and the coefficient of variation is greater than the preset variation threshold. The outlier values in the first round of abnormal sliding tree are updated to the sliding window average value, and the second round of sliding window calculation is entered until all sequences corresponding to the first abnormal sliding trees are traversed to obtain adjacent outliers.
[0017] A computer device comprises a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the above method when executing the computer program.
[0018] A computer-readable storage medium stores a computer program, which implements the steps of the above method when executed by a processor.
[0019] The above-mentioned method, device, computer equipment and storage medium for detecting outliers in time series data, the method is to obtain time series data of commodity sales growth; generate an extreme sliding tree according to the time series data and a preset sliding window size; generate a sliding tree of the sliding window deviation from the mean and the maximum mean difference according to the extreme sliding tree; calculate the coefficient of variation of each sliding tree of deviation from the mean, calculate the full window rate and the sliding anomaly rate, determine the abnormal sliding tree and the anomaly therein according to the sliding anomaly rate; determine whether there are adjacent anomalies, and identify adjacent anomalies. This method can effectively solve the problem of outlier masking; secondly, there is no need to strictly require whether the time series data conforms to the Gaussian distribution or other distributions, and it shows good detection effect in commercial sales time series data; because the method is efficient and easy to understand, it is suitable for data visualization analysis and intelligent detection, so it has general market promotion value. BRIEF DESCRIPTION OF THE DRAWINGS
[0020] Figure 1 1 is a flow chart of a method for detecting abnormal values of time series data in one embodiment;
[0021] Figure 2 A schematic diagram of a process of generating an extreme sliding tree and a sliding forest from time series data in another embodiment;
[0022] Figure 3 A schematic diagram of a process of merging the same maximum values into a sliding tree in another embodiment;
[0023] Figure 4 A process for generating a sliding tree of deviations from the mean in another embodiment;
[0024] Figure 5 is the time series data of commodity sales in another embodiment;
[0025] Figure 6 is a structural block diagram of a device for detecting abnormal values of time series data in one embodiment;
[0026] Figure 7 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION
[0027] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.
[0028] In one embodiment, Figure 1 As shown, a method for detecting outliers in time series data is provided, and the method comprises the following steps:
[0029] Step 100: Obtain time series data of commodity sales growth.
[0030] Specifically, the time series data of commodity sales growth is generally the daily sales, weekly sales or monthly sales of merchants' commodities. The time series data of commodity sales shows a growth trend due to the sales goals and sales strategies of merchants. For example, the daily sales time series data of an e-commerce company in the first month of its operation are: {738, 717, 694, 815, 738, 710, 622, 685, 674,643, 734, 1055, 1445, 896, 877, 1731, 818, 899, 1050, 1019, 1018, 1237, 1339,1181, 1269, 1218, 1242, 1287, 1483, 1623, 1489}. We can observe an obvious growth trend of the time series data from the example. This type of time series data is the sales growth time series data.
[0031] Step 102: Generate a minimum sliding tree and a maximum sliding tree for each sliding window according to the time series data and a preset sliding window size.
[0032] Specifically, the core idea of generating sliding trees from time series data is to obtain the maximum value as the root node through the sliding window, and at the same time, the time series data points in the sliding window are used as leaves (including the maximum value itself). The schematic diagram of the process of generating sliding trees and sliding forests from time series data is as follows: Figure 2 As shown in Figure 1, when the root node data of each sliding tree in the sliding window has the same value, multiple sliding trees are merged into one sliding tree. The process diagram of merging the same maximum value into one sliding tree is shown in Figure 1. Figure 3 shown.
[0033] Will Figure 2 The sequence in is called , set the default sliding window size , then from Start each arrive should be slid 7 times. For example, the sequence From sliding window 1 to 7, the generated sliding tree. Figure 3 As shown, each of the 7 minimum sliding trees is generated with 3 as the root, so they can be merged into one sliding tree.
[0034] Preset sliding window size k It is usually set to an odd number as the preferred preset sliding window size k Set to 7.
[0035] Step 104: According to the minimum sliding tree and the maximum sliding tree of each sliding window, determine the mean difference sliding tree and the maximum mean difference value of the corresponding sliding window.
[0036] Specifically, the maximum mean difference is the maximum value in a certain section of the sequence. When the sliding window slides once or multiple times, the sequence maximum value corresponding to the maximum mean difference in the sequence obtained by calculating the deviation from the mean is always the same value, that is, the maximum value of the sequence segment, and the maximum value may be the minimum value or the largest maximum value, but only one of the maximum values can be calculated as the maximum mean difference.
[0037] Step 106: Calculate the coefficient of variation of each sliding tree from the mean difference.
[0038] Specifically, in the sequence data generation In a forest of sliding trees, all of them are maximum sliding trees. Generally, the preset sliding window size is set k The smaller the value, the more sliding trees are generated. Therefore, it is necessary to identify the abnormal sliding trees from the maximum sliding trees and then filter out the non-abnormal sliding trees. For this situation, the coefficient of variation is introduced and used as the preset mutation threshold. change .
[0039] Generally, the coefficient of variation caused by the maximum value in the time series data sequence and the sequence fluctuation is greater than 25% and is recorded as an outlier, which can also be set to 30%. In this embodiment, the coefficient of variation of each sliding tree from the mean difference is calculated, and the sliding trees with less than 25% are filtered out. This process is called removal.
[0040] (1)
[0041] (2)
[0042] in, is the standard deviation of the sliding window, is the sliding average of the sliding window, is the coefficient of variation of the sliding window.
[0043] Step 108: Determine the full window rate of the sliding window according to the maximum mean difference value and the preset sliding window size.
[0044] Step 110: Determine the sliding anomaly rate according to the coefficient of variation of each sliding tree from the mean difference and the full window rate of the corresponding sliding window, and determine the abnormal sliding tree and the abnormal value therein according to the sliding anomaly rate.
[0045] Step 112: When the maximum mean difference value identified within the first round of sliding window is not less than 1 time, and the coefficient of variation is greater than the preset variation threshold, the abnormal sliding trees are merged into a first abnormal sliding tree, and the abnormal values in the first round of abnormal sliding tree are updated to the sliding window average value, and the second round of sliding window calculation is entered until all sequences corresponding to the first abnormal sliding trees are traversed to obtain adjacent abnormal values.
[0046] Specifically, in actual scenarios, operational promotions are continuous events, so outliers may appear adjacent or continuous. Since the sliding forest algorithm uses the method of generating the maximum value and the maximum deviation from the mean, adjacent outliers will be obscured.
[0047] The starting conditions for processing adjacent outliers in time series data are: the window size and the first round of sliding Keep consistent, that is ; The maximum mean difference value is identified within the first round of sliding window 1 time and the coefficient of variation is greater than the set threshold, that is ; The algorithm should enter the judgment of adjacent outliers.
[0048] The specific steps for judging adjacent outliers are as follows: 1) The identified abnormal sliding trees are merged into one sliding tree because the maximum mean difference values are the same; 2) The maximum mean difference values ≥ 1 times within the first round of sliding window are identified and the coefficient of variation is greater than the set threshold; 3) The anomalies in the first round of abnormal sliding trees are changed to the sliding window average value and enter the second round of sliding forest algorithm judgment.
[0049] For example, time series data: {3785, 3611, 3372, 3348, 3156, 3423, 8744, 9946,3797, 3663, 3531, 3240, 3873, 4164}, 8744 and 9946 are adjacent outliers and were not identified in the first round of calculations, as shown in Table 1-a. This is because some of the indicators calculated in the first round meet the conditions for entering adjacent outliers, as shown in Table 1-b, which meet the conditions for appearing as the maximum mean difference ≥ 1 times and the coefficient of variation is 44.28%, which is greater than the set 25%. Calculate The average value is 4451 (remove 3785 and calculate again, because the sliding tree with 9946 as the node does not contain 3785). Replace the previous 9946 and enter the second round of sliding forest calculation, identifying 8744 as an outlier, as shown in Table 1-c.
[0050] Table 1-a
[0051]
[0052] Table 1-b
[0053]
[0054] Table 1-c
[0055]
[0056] It is worth noting that the preset sliding window size k It is usually set to an odd number as the preferred preset sliding window sizek Set to 7, the preset mutation threshold is usually 25%; the larger the preset sliding window size and the preset mutation threshold are, the fewer outliers are found. Therefore, users can adjust them according to actual application needs until the actual effect is achieved.
[0057] In the above-mentioned time series data outlier detection method, the method obtains the time series data of the growth type of commodity sales; generates the maximum sliding tree according to the time series data and the preset sliding window size; generates the sliding tree of the sliding window deviation from the mean and the maximum mean difference according to the maximum sliding tree; calculates the coefficient of variation of each sliding tree of the deviation from the mean, calculates the full window rate and the sliding anomaly rate, and determines the abnormal sliding tree and the anomaly therein according to the sliding anomaly rate; determines whether there are adjacent anomalies, and identifies the adjacent anomalies. This method can effectively solve the problem of outlier masking; secondly, there is no need to strictly require whether the time series data conforms to the Gaussian distribution or other distributions, and it shows good detection effect in commercial sales time series data; because the method is efficient and easy to understand, it is suitable for data visualization analysis and intelligent detection, so it has general market promotion value.
[0058] In one embodiment, the method further includes: using a special tail processing method to identify tail outliers; wherein in the special tail processing method, conventional calculations are performed from sliding window 1 to sliding window 7, regardless of whether the condition that the sliding window is equal to 7 is satisfied, and only whether there is a computable time series value is considered.
[0059] Specifically, due to the shortcomings of the sliding window itself: the head and tail of the time series data cannot be set according to the It slides 7 times, so it cannot be identified as the maximum mean difference of 7 times. However, the sliding forest needs to calculate the full window rate, so special processing is required. The calculation process of the sliding anomaly rate is shown in Table 2.
[0060] Product sales time series data such as Figure 5 As shown, Figure 5 The sales time series data of the goods in the figure are the daily sales data of an e-commerce company from January to May 2022. It can be observed that the circled values in the figure are the tail of the sequence, so "tail special processing" is used to identify tail outliers. In Table 2, from sliding windows 1 to 7, conventional calculations are performed regardless of whether they meet k =7, only considers whether there is a computable time series value.
[0061] Table 2 Calculation process of sliding abnormality rate
[0062]
[0063] In one embodiment, the method further comprises: k=7, when at least two equal values appear in the seven sliding windows and are judged as outliers, during the first round of sliding window calculation, the maximum and minimum values of the windows are set to random values respectively. After the first round of sliding window calculation, the equal values are directly marked as outliers; k The preset sliding window size.
[0064] Specifically, for k =7 appears in the 7 sliding windows If the value is equal to and is judged as an outlier, then a value is randomly selected in the steps of obtaining the minimum and maximum values of the window, and after the first round of calculation, it can be directly marked as an outlier.
[0065] In one embodiment, step 102 includes: setting the preset sliding window size to k ; Use a window size of k The window of is slid on the time series data to obtain several sliding windows; the sliding window includes k sequence points; calculate the maximum and minimum values of each sliding window; take the maximum value of each sliding window as the root node of the sequence points in the sliding window, and the sequence points in the sliding window as leaves to generate the sliding tree of the maximum value of the sliding window; take the minimum value of each sliding window as the root node of the sequence points in the sliding window, and the sequence points in the sliding window as leaves to generate the sliding tree of the minimum value of the sliding window; when the continuous k When the root nodes of the maximum sliding trees have the same value, k Maximum sliding trees are merged into a maximum sliding tree; when consecutive k When the root nodes of the minimum sliding trees have the same value, k The minimum sliding trees are merged into one minimum sliding tree.
[0066] Specifically, if the outlier is the sequence Or sequence segment The minimum or maximum data point, and the data point is a numeric type. Then there is
[0067] (3)
[0068] (4)
[0069] in, represents the maximum value of the sliding window, The sequence representing the sliding window range, subscript Indicates the parameters for setting the sliding window size. Indicates the minimum value of the sliding window.
[0070] Will The table is used as the root node of the sequence point in the sliding window, and the sequence point (including the maximum value itself) is used as the leaf (for calculation and ), so a maximum sliding tree can be generated, Similarly, generate the minimum sliding tree, such as Figure 2 shown.
[0071] In one embodiment, step 104 includes: calculating the maximum mean difference distance and the minimum mean difference distance of each sliding window, i The expressions of the maximum mean difference distance and the minimum mean difference distance of the sliding windows are:
[0072] (5)
[0073] (6)
[0074] (7)
[0075] in, Indicates i The maximum mean difference distance of sliding windows, Indicates i The maximum value of sliding windows, is the sliding average of the sliding window, The sequence segment representing the sliding window range, subscript Indicates the preset sliding window size; represents the minimum mean difference distance, Indicates the minimum value of the sliding window.
[0076] The sequence data point corresponding to the root node of the sliding tree corresponding to the larger value of the maximum mean difference distance and the minimum mean difference distance of each sliding window is taken as the maximum mean difference value; the maximum mean difference value is taken as the root node of the maximum sliding tree and the minimum sliding tree to generate a mean difference sliding tree.
[0077] Specifically, each time the maximum sliding tree window slides, two trees are generated. Therefore, to determine which one is the real maximum sliding tree of the window, the two sliding trees are processed with the mean difference to generate the mean difference sliding tree. The process of generating the mean difference sliding tree is as follows: Figure 4 shown.
[0078] Steps to generate a sliding tree of deviations from the mean: 1. Obtain the sliding tree of the extreme values; 2. Calculate the distance of the two extreme values from the mean; 3. Generate a sliding tree of deviations from the mean.
[0079] 1) Get the maximum sliding tree
[0080] The sliding tree is generated using equations (1) and (2) to generate a sliding tree of deviations from the mean.
[0081] 2) Calculate the distance between the two extreme values and the mean difference
[0082] The greater the distance between the minimum or maximum value in the sliding window and the average value of the window sequence, the more likely it is that the maximum value is an outlier. Therefore, it is necessary to process the maximum value from the mean difference and filter the maximum value sliding tree with a smaller distance. First, use formula (7) to calculate the sliding average of the sliding window; then use formula (5) and formula (6) to calculate the maximum value. ,, and Perform Euclidean distance calculation.
[0083] 3) Generate a sliding tree of deviations from the mean
[0084] based on and The sequence data points with the largest distance difference (here referred to as the maximum mean difference value) are judged as the root nodes of the two sliding trees to generate the mean difference sliding tree. The calculation formula is as follows: Figure 4 shown.
[0085] (8)
[0086] In one embodiment, step 108 includes: determining the full window rate of the sliding window according to the maximum mean difference value and the preset sliding window size as follows:
[0087] (9)
[0088] in, represents the full window rate, Indicates the number of times the maximum mean difference of the sliding window is identified, k The preset sliding window size.
[0089] Specifically, the maximum mean deviation value is taken as the maximum value in a certain sequence segment, so when the sliding window slides once or multiple times, the maximum mean deviation value in the sequence is always the same value, that is, the maximum value of the sequence segment. k =7, then the same maximum value of 7 leaf nodes will appear 7 times in the sliding interval; set k =3 The same maximum value appears 3 times, and so on. Take {12, 13, 18, 10, 9, 13, 3, 17,22, 18, 15, 14, 13} as an example to calculate the full window rate. The calculation process of the full window rate is shown in Table 3:
[0090] Table 3 Calculation process of full window rate
[0091]
[0092] The maximum mean difference value in Table 3 is obtained based on the calculation indicators of "minimum value, maximum value, average value, standard deviation, coefficient of variation, minimum mean difference distance and maximum mean difference distance" in the previous part. k =7, the number of times the same maximum mean difference value 3 is recognized is 7, so the full window rate is 100%. The calculation formula of the full window rate is shown in formula (9).
[0093] In one embodiment, step 110 includes: determining the sliding anomaly rate according to the coefficient of variation of each sliding tree from the mean difference and the full window rate of the corresponding sliding window as follows:
[0094] (10)
[0095] (11)
[0096] (12)
[0097] in, represents the sliding abnormality rate, represents the 50% coefficient of variation, represents the coefficient of variation actually calculated by the sliding window, represents the coefficient of variation of the preset variation threshold, Indicates a 50% full window rate, Indicates the set threshold full window rate, usually 100%; Indicates the actual calculated sliding window rate.
[0098] The mean difference sliding tree with a sliding anomaly rate greater than 100% is identified as an abnormal sliding tree, and the sequence point corresponding to the maximum mean difference value of the abnormal sliding tree is identified as an outlier.
[0099] Specifically, the sliding anomaly rate is an indicator that combines the coefficient of variation and the full window rate, with each accounting for 50% of the weight. The background of the design of this indicator is that the data points in the sliding window sequence cause the coefficient of variation to reach the set parameters, but do not reach the full number of sliding windows, indicating that the data points are not "abnormal" enough. Design a decisive indicator, that is, use this indicator to remove non-abnormal sliding trees.
[0100] In order to better reflect the qualitative change decision, the sliding abnormality rate is expressed in percentage form, and it is stipulated that those greater than 100% are recorded as abnormal sliding trees and abnormal values. The main calculation process is to take a 50% weight for the coefficient of variation, and then take a 50% weight for the full window rate, and finally add them together to get the sliding abnormality rate.
[0101] 1) Calculate the 50% coefficient of variation using formula (11) .
[0102] 2) Use formula (12) to calculate the 50% full window rate .
[0103] 3) Calculate the sliding abnormality rate using formula (10) .
[0104] The calculation process of sliding abnormality rate is shown in Table 4.
[0105] Table 4 Calculation process of sliding abnormality rate
[0106]
[0107] Through the above series of calculations, it is determined that the sliding abnormality rate exceeds 100% and is identified as an outlier in the sliding tree by the algorithm.
[0108] It should be understood that although Figure 1 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, Figure 1 At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.
[0109] In one embodiment, Figure 6 As shown, a time series data outlier detection device is provided, comprising: a time series data acquisition module, a mean difference sliding tree generation module, an abnormal sliding tree identification module and an adjacent outlier detection module, wherein:
[0110] The time series data acquisition module is used to obtain the time series data of commodity sales growth.
[0111] The mean difference sliding tree generation module is used to generate the minimum sliding tree and the maximum sliding tree of each sliding window according to the time series data and the preset sliding window size; according to the minimum sliding tree and the maximum sliding tree of each sliding window, the mean difference sliding tree and the maximum mean difference value of the corresponding sliding window are determined.
[0112] The abnormal sliding tree identification module is used to calculate the coefficient of variation of each mean difference sliding tree; determine the full window rate of the sliding window according to the maximum mean difference value and the preset sliding window size; determine the sliding anomaly rate according to the coefficient of variation of each mean difference sliding tree and the full window rate of the corresponding sliding window, and determine the abnormal sliding tree and the abnormal value therein according to the sliding anomaly rate.
[0113] The adjacent outlier detection module is used to merge the abnormal sliding trees into a first abnormal sliding tree when the maximum mean difference value identified within the first round of sliding window is not less than 1 time and the coefficient of variation is greater than the preset variation threshold. The outlier values in the first round of abnormal sliding tree are updated to the sliding window average value, and the second round of sliding window calculation is entered until all sequences corresponding to the first abnormal sliding trees are traversed to obtain adjacent outliers.
[0114] In one embodiment, the device also includes a tail outlier processing module for identifying tail outliers using a tail special processing method; in the tail special processing method, conventional calculations are performed from sliding window 1 to sliding window 7, regardless of whether the condition that the sliding window is equal to 7 is met, and only whether there is a computable time series value is considered.
[0115] In one embodiment, the device further includes an adjacent outlier detection module for detecting an adjacent outlier in the When at least two equal values appear in the seven sliding windows and are judged as outliers, during the first round of sliding window calculation, the maximum and minimum values of the windows are set to random values respectively. After the first round of sliding window calculation, the equal values are directly marked as outliers; k The preset sliding window size.
[0116] In one embodiment, the mean difference sliding tree generation module is further used to set the preset sliding window size to k ; Use a window size of k The window of is slid on the time series data to obtain several sliding windows; the sliding window includes k sequence points; calculate the maximum and minimum values of each sliding window; take the maximum value of each sliding window as the root node of the sequence points in the sliding window, and the sequence points in the sliding window as leaves to generate the sliding tree of the maximum value of the sliding window; take the minimum value of each sliding window as the root node of the sequence points in the sliding window, and the sequence points in the sliding window as leaves to generate the sliding tree of the minimum value of the sliding window; when the continuous k When the root nodes of the maximum sliding trees have the same value, k Maximum sliding trees are merged into a maximum sliding tree; when consecutive k When the root nodes of the minimum sliding trees have the same value,k The minimum sliding trees are merged into one minimum sliding tree.
[0117] In one embodiment, the mean difference sliding tree generation module is also used to calculate the maximum mean difference distance and the minimum mean difference distance of each sliding window. i The expressions of the maximum mean difference distance and the minimum mean difference distance of the sliding windows are shown in formulas (5) to (7).
[0118] The sequence data point corresponding to the root node of the sliding tree corresponding to the larger value of the maximum mean difference distance and the minimum mean difference distance of each sliding window is taken as the maximum mean difference value; the maximum mean difference value is taken as the root node of the maximum sliding tree and the minimum sliding tree to generate a mean difference sliding tree.
[0119] In one of the embodiments, the abnormal sliding tree identification module is further used to determine the full window rate of the sliding window according to the maximum mean difference value and the preset sliding window size as shown in formula (9).
[0120] In one embodiment, the abnormal sliding tree identification module is further used to determine the sliding anomaly rate according to the coefficient of variation of each mean difference sliding tree and the full window rate of the corresponding sliding window, and the expression of the sliding anomaly rate is shown in formula (10) to formula (12); the mean difference sliding tree with a sliding anomaly rate greater than 100% is identified as an abnormal sliding tree, and the sequence point corresponding to the maximum mean difference value of the abnormal sliding tree is identified as an outlier.
[0121] For the specific definition of the time series data anomaly detection device, please refer to the definition of the time series data anomaly detection method above, which will not be repeated here. Each module in the above-mentioned time series data anomaly detection device can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.
[0122] In one embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as follows: Figure 7As shown. The computer device includes a processor, a memory, a network interface, a display screen and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a method for detecting anomalies in time series data is implemented. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covering the display screen, or a key, trackball or touchpad set on the computer device housing, or an external keyboard, touchpad or mouse, etc.
[0123] Those skilled in the art will understand that Figure 7 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.
[0124] In one embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps in any of the above method embodiments when executing the computer program.
[0125] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in any of the above method embodiments are implemented.
[0126] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.
[0127] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.
[0128] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.
Claims
1. A method for detecting outliers in time series data, characterized in that: The method comprises: Obtain time series data on the growth of product sales; Generate a minimum sliding tree and a maximum sliding tree for each sliding window according to the time series data and the preset sliding window size; wherein the sliding tree is a tree generated by taking the maximum value of the sliding window as the root node and taking the time series data points in the sliding window as leaves; According to the minimum sliding tree and the maximum sliding tree of each sliding window, the mean difference sliding tree and the maximum mean difference value of the corresponding sliding window are determined; Calculate the coefficient of variation of each sliding tree from the mean difference; According to the maximum mean difference value and the preset sliding window size, the full window rate of the sliding window is determined as: in, represents the full window rate, Indicates the number of times the maximum mean difference of the sliding window is identified, k Indicates the preset sliding window size; Determine a sliding anomaly rate according to the coefficient of variation of each of the mean difference sliding trees and the full window rate of the corresponding sliding window, and determine an abnormal sliding tree and an abnormal value therein according to the sliding anomaly rate; When the maximum average difference value identified within the first round of sliding window is not less than 1 time, and the coefficient of variation is greater than the preset variation threshold, the abnormal sliding trees are merged into a first abnormal sliding tree, and the abnormal values in the first round of abnormal sliding tree are updated to the sliding window average value, and the second round of sliding window calculation is performed until all the sequences corresponding to the first abnormal sliding trees are traversed to obtain adjacent abnormal values; Wherein, generating a minimum sliding tree and a maximum sliding tree of each sliding window according to the time series data and the preset sliding window size includes: Set the default sliding window size to k ; The window size is k The window of is slid on the time series data to obtain several sliding windows; the sliding window includes k sequence points; Calculate the maximum and minimum values of each sliding window; The maximum value of each sliding window is used as the root node of the sequence points in the sliding window, and the sequence points in the sliding window are used as leaves to generate a sliding tree of the maximum value of the sliding window; The minimum value of each sliding window is used as the root node of the sequence points in the sliding window, and the sequence points in the sliding window are used as leaves to generate a sliding tree of the minimum value of the sliding window; When continuous k When the root nodes of the maximum sliding trees have the same value, k The maximum sliding trees are merged into one maximum sliding tree; When continuous k When the root nodes of the minimum sliding trees have the same value, k The minimum sliding trees are merged into one minimum sliding tree.
2. The method according to claim 1, characterized in that The method also includes: using a tail special processing method to identify tail outliers; wherein in the tail special processing method, conventional calculations are performed from sliding window 1 to sliding window 7, regardless of whether the condition that the sliding window is equal to 7 is met, and only whether there is a computable time series value is considered.
3. The method according to claim 1, characterized in that The method further comprises: When at least two equal values appear in the seven sliding windows and are judged as abnormal values, during the first round of sliding window calculation, the maximum and minimum values of the windows are set to random values respectively, and after the first round of sliding window calculation, the equal values are directly marked as abnormal values; k The preset sliding window size.
4. The method according to claim 1, characterized in that: According to the minimum sliding tree and the maximum sliding tree of each sliding window, the mean difference sliding tree and the maximum mean difference value of the corresponding sliding window are determined, including: Calculate the maximum mean difference distance and the minimum mean difference distance of each sliding window. i The expressions of the maximum mean difference distance and the minimum mean difference distance of the sliding windows are: in, Indicates i The maximum mean difference distance of the sliding windows, Indicates i The maximum value of sliding windows, is the sliding average of the sliding window, The sequence segment representing the sliding window range, subscript Indicates the preset sliding window size; represents the minimum mean difference distance, Indicates the minimum value of the sliding window; The sequence data point corresponding to the root node of the sliding tree corresponding to the larger value of the maximum mean difference distance and the minimum mean difference distance of each sliding window is taken as the maximum mean difference value; The maximum mean difference value is used as the root node of the maximum sliding tree and the minimum sliding tree to generate a mean difference sliding tree.
5. The method according to claim 1, characterized in that: Determining a sliding anomaly rate according to the coefficient of variation of each of the mean difference sliding trees and the full window rate of the corresponding sliding window, and determining an abnormal sliding tree and an abnormal value therein according to the sliding anomaly rate, including: According to the coefficient of variation of each of the mean difference sliding trees and the full window rate of the corresponding sliding window, the sliding anomaly rate is determined as: in, represents the sliding abnormality rate, represents the 50% coefficient of variation, represents the coefficient of variation actually calculated by the sliding window, represents the coefficient of variation of the preset variation threshold, Indicates a 50% full window rate, Indicates the set threshold full window rate, the value is 100%; Indicates the actual calculated sliding window rate; The mean difference sliding tree with a sliding abnormality rate greater than 100% is identified as an abnormal sliding tree, and the sequence point corresponding to the maximum mean difference value of the abnormal sliding tree is identified as an abnormal value.
6. A time series data abnormal value detection device, characterized in that: The device comprises: Time series data acquisition module, used to obtain time series data of commodity sales growth; The module for generating a sliding tree of deviation from the mean is used to generate a minimum sliding tree and a maximum sliding tree of each sliding window according to the time series data and a preset sliding window size; and to determine a sliding tree of deviation from the mean and a maximum mean difference value of the corresponding sliding window according to the minimum sliding tree and the maximum sliding tree of each sliding window; wherein the sliding tree is a tree generated by taking the maximum value of the sliding window as the root node and taking the time series data points in the sliding window as leaves; The abnormal sliding tree identification module is used to calculate the coefficient of variation of each of the mean difference sliding trees; determine the full window rate of the sliding window according to the maximum mean difference value and the preset sliding window size; determine the sliding abnormality rate according to the coefficient of variation of each of the mean difference sliding trees and the full window rate of the corresponding sliding window, and determine the abnormal sliding tree and the abnormal value therein according to the sliding abnormality rate; wherein the full window rate of the sliding window is: in, represents the full window rate, Indicates the number of times the maximum mean difference of the sliding window is identified, k Indicates the preset sliding window size; The adjacent outlier detection module is used to merge the abnormal sliding trees into a first abnormal sliding tree when the maximum mean difference value identified within the first round of sliding window is not less than 1 time and the coefficient of variation is greater than the preset variation threshold, update the outlier value in the first round of abnormal sliding tree to the sliding window average value, and enter the second round of sliding window calculation until all sequences corresponding to the first abnormal sliding trees are traversed to obtain adjacent outliers; Among them, the mean difference sliding tree generation module is also used to set the preset sliding window size to k ; Use a window size of k The window of is slid on the time series data to obtain several sliding windows; the sliding window includes k sequence points; calculate the maximum and minimum values of each sliding window; take the maximum value of each sliding window as the root node of the sequence points in the sliding window, and the sequence points in the sliding window as leaves to generate the sliding tree of the maximum value of the sliding window; take the minimum value of each sliding window as the root node of the sequence points in the sliding window, and the sequence points in the sliding window as leaves to generate the sliding tree of the minimum value of the sliding window; when the continuous k When the root nodes of the maximum sliding trees have the same value, k Maximum sliding trees are merged into a maximum sliding tree; when consecutive k When the root nodes of the minimum sliding trees have the same value, k The minimum sliding trees are merged into one minimum sliding tree.
7. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, the method according to any one of claims 1 to 5 is implemented.
8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 5 is implemented.
