Data anomaly detection method and device, equipment and storage medium

Abstract

The invention discloses a data anomaly detection method and device, equipment and a storage medium. The method comprises the steps of obtaining data corresponding to a query condition; calculating a maximum value and a minimum value based on the data corresponding to the query condition; based on a median between the maximum value and the minimum value, the maximum value, the minimum value, a preset maximum tolerance and a numerical value interval, calculating to obtain an upper limit value and a lower limit value of the median; and determining the data not in the range of the upper limit value and the lower limit value as abnormal data. According to the invention, on the basis of operation and maintenance data exception judgment based on the dynamic threshold value or the dynamic segmentation threshold value, the design of the exception value tolerance is carried out, the designed exception value judgment is more in line with the operation and maintenance requirements, the frequency of invalid exception data alarm is reduced, the accuracy of exception data judgment is improved, and the user experience is improved. Therefore, the working efficiency of the operation and maintenance personnel is improved, and the technical problem that the accuracy of the judgment result of the abnormal data is low in the related technology is solved.

Description

Technical Field

[0001] The present invention relates to the field of data monitoring technology, and in particular to a data anomaly detection method, device, equipment and storage medium. Background Art

[0002] At present, a single server is decomposed into several services according to the design requirements of microservices. However, whether it is a traditional large-scale single server or a more fine-grained microservice, it is not a wise decision to use a unified threshold to define the health of a machine or service. If the value is set too large, no alarm or exception will be generated; if the value is set too small, alarms will be frequent. In either case, the operation and maintenance personnel will lose sensitivity to perceived failures or exceptions and lose their perception of the health of the business.

[0003] In related technologies, an automatic threshold algorithm is designed in the operation and maintenance link to judge the anomaly of a single indicator, so as to solve various problems such as manual setting of thresholds, maintenance of thresholds, and inaccurate threshold configuration by operation and maintenance personnel without data accumulation. However, at the same time, the setting of thresholds based on algorithms also brings new challenges to operation and maintenance:

[0004] 1. Automatic thresholds usually construct a probability distribution model of data extreme values ​​based on extreme value theory, and judge anomalies by checking whether one or more objects obey the distribution. Although the automatic threshold algorithm has reduced its reliance on the original data distribution model through extreme values, it is still based on historical data and specific distributions. Therefore, although the automatic threshold algorithm can adapt to a variety of distributions, it cannot cover all of them. In particular, when the data distribution changes due to changes in business behavior, the accuracy of the judgment results of the automatic threshold algorithm for abnormal data cannot be guaranteed;

[0005] 2. Since the various performance indicators and business characteristic indicators involved in operation and maintenance have different operation and maintenance requirements in different value areas, the use of automatic threshold algorithms in actual operation and maintenance scenarios will result in over-matching, which will reduce the accuracy of abnormal data judgment results. This will bring a large number of unnecessary intervention and processing scenarios to operation and maintenance personnel, greatly affecting the operation and maintenance effect. Summary of the invention

[0006] The present invention provides a data anomaly detection method, device, equipment and storage medium, which can solve the technical problem of low accuracy of abnormal data judgment results existing in the prior art.

[0007] To achieve the above object, the present invention provides the following technical solutions:

[0008] In a first aspect, an embodiment of the present invention provides a data anomaly detection method, the method comprising:

[0009] Get the data corresponding to the query conditions;

[0010] Calculate the maximum and minimum values ​​based on the data corresponding to the query conditions;

[0011] An upper limit value and a lower limit value are calculated based on the median between the maximum value and the minimum value, the maximum value, the minimum value, a preset maximum tolerance, and a numerical range;

[0012] The data not falling within the range of the upper limit value and the lower limit value is determined to be abnormal data.

[0013] In a second aspect, an embodiment of the present invention provides a data anomaly detection device, the device comprising:

[0014] An information acquisition module is configured to acquire data corresponding to a query condition;

[0015] A first calculation module is configured to calculate a maximum value and a minimum value based on data corresponding to the query condition;

[0016] A second calculation module is configured to calculate an upper limit value and a lower limit value based on a median between the maximum value and the minimum value, the maximum value, the minimum value, a preset maximum tolerance, and a numerical range;

[0017] The information determination module is configured to determine that data not within the range of the upper limit value and the lower limit value is abnormal data.

[0018] In a third aspect, an embodiment of the present invention further provides an electronic device, comprising: a memory and a processor; the processor is used to read and execute a computer program stored in the memory to implement the steps of the aforementioned data anomaly detection method.

[0019] In a fourth aspect, an embodiment of the present invention further provides a computer storage medium, wherein the computer storage medium stores computer executable instructions, and the computer executable instructions, when executed, implement the steps of the aforementioned data anomaly detection method.

[0020] In a fifth aspect, an embodiment of the present invention further provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the steps of the aforementioned data anomaly detection method.

[0021] The beneficial effects brought by the technical solution provided by the embodiment of the present invention include:

[0022] Acquire the data corresponding to the query conditions; calculate the maximum and minimum values ​​based on the data corresponding to the query conditions; calculate the upper limit and lower limit of the median based on the median between the maximum and minimum values, the maximum, the minimum, the preset maximum tolerance and the numerical interval; determine that the data not within the range of the upper limit and the lower limit are abnormal data. Through the present invention, on the basis of judging the abnormality of operation and maintenance data by dynamic threshold or dynamic segmentation threshold, the design of abnormal value tolerance is carried out. The abnormal value judgment after the design is more in line with the requirements of operation and maintenance, reduces the frequency of invalid abnormal data alarms, improves the accuracy of judging abnormal data, thereby improving the work efficiency of operation and maintenance personnel, and solves the technical problem of low accuracy of abnormal data judgment results in related technologies. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0024] Figure 1 This is a flow chart of a first embodiment of a method for detecting data anomalies according to the present invention;

[0025] Figure 2 It is a schematic diagram of the overall structure of the present invention;

[0026] Figure 3 for Figure 1 Detailed flow chart of step S30;

[0027] Figure 4 This is a flow chart of a second embodiment of a method for detecting data anomalies according to the present invention;

[0028] Figure 5 This is a schematic diagram of functional modules of an embodiment of a data anomaly detection device of the present invention;

[0029] Figure 6 The figure is a schematic diagram of the structure of an electronic device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0030] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0031] To make the objectives, technical solutions and advantages of the present invention more clear, the embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0032] In a first aspect, an embodiment of the present invention provides a method for detecting anomalies in data.

[0033] In one embodiment, referring to Figure 1 , Figure 1 FIG. 1 is a flow chart of the first embodiment of the data anomaly detection method of the present invention. Figure 1 As shown, the data anomaly detection method includes:

[0034] Step S10, obtaining data corresponding to the query condition;

[0035] In some specific embodiments, step S10 includes:

[0036] Detect whether data corresponding to the query condition is stored in the local storage device;

[0037] If data corresponding to the query condition is stored in the local storage device, the data corresponding to the query condition is read from the local storage device;

[0038] If the data corresponding to the query condition is not stored in the local storage device, the data corresponding to the query condition is read from the data warehouse.

[0039] In this embodiment, refer to Figure 2 , Figure 2 This is a schematic diagram of the overall structure of the present invention. Figure 2 As shown, the alarm rules set when the user accesses the interface are managed through the front-end interface included in the rule management module, including the addition, deletion, modification, query, opening and stopping of the alarm rules. Different service interfaces are called through the back-end interface included in the rule management module to receive all requests initiated by the front-end interface. After receiving the request, the operation initiated by the user is executed and recorded, and the result of the operation is returned to the front-end interface.

[0040] The rule management module designs a set of data structures for various subdivided scenarios of indicator alarms. The data structures can be divided into four categories based on their functional types:

[0041] Basic attributes: This type of configuration describes the attributes of the rule itself, such as the rule ID, name, description, status, type, etc.

[0042] Data selection: This type of configuration describes the input information of the rule, for example, the name of the indicator data table in the data warehouse, the name and type of the indicator to be aggregated, data screening conditions, etc.

[0043] Alarm configuration, which describes the required alarm information in the alarm rule. For example, the detection cycle of the alarm rule, the data statistics cycle, the algorithm and threshold information related to anomaly detection, whether the intelligent algorithm results are substituted into the tolerance calculation model, the distribution of abnormal data points, and the alarm channel information.

[0044] Advanced alarm configuration, which describes the optional alarm information in the alarm rule, such as the delay calculation time of the alarm rule, the monitoring period, the silent time of the alarm, etc.

[0045] The task scheduling module is a single backend service that is responsible for scheduling all scheduled tasks in the system and tracking the execution results of each task. It mainly includes two types of scheduled tasks:

[0046] Intelligent algorithm task: This task is responsible for triggering the offline calculation of the intelligent algorithm. The backend service will periodically read all the alarm rules in the system with the status information "on", and then cyclically parse the data selection and alarm configuration in each alarm rule. After the alarm rules are formatted and supplemented with default data, they will be placed in the intelligent algorithm calculation queue for the intelligent algorithm module to obtain.

[0047] Indicator abnormality alarm task: This task is responsible for triggering abnormality detection of alarm rules. The backend service will periodically read all alarm rules in the system whose status information is "on", and then cyclically parse the alarm configuration in each alarm rule. The program will comprehensively judge whether to trigger abnormality detection based on the detection cycle in the alarm configuration and the time when the alarm rule detection was last executed. If the triggering conditions are met, the alarm rule will be placed in the abnormality detection queue for consumption by the abnormality detection module. For example, the execution cycle of the indicator abnormality alarm task is one minute, and the detection cycle of a certain alarm rule is five minutes, then the backend service will only send one alarm rule to the abnormality detection queue within five minutes.

[0048] The intelligent algorithm module is a single backend service responsible for implementing the offline computing function of the intelligent algorithm.

[0049] When the program starts, the backend service will start multiple threads to continuously obtain the alarm rules in the intelligent algorithm queue. Every time an alarm rule is obtained, the intelligent algorithm offline calculation process is executed according to the configuration of the alarm rule.

[0050] Specifically, taking the local storage device as the local hard disk as an example, the offline calculation process of the intelligent algorithm first parses the data selection information in the alarm rule, obtains the alarm indicator information, the indicator information that needs to be aggregated, the data screening conditions, etc., and then parses the alarm configuration information in the alarm rule to obtain the variables of the intelligent algorithm, the time range of historical data, etc., and forms the query conditions for querying historical data from the data warehouse based on the above data.

[0051] In order to reduce the pressure on the data warehouse and avoid frequent and large-scale data queries, the program will first check whether the local hard disk has cached data corresponding to the query conditions obtained last time on a daily basis. If so, the process of querying the data warehouse is skipped and the data corresponding to the query conditions is read from the local hard disk. If not, the data warehouse is queried and the data corresponding to the query conditions is read from the data warehouse. Furthermore, after reading the data corresponding to the query conditions from the data warehouse, the data corresponding to the query conditions is written to the local hard disk so that the data corresponding to the query conditions can be directly read from the local hard disk later.

[0052] In another embodiment, if the local hard disk has not cached the data corresponding to the query conditions obtained last time, the data warehouse is queried based on the query conditions, and the data corresponding to the query conditions are written to the local hard disk. Finally, all data corresponding to the query conditions are read from the local hard disk.

[0053] For example, when executing the offline calculation process of the intelligent algorithm, the time range of historical data required by the intelligent algorithm is 30 days. The back-end service will query the local hard disk whether there is data corresponding to the query conditions in the past 30 days. The data for each day is a file. If it is missing, it will be queried again from the data warehouse and written to the hard disk. After the files are collected, the back-end service will read all the data corresponding to the query conditions within 30 days at one time.

[0054] Step S20, calculating the maximum value and the minimum value based on the data corresponding to the query condition;

[0055] In some specific embodiments, step S20 includes:

[0056] Based on the data corresponding to the query conditions obtained n times, n extreme values ​​are calculated by using a univariate time series streaming anomaly detection algorithm based on extreme value theory;

[0057] Determine the maximum and minimum values ​​from n extreme values.

[0058] In this embodiment, each time the data corresponding to the query condition is obtained, the data corresponding to the query condition is merged into a string of data and then calculated by an intelligent algorithm to obtain an extreme value. In this embodiment, the intelligent algorithm is the SPOT algorithm (Streaming Peaks-Over-Threshold, a univariate time series streaming anomaly detection algorithm based on extreme value theory). The univariate time series streaming anomaly detection algorithm (SPOT algorithm) based on extreme value theory can be used for univariate streaming data anomaly detection, but is more commonly used to automatically set thresholds (such as OmniAnomaly uses this algorithm to set thresholds for anomaly scores). The algorithm calculates the extreme value under a specified probability by fitting the extreme value distribution, and the extreme value is considered to be the threshold for determining anomalies.

[0059] Based on the data corresponding to the query conditions obtained n times, n extreme values ​​can be calculated through the univariate time series streaming anomaly detection algorithm (SPOT algorithm) based on extreme value theory. Then, the maximum and minimum values ​​are selected from the n extreme values ​​for subsequent use to solve the problem that the algorithm prediction results do not meet the alarm processing principles due to insufficient historical data points, mismatch between the algorithm model and the actual data model, or overfitting.

[0060] Step S30, calculating an upper limit value and a lower limit value based on the median between the maximum value and the minimum value, the maximum value, the minimum value, a preset maximum tolerance and a numerical range;

[0061] In some specific embodiments, referring to Figure 3 , Figure 3 for Figure 1 Detailed flow chart of step S30 in FIG. Figure 3 As shown, step S30 includes:

[0062] Step S301, calculating a first upper limit value and a first lower limit value based on the median between the maximum value and the minimum value and a preset maximum tolerance;

[0063] Step S302, calculating a second upper limit value and a second lower limit value based on the median between the maximum value and the minimum value, the maximum value, the minimum value and the numerical interval;

[0064] Step S303, taking the maximum value of the first upper limit value and the second upper limit value as the final upper limit value;

[0065] Step S304: taking the minimum value between the first lower limit value and the second lower limit value as the final lower limit value.

[0066] In this embodiment, the median between the maximum value and the minimum value is multiplied by the sum of the preset maximum tolerance plus one to obtain the product as the first upper limit value of the median; the median between the maximum value and the minimum value is multiplied by the difference between one and the preset maximum tolerance to obtain the product as the first lower limit value.

[0067] Specifically, taking the maximum value Pmax=80, the minimum value Pmin=20, and the preset maximum tolerance mtol=70% as an example, the median pmedian between the maximum value Pmax=80 and the minimum value Pmin=20 is 60, and the first upper limit value maxM=pmedian*(1+mtol)=60*(1+70%)=102. The first lower limit value minM=pmedian*(1-mtol)=60*(1-70%)=18.

[0068] Determine a target numerical interval, wherein the target numerical interval is the numerical interval to which the median between the maximum value and the minimum value belongs; obtain a tolerance value corresponding to the target numerical interval; multiply the maximum value by the sum of the tolerance value plus one to obtain the product as the second upper limit value of the median; multiply the minimum value by one minus the tolerance value to obtain the product as the second lower limit value. The numerical interval is determined based on the floating interval of the maximum value, the minimum value, and the preset median.

[0069] For example, if the preset floating interval of the median between the maximum value and the minimum value is 50%, the numerical intervals are determined to be [-∞, 50) and [50, +∞]. The first numerical interval [-∞, 50) is determined as follows:

[0070] (pmax-pmin)*(part1-0)+pmin=(80-20)*(0.5-0)+20=50 The second numerical interval [50, +∞] is determined by:

[0071] pmax-(pmax-pmin)*(1-part n )=80-(80-20)*(1-0.5)=50

[0072] It is easy to think that, because there is only one floating interval, part1 is partn, that is, n=1.

[0073] For another example, if the preset floating intervals of the median between the maximum value and the minimum value are 0-40%, 40%-60%, 60%-80% and 80%-100%, there are three floating intervals, so part1=40%, part2=60%, part3=80%, and part3 is partn, that is, n=3. Then the determined numerical intervals are [-∞, 44), [44, 68), [56, 68) and [68, +∞].

[0074] Among them, the first numerical interval is [-∞, 44) and the determination process is:

[0075] (pmax-pmin)*(part1-0)+pmin=(80-20)*0.4+20=44

[0076] The second numerical interval is [44, 56) and the determination process is:

[0077] [(pmax-pmin)*part i-1 +pmin, (pmax-pmin)*part i+pmin)=[(80-20)*0.4+20,(80-20)*0.6+20)=[44,56), where i=2~n, part i-1 =part1,part i =part2.

[0078] The third numerical interval is [56, 68) and the determination process is:

[0079] [(pmax-pmin)*part i-1 +pmin, (pmax-pmin)*part i +pmin)

[0080] =[(80-20)*0.6+20, (80-20)*0.8+20)=[56,68), where i=2~n, part i-1 =part2, part i =part3.

[0081] The fourth numerical interval is [68, +∞] and the determination process is:

[0082] pmax-(pmax-pmin)*(1-part n )=80-(80-20)*(1-0.8)=68

[0083] The numerical interval to which the median pmedian=60 between the maximum value and the minimum value belongs is the third numerical interval, and the target numerical interval is determined to be the third numerical interval.

[0084] Obtain the tolerance value corresponding to the third numerical interval. If the tolerance value corresponding to the third numerical interval is tol=20%, then the second upper limit maxT=pmax*(1+tol)=80*(1+20%)=96. The second lower limit minT=pmin*(1-tol)=20*(1-20%)=16.

[0085] The maximum value between the first upper limit value and the second upper limit value is taken as the final upper limit value, that is, the final upper limit value ThresUp=max(maxM, maxT)=102.

[0086] The minimum value between the first lower limit value and the second lower limit value is taken as the final lower limit value, that is, the final lower limit value ThresDown=min(minM, minT)=16.

[0087] After obtaining the final upper and lower limits THresUp and ThresDown, the final upper and lower limits and the execution record of the intelligent algorithm will be stored in the database. It should be noted that the parameters in this embodiment are for reference only and are not limited here.

[0088] Step S40: determining that data not within the range of the upper limit value and the lower limit value is abnormal data.

[0089] In this embodiment, the range formed by the upper limit value and the lower limit value is [102, 16]. Figure 2 , ,The anomaly detection module is a single back-end service that is ,mainly responsible for implementing the anomaly detection function of ,alarm rules.

[0090] When the program starts, the background service starts multiple threads to continuously obtain the alarm rules in the anomaly detection queue. When an alarm rule is obtained, the anomaly detection process will be executed according to the configuration of the alarm rule.

[0091] Specifically, the anomaly detection process first parses the alarm configuration information and advanced alarm configuration information in the alarm rule, obtains the detection cycle and monitoring period, and then determines whether the current anomaly detection task meets the detection cycle and the current time is within the monitoring period based on the time of the last anomaly detection. If all meet, the data selection information in the alarm rule is parsed to obtain the alarm indicator information, the indicator information that needs to be aggregated, and the data screening conditions, generate the query conditions for querying the data warehouse and query the real-time data from the data warehouse; secondly, after querying the database for the execution record of the last intelligent algorithm, query the database for the range composed of the upper and lower limits calculated by the most recent intelligent algorithm module; finally, compare the real-time data with the range composed of the upper and lower limits, and determine that the data not within the range composed of the upper and lower limits is abnormal data.

[0092] It is easy to think that the upper limit value and the lower limit value are the upper limit value and the lower limit value used to detect whether the data is within the normal data range.

[0093] In this embodiment, data corresponding to the query condition is obtained; the maximum value and the minimum value are calculated based on the data corresponding to the query condition; the upper limit and the lower limit of the median are calculated based on the median between the maximum value and the minimum value, the maximum value, the minimum value, the preset maximum tolerance and the numerical interval; the data not within the range of the upper limit and the lower limit are determined to be abnormal data. Through this embodiment, on the basis of judging the abnormality of operation and maintenance data by dynamic threshold or dynamic segmentation threshold, the tolerance of abnormal value is designed. The abnormal value judgment after the design is more in line with the requirements of operation and maintenance, reduces the frequency of invalid abnormal data alarms, improves the accuracy of judging abnormal data, thereby improving the work efficiency of operation and maintenance personnel, and solves the technical problem of low accuracy of abnormal data judgment results in related technologies.

[0094] Optionally, in one embodiment, referring to Figure 4 , Figure 4FIG. 2 is a flow chart of the second embodiment of the data anomaly detection method of the present invention. Figure 4 As shown, after step S40, the following steps are included:

[0095] Step S50, detecting whether the abnormal data meets the data abnormality alarm triggering condition;

[0096] Step S60: If the conditions are met, an alarm is generated according to a preset alarm strategy.

[0097] In this embodiment, after determining the abnormal data, it is judged whether the abnormal data queried this time meets the data abnormal alarm triggering condition according to the data abnormal alarm triggering condition in the alarm configuration information. If it does, the alarm rules and abnormal data are packaged and sent to the alarm processing queue.

[0098] For example, the detection time configured in the alarm rule is 2 minutes, the monitoring period is 8:00-22:00, and the data anomaly alarm trigger condition is that at least 2 data points are abnormal to trigger the alarm. First, only when the detection task has not been triggered in the last two minutes and the current time is between 8:00-22:00, the task will continue the subsequent process; then the program finds the real-time data from the data warehouse as 10, 13, 14, 15, 12, 10, 11 respectively. If the upper and lower limits calculated by the intelligent algorithm are queried from the database and the range is [14, 9], it can be seen that only data point 15 is an abnormal point. Since there is only one abnormal point, the data anomaly alarm trigger condition is not met at the current time, and no alarm prompt is generated. If the upper and lower limits calculated by the intelligent algorithm are queried from the database and the range is [13, 9], then there are two data points 14 and 15 as abnormal points, which meet the data anomaly alarm trigger condition, and an alarm prompt is generated according to the preset alarm strategy.

[0099] Specifically, continue to refer to Figure 2 ,Alarm processing module This module is a single backend service, responsible for pushing abnormal alarm rules to different alarm channels. When the program starts, the backend service will start multiple threads to process the alarm processing queue in parallel. Every time an alarm rule is obtained, the alarm processing process is executed according to the configuration of the alarm rule.

[0100] The alarm processing flow first parses the advanced alarm configuration information in the alarm rules, obtains the silent time configuration of the alarm, and then determines whether it is necessary to send this alarm based on the time of the last alarm and the silent time configuration. If necessary, parse the alarm configuration information in the alarm rules, obtain the alarm channel information and alarm channel configuration information, splice the alarm rules and abnormal data into alarm information that meets the channel specifications, and send the alarm information to the selected alarm channel to generate an alarm prompt.

[0101] For example, the silent time configured in the alarm rule is 10 minutes, and the alarm channels are email and SMS. Every time the backend service receives an alarm rule, it needs to check whether the same alarm has been sent in the previous 10 minutes. If it has been sent, it will not send the alarm again; because the alarm channels are SMS and email, the backend service will splice the alarm rule and abnormal information into the content of the SMS, the subject and body of the email according to the template, and then send the SMS and email to the recipient respectively.

[0102] In a second aspect, an embodiment of the present invention further provides a data anomaly detection device.

[0103] In one embodiment, referring to Figure 5 , Figure 5 Schematic diagram of the functional modules of an embodiment of the data anomaly detection device of the present invention. Figure 5 As shown, the data anomaly detection device includes:

[0104] The information acquisition module 10 is configured to acquire data corresponding to the query condition;

[0105] A first calculation module 20 is configured to calculate a maximum value and a minimum value based on data corresponding to the query condition;

[0106] A second calculation module 30 is configured to calculate an upper limit value and a lower limit value based on a median between the maximum value and the minimum value, the maximum value, the minimum value, a preset maximum tolerance, and a numerical range;

[0107] The information determination module 40 is configured to determine that data not within the range of the upper limit value and the lower limit value is abnormal data.

[0108] Optionally, in one embodiment, the information acquisition module 10 is configured to:

[0109] Detect whether data corresponding to the query condition is stored in the local storage device;

[0110] If data corresponding to the query condition is stored in the local storage device, the data corresponding to the query condition is read from the local storage device;

[0111] If the data corresponding to the query condition is not stored in the local storage device, the data corresponding to the query condition is read from the data warehouse.

[0112] Optionally, in one embodiment, the first calculation module 20 is configured to:

[0113] Based on the data corresponding to the query conditions obtained n times, n extreme values ​​are calculated by using a univariate time series streaming anomaly detection algorithm based on extreme value theory;

[0114] Determine the maximum and minimum values ​​from n extreme values.

[0115] Optionally, in one embodiment, the second calculation module 30 is configured to:

[0116] A first upper limit value and a first lower limit value are calculated based on the median between the maximum value and the minimum value and a preset maximum tolerance;

[0117] Based on the median between the maximum value and the minimum value, the maximum value, the minimum value, and the numerical interval, a second upper limit value and a second lower limit value are calculated;

[0118] taking the maximum value of the first upper limit value and the second upper limit value as the final upper limit value;

[0119] The minimum value between the first lower limit value and the second lower limit value is taken as the final lower limit value.

[0120] Optionally, in one embodiment, the second calculation module 30 is configured to:

[0121] The median between the maximum value and the minimum value is multiplied by the sum of a preset maximum tolerance plus one, so as to obtain a product as a first upper limit value of the median;

[0122] The median between the maximum value and the minimum value is multiplied by one minus a difference of a preset maximum tolerance to obtain a product as a first lower limit value of the median.

[0123] Optionally, in one embodiment, the second calculation module 30 is configured to:

[0124] Determine a target numerical interval, wherein the target numerical interval is a numerical interval to which the median between the maximum value and the minimum value belongs;

[0125] Obtaining a tolerance value corresponding to the target numerical range;

[0126] multiplying the maximum value by the sum of the tolerance value plus one to obtain a product as a second upper limit value of the median;

[0127] The minimum value is multiplied by one minus the tolerance value to obtain a product as the second lower limit value of the median.

[0128] Optionally, in one embodiment, the data anomaly detection device further includes an alarm module configured to:

[0129] Detecting whether the abnormal data meets the data abnormality alarm triggering condition;

[0130] If the conditions are met, an alarm will be generated according to the preset alarm strategy.

[0131] Among them, the functional implementation of each module in the above-mentioned data anomaly detection device corresponds to the steps in the above-mentioned data anomaly detection method embodiment, and its functions and implementation processes are no longer repeated here.

[0132] In a third aspect, an embodiment of the present invention further provides an electronic device, whose structure is as follows: Figure 6 As shown, it includes: a memory and a processor, and the processor is used to read and execute the computer program stored in the memory to implement the aforementioned data anomaly detection method.

[0133] In a fourth aspect, an embodiment of the present invention further provides a computer storage medium, wherein the computer storage medium stores computer executable instructions, and the computer executable instructions implement the aforementioned data anomaly detection method when executed.

[0134] In a fifth aspect, an embodiment of the present invention provides a computer program product, which is stored in a storage medium and is executed by at least one processor to implement the various processes of the above-mentioned data anomaly detection method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0135] Finally, it should be noted that: in some processes described in the embodiments of the present invention, multiple operations or steps appearing in a specific order are included, but it should be understood that these operations or steps may not be executed in the order in which they appear in the embodiments of the present invention or may be executed in parallel, and the sequence number of the operation is only used to distinguish between different operations, and the sequence number itself does not represent any execution order. In addition, these processes may include more or fewer operations, and these operations or steps may be executed in sequence or in parallel, and these operations or steps may be combined.

[0136] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the aforementioned embodiments, it is still possible for those skilled in the art to modify the technical solutions described in the aforementioned embodiments, or to make equivalent substitutions for some of the technical features therein. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A data anomaly detection method, characterized in that: The method comprises: Get the data corresponding to the query conditions; Calculate the maximum and minimum values ​​based on the data corresponding to the query conditions; An upper limit value and a lower limit value are calculated based on the median between the maximum value and the minimum value, the maximum value, the minimum value, a preset maximum tolerance, and a numerical range; The data not falling within the range of the upper limit value and the lower limit value is determined to be abnormal data.

2. The data anomaly detection method according to claim 1, characterized in that: The step of obtaining data corresponding to the query condition includes: Detect whether data corresponding to the query condition is stored in the local storage device; If data corresponding to the query condition is stored in the local storage device, the data corresponding to the query condition is read from the local storage device; If the data corresponding to the query condition is not stored in the local storage device, the data corresponding to the query condition is read from the data warehouse.

3. The data anomaly detection method according to claim 1, characterized in that: The step of calculating the maximum value and the minimum value based on the data corresponding to the query condition includes: Based on the data corresponding to the query conditions obtained n times, n extreme values ​​are calculated by using a univariate time series streaming anomaly detection algorithm based on extreme value theory; Determine the maximum and minimum values ​​from n extreme values.

4. The data anomaly detection method according to claim 1, characterized in that: The step of calculating the upper limit value and the lower limit value based on the median between the maximum value and the minimum value, the maximum value, the minimum value, a preset maximum tolerance and a numerical range comprises: A first upper limit value and a first lower limit value are calculated based on the median between the maximum value and the minimum value and a preset maximum tolerance; Based on the median between the maximum value and the minimum value, the maximum value, the minimum value, and the numerical interval, a second upper limit value and a second lower limit value are calculated; taking the maximum value of the first upper limit value and the second upper limit value as the final upper limit value; The minimum value between the first lower limit value and the second lower limit value is taken as the final lower limit value.

5. The data anomaly detection method according to claim 4, characterized in that: The step of calculating the first upper limit value and the first lower limit value based on the median between the maximum value and the minimum value and a preset maximum tolerance comprises: The median between the maximum value and the minimum value is multiplied by the sum of a preset maximum tolerance plus one, so as to obtain a product as a first upper limit value of the median; The median between the maximum value and the minimum value is multiplied by one minus a difference of a preset maximum tolerance to obtain a product as a first lower limit value of the median.

6. The data anomaly detection method according to claim 4, characterized in that: The step of calculating the second upper limit value and the second lower limit value based on the median between the maximum value and the minimum value, the maximum value, the minimum value and the numerical interval comprises: Determine a target numerical interval, wherein the target numerical interval is a numerical interval to which the median between the maximum value and the minimum value belongs; Obtaining a tolerance value corresponding to the target numerical range; multiplying the maximum value by the sum of the tolerance value plus one to obtain a product as a second upper limit value of the median; The minimum value is multiplied by one minus the tolerance value to obtain a product as the second lower limit value of the median.

7. The data anomaly detection method according to claim 1, characterized in that: After the step of determining that the data not within the range of the upper limit value and the lower limit value is abnormal data, the method further comprises: Detecting whether the abnormal data meets the data abnormality alarm triggering condition; If the conditions are met, an alarm will be generated according to the preset alarm strategy.

8. A data anomaly detection device, characterized in that: The device comprises: An information acquisition module is configured to acquire data corresponding to a query condition; A first calculation module is configured to calculate a maximum value and a minimum value based on data corresponding to the query condition; A second calculation module is configured to calculate an upper limit value and a lower limit value based on a median between the maximum value and the minimum value, the maximum value, the minimum value, a preset maximum tolerance, and a numerical range; The information determination module is configured to determine that data not within the range of the upper limit value and the lower limit value is abnormal data.

9. An electronic device, characterized in that: include: Memory and processor; The processor is used to read and execute the computer program stored in the memory to implement the steps of the data anomaly detection method as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, and when the computer-executable instructions are executed, the steps of the data anomaly detection method as described in any one of claims 1 to 7 are implemented.

11. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instruction is executed by a processor, the steps of the data anomaly detection method as described in any one of claims 1-7 are implemented.

Images