Data processing method and device, equipment and storage medium
By obtaining historical business data, extracting data characteristics and determining monitoring models and early warning parameters, the problem of manual configuration thresholds in the existing technology is solved, and automated data monitoring and early warning is realized, adapting to different data distributions and improving Monitoring accuracy and efficiency.
Patent Information
- Application Number
- CN202310639114.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-05-31
- Publication Date
- 2024-12-03
AI Technical Summary
In the data monitoring, the prior art has problems such as manual threshold configuring is large and the threshold setting requires long-term labor optimization, resulting in poor data monitoring effect and it is difficult to meet the monitoring needs of different business data.
By obtaining historical business data, extracting data characteristics, determining data monitoring models and early warning parameters, using these parameters to perform abnormal detection, dynamically adjusting the monitoring model and threshold range, and achieving automated monitoring and early warning.
While saving manpower, the accuracy and efficiency of data monitoring are improved, different data distributions are adapted to different data, the need for manual parameter setting is reduced, and more accurate data monitoring and early warning is achieved.
Smart Images

Figure CN119066828A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of artificial intelligence, and in particular to a data processing method, apparatus, device and storage medium. Background Art
[0002] As Internet technology (IT) systems become increasingly complex and their stability and reliability have an increasingly greater impact on corporate business and even social life, more efficient and reliable operation and maintenance tools are needed to ensure the stability and security of the system.
[0003] As the business continues to expand, data monitoring capabilities face great challenges: different business data requires different monitoring indicators and monitoring dimensions. At the same time, different business data has different data characteristics, which leads to problems such as poor results of traditional monitoring methods, huge workload of manually configuring thresholds, and long-term human optimization required to set thresholds.
[0004] Therefore, there is an urgent need to provide a solution that can improve data monitoring effects while saving manpower. Summary of the invention
[0005] The embodiments of the present application provide a data processing method, apparatus, device, and storage medium for realizing accurate monitoring and early warning of data while saving manpower.
[0006] In view of this, on the one hand, the present application provides a data processing method, including: obtaining historical business data; extracting data features corresponding to the historical business data, the data features are used to indicate the data distribution of the historical business data; determining a data monitoring model corresponding to the historical business data based on the data features; using the historical business data to train to obtain model parameters and early warning parameters of the data monitoring model, the early warning parameters are evaluation thresholds for determining abnormalities in business data, and the model parameters include but are not limited to model types and monitoring methods; and using the model parameters and the early warning parameters to perform anomaly detection on the business data.
[0007] On the other hand, the present application provides a data processing device, including: an acquisition module, for acquiring historical business data;
[0008] A processing module, used to extract data features corresponding to the historical business data, where the data features are used to indicate data distribution of the historical business data; and determine a data monitoring model corresponding to the historical business data according to the data features;
[0009] A training module, used to train the historical business data to obtain model parameters and warning parameters of the data monitoring model, wherein the warning parameter is an evaluation threshold for determining abnormalities in the business data, and the model parameter includes but is not limited to a model type and a monitoring method;
[0010] The processing module is used to perform anomaly detection on the business data using the model parameters and the warning parameters.
[0011] In one possible design, in another implementation of another aspect of the embodiment of the present application, the data characteristics of the historical business data include at least pulse data, sawtooth data, continuous fluctuation data and continuous stable data, and the processing module is specifically used to determine that the data monitoring model is a single-point threshold comparison model when the data characteristics of the historical business data are pulse data, the early warning parameter is the overall threshold range of the pulse data, and the monitoring method of the single-point threshold comparison model is to take the value of the current data point and the overall threshold range of the pulse data for judgment;
[0012] When the data feature of the historical business data is sawtooth data, the data monitoring model is determined to be a single-point threshold comparison model, the warning parameter is the threshold range of different water levels in the sawtooth data, and the monitoring method of the single-point threshold comparison model is to take the value of the current data point and the threshold range corresponding to the water level at which it is located for judgment;
[0013] When the data characteristics of the historical business data are continuously fluctuating data or continuously stable data, the data monitoring model is determined to be a sudden increase or decrease score comparison model, the warning parameter is the threshold range of the sudden increase or decrease score, and the monitoring method of the sudden increase or decrease score comparison model is to take the previous N points of the current data point compared with the previous month to judge the sudden increase or decrease score.
[0014] In one possible design, in another implementation of another aspect of the embodiment of the present application, the training module is specifically used to use the historical business data to train to obtain the threshold range and / or upper and lower water levels and the model parameters of the data monitoring model, and use the threshold range and / or upper and lower water levels as the warning parameters.
[0015] In one possible design, in another implementation of another aspect of the embodiment of the present application, the training module is specifically used to preprocess the historical business data to obtain an intermediate data set; perform average processing on the intermediate data set to obtain a first population mean, and perform standard deviation processing on the intermediate data set to obtain a first standard deviation;
[0016] Performing normal fitting on the first population mean and the first standard deviation to obtain a first threshold range;
[0017] Deleting the outliers beyond the first threshold to obtain a second difference value set;
[0018] Performing an average process on the second difference set to obtain a second population mean, and performing a standard deviation process on the second difference set to obtain a second standard deviation;
[0019] Performing normal fitting on the second population mean and the second standard deviation to obtain a second threshold range;
[0020] Deleting the outliers beyond the second threshold to obtain a third difference value set;
[0021] The above operation is repeated until there are no abnormal points to obtain a target threshold range, and the upper limit and lower limit of the target threshold range are used as the threshold range or the threshold range of the sudden rise and fall score.
[0022] In one possible design, in another implementation of another aspect of the embodiment of the present application,
[0023] The training module is used to obtain a set of predicted values and a set of true values corresponding to the historical business data;
[0024] Performing difference processing on the predicted value set and the true value set to obtain a first difference set, and using the first difference set as the intermediate data set;
[0025] or,
[0026] The training module is used to use the real value set corresponding to the historical business data as the intermediate data set;
[0027] or,
[0028] The training module is used to obtain a set of sudden increase and decrease scores corresponding to the historical business data, and the set of sudden increase and decrease scores is used as the intermediate data set.
[0029] In a possible design, in another implementation of another aspect of the embodiment of the present application, the training module is specifically used to sort the historical business data according to the order of the real value from small to large to obtain a first data column;
[0030] Taking the Kth data point in the first data column as a dividing point, the first data column is divided into a second data column and a third data column, where K is a positive integer greater than 1;
[0031] Performing variance processing on the second data column and the third data column respectively to obtain a first variance value and a second variance value;
[0032] Summing the first variance value and the second variance value to obtain the statistic of the K-th data point;
[0033] Repeat the above operation to obtain a set of statistics corresponding to the first data column;
[0034] The first data column is split based on the data point corresponding to the minimum statistic in the statistic set to obtain a first data column and a second data column corresponding to the historical business data, wherein the minimum value of the first data column is greater than the maximum value of the second data column; the water level of the first data column is determined as the upper water level of the historical business data, and the water level of the second data column is determined as the lower water level of the historical business data.
[0035] In one possible design, in another implementation of another aspect of the embodiment of the present application, the processing module is further used to perform data smoothing processing on the historical business data to eliminate the trend characteristics of the historical business data to obtain first data, and the first data retains the fluctuation characteristics of the historical business data;
[0036] Performing panel data processing on the first data to obtain second data;
[0037] The second data is processed based on variance segmentation to remove abnormal points to obtain third data, where the third data is one-dimensional time series data;
[0038] Time segmenting the third data by using odd-even group cross validation to obtain at least two data segments of the historical service data;
[0039] The training module is also used to use the at least two data segments to respectively train and obtain at least two groups of model parameters and warning parameters of the data monitoring model.
[0040] In a possible design, in another implementation of another aspect of the embodiment of the present application, when the data feature of the historical business data is continuous fluctuating data or continuous stable data, the warning parameter also includes a year-on-year threshold range parameter, and the monitoring method of the sudden rise and fall score comparison model is to take the previous X points of the current data point compared with the previous month to judge the sudden rise and fall score, and take the year-on-year Y points of the current data point to judge the year-on-year threshold range;
[0041] or,
[0042] When the data characteristics of the historical business data are continuously fluctuating data or continuously stable data, the warning parameters also include a year-on-year threshold range parameter and a holiday threshold range parameter. The monitoring method of the sudden increase or decrease score comparison model is to take the previous X points of the current data point compared with the previous month to judge the sudden increase or decrease score, and take the year-on-year Y points of the current data point to judge the year-on-year threshold range, and take the year-on-year Z points of the holidays to judge the holiday threshold range, where X, Y and Z are positive integers.
[0043] In one possible design, in another implementation of another aspect of the embodiment of the present application, the processing module is specifically used to obtain a set of change points corresponding to N consecutive data points in the historical business data, where the N consecutive data points are reference data points when predicting a target data point, and the target data point is the first data point after the Nth data point in the N consecutive data points, where N is a positive integer;
[0044] Segmenting the N consecutive data points according to each change point in the change point set to obtain at least one data segment;
[0045] Performing linear regression processing on the last data segment of the at least one data segment to obtain a predicted value of the target data point;
[0046] Obtain a difference between the predicted value of the target data point and the true value of the target data point;
[0047] Repeat the above operation to obtain the difference value set corresponding to the historical business data;
[0048] The difference set is processed by kernel density estimation algorithm and Gaussian mixture model to obtain corresponding data distribution information;
[0049] The data characteristics corresponding to the historical business data are determined according to the data distribution information.
[0050] In one possible design, in another implementation of another aspect of the embodiment of the present application, the processing module is specifically configured to determine that the data feature of the historical service data is pulse data when the data distribution information indicates that the difference in the difference set conforms to the white noise distribution;
[0051] If the data distribution information indicates that the difference in the difference set satisfies at least two distribution types and the at least two distribution types are arranged at intervals and the duration exceeds a threshold, it is determined that the data feature of the historical business data is sawtooth data;
[0052] If the data distribution information indicates that the differences in the difference set are at least two normal distributions, it is determined that the data feature of the historical business data is continuous fluctuation data;
[0053] When the data distribution information indicates that the differences in the difference set are normally distributed, it is determined that the data feature of the historical business data is continuous and stable data.
[0054] In one possible design, in another implementation of another aspect of the embodiment of the present application, the processing module is specifically used to perform denoising processing on the last data segment of the at least one data segment, and delete abnormal data points in the last data segment to obtain a smooth data segment;
[0055] Perform linear regression on the smoothed data segment to obtain the predicted value of the target data point.
[0056] In one possible design, in another implementation of another aspect of the embodiment of the present application, the processing module is specifically used to obtain a first score of the Ith data point among the consecutive N data points, the first score being an evaluation score indicating the sudden rise or fall trend of the Ith data point, wherein if the first score is greater than 0, it indicates that the Ith data point has a sudden rise trend, and if the first score is less than 0, it indicates that the Ith data point has a sudden fall trend;
[0057] Determine a classification result of the I-th data point according to the first score, where the classification result is used to indicate whether the I-th data point is a change point;
[0058] When the classification result indicates that the I-th data point is a change point, record the I-th data point as a change point;
[0059] Repeat the above operation, traverse the N consecutive data points to obtain the candidate change point set corresponding to the N consecutive data points;
[0060] Using the candidate change point set to segment the N consecutive data points to obtain at least one data segment;
[0061] Determining water level drop and slope change information of the at least one data segment;
[0062] The change point set is obtained by merging each change point in the candidate change point set according to the water level drop and the slope change information.
[0063] In one possible design, in another implementation of another aspect of the embodiment of the present application, the processing module is specifically used to use mad denoising to delete abnormal values from the first data point to the I-1th data point in the continuous N data points to obtain smoothed data and to delete the first data after the maximum value and the minimum value from the first data point to the I-1th data point;
[0064] Using linear regression processing to obtain a set of predicted values corresponding to the smoothed data and a predicted value of the I-th data point;
[0065] Perform residual processing on the predicted value set of the smoothed data and the true value set of the smoothed data to obtain the residual standard deviation;
[0066] The first score is determined according to the residual standard deviation, the true value of the I-th data point, and the predicted value of the I-th data point.
[0067] In one possible design, in another implementation of another aspect of the embodiment of the present application, the processing module is specifically used to obtain the current segment length corresponding to the I-th data point;
[0068] When the current segment length is less than 2, the sudden rise and fall score and the current segment length are cleared;
[0069] If the current segment length is greater than or equal to 2, the first score is greater than or equal to the upper limit, or less than or equal to the lower limit, and there is no change point before the I-th data point or the sudden increase or decrease score after the change point is 0, then the position of the I-th data point and the first score are recorded, and the current segment length is updated to 1;
[0070] When the current segment length is greater than or equal to 2, the first score is greater than or equal to the upper limit, or less than or equal to the lower limit, and the sign of the sudden increase or decrease score recorded before the I data point is the same as the sign of the first score, the current segment length is increased by 1 and the I-1 data point is determined as the change point;
[0071] When the current segment length is greater than or equal to 2, the first score is greater than or equal to the upper limit, or less than or equal to the lower limit, and the sign of the sudden increase or decrease score recorded before the first data point is different from the sign of the first score, the position of the first data point and the first score are recorded, and the current segment length is updated to 1;
[0072] If the current segment length is greater than or equal to 2, the first score is less than the upper limit and greater than the lower limit, the sudden rise and fall score recorded before the first data point is greater than or equal to the upper limit, and the value of the first data point is greater than the maximum value of the first data, then the current segment length is updated to 1;
[0073] If the current segment length is greater than or equal to 2, the first score is less than the upper limit and greater than the lower limit, the sudden increase or decrease score recorded before the first data point is less than or equal to the lower limit, and the value of the first data point is less than the minimum value of the first data, then the current segment length is updated to 1;
[0074] If the current segment length is greater than or equal to 2, the first score is less than the upper limit and greater than the lower limit, the sudden rise and fall score recorded before the first data point is less than the upper limit, or the value of the first data point is less than or equal to the maximum value of the first data, then the sudden rise and fall score and the current segment length are updated to 0;
[0075] If the current segment length is greater than or equal to 2, the first score is less than the upper limit value and greater than the lower limit value, the sudden increase or decrease score recorded before the I data point is greater than the lower limit value, or the value of the I data point is greater than or equal to the minimum value of the first data, then the sudden increase or decrease score and the current segment length are updated to 0.
[0076] In one possible design, in another implementation of another aspect of the embodiment of the present application, the processing module is specifically configured to perform difference processing on real values of adjacent data points of a first data segment in the at least one data segment to obtain first interval data;
[0077] Performing denoising on the first interval data to obtain second interval data;
[0078] Performing an average process on the second interval data to obtain a third overall mean, and performing a difference process on each interval data in the second interval data set to obtain a third standard deviation;
[0079] Performing normal fitting on the third population mean and the third standard deviation to obtain a third threshold range;
[0080] Deleting abnormal points in the second interval data according to the third threshold range to obtain third interval data;
[0081] Obtaining a data spacing standard statistic according to the third interval data, wherein the data spacing standard statistic is used to indicate a basic magnitude of a data segment interval;
[0082] Compressing the first to the first data segment according to the data interval standard statistic to obtain a first compressed number sequence;
[0083] Repeat the above operation on other data segments in the at least one data segment to obtain at least one compressed sequence;
[0084] The at least one compressed data column is subjected to linear regression processing to obtain a compressed slope set, and the at least one compressed data column is subjected to mean processing to obtain a mean set, and the compressed slope set is used as the slope change information, and the mean set is used as the water level drop.
[0085] In a possible design, in another implementation of another aspect of the embodiment of the present application, the candidate change point set includes k candidate change points, the compression slope set includes k+1 compression slopes, the mean set includes k+1 means, and the processing module is specifically used to traverse the first compression slope to the kth compression slope in the compression slope set, and when the absolute value of the i-th compression slope is greater than or equal to 1 and i is equal to k, attribute the kth candidate change point to the change point set;
[0086] When the absolute value of the i-th compression slope is greater than or equal to 1, i is not equal to k, the i-th compression slope is greater than 0, and the i-1-th mean is less than the i+1-th mean, the i-th candidate change point is included in the change point set;
[0087] When the absolute value of the i-th compression slope is greater than or equal to 1, i is not equal to k, the i-th compression slope is less than 0, and the i-1-th mean is greater than the i+1-th mean, the i-th candidate change point is included in the change point set;
[0088] When the absolute value of the i-th compression slope is less than 1 and i is equal to k, obtaining a first data segment of a k-1-th candidate change point and a k-th candidate change point, and a second data segment after the k-th change point;
[0089] Obtain a first maximum value and a first minimum value in the first data segment after removing the extreme value, and a second maximum value and a second minimum value in the second data segment after removing the extreme value;
[0090] When the first maximum value is smaller than the second minimum value or the first minimum value is larger than the second maximum value, the kth candidate change point is assigned to the change point set.
[0091] In a possible design, in another implementation of another aspect of the embodiment of the present application, the acquisition module is further used to acquire real-time business data;
[0092] The processing module is also used to call the data monitoring model corresponding to the real-time business data to monitor the real-time business data;
[0093] When abnormal data occurs, an early warning will be issued.
[0094] Another aspect of the present application provides a computer device, including: a memory, a processor, and a bus system;
[0095] Wherein, the memory is used to store programs;
[0096] The processor is used to execute the program in the memory, and the processor is used to execute the above-mentioned methods according to the instructions in the program code;
[0097] The bus system is used to connect the memory and the processor so that the memory and the processor can communicate with each other.
[0098] Another aspect of the present application provides a computer-readable storage medium, in which instructions are stored. When the computer-readable storage medium is run on a computer, the computer is enabled to execute the above-mentioned methods.
[0099] Another aspect of the present application provides a computer program product or a computer program, the computer program product or the computer program includes computer instructions, the computer instructions are stored in a computer-readable storage medium. The processor of the computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the methods provided by the above aspects.
[0100] It can be seen from the above technical scheme that the embodiments of the present application have the following advantages: different monitoring models are used for different data distributions. At the same time, the data monitoring model can be trained according to historical data to obtain model parameters and warning parameters, so that the model parameters and warning parameters can be adjusted accordingly according to the real data, and there is no need to manually set the parameters, so that the data can be monitored and alarmed more accurately. BRIEF DESCRIPTION OF THE DRAWINGS
[0101] Figure 1 This is a schematic diagram of an architecture of an application scenario in an embodiment of the present application;
[0102] Figure 2a A schematic diagram of the structure of the service features in the embodiment of the present application;
[0103] Figure 2b A structural diagram of data features in an embodiment of the present application;
[0104] Figure 3 A framework flow chart of the data processing method in the embodiment of the present application;
[0105] Figure 4 This is a schematic diagram of an embodiment of a data processing method in an embodiment of the present application;
[0106] Figure 4a A schematic diagram of a simple linear regression prediction in an embodiment of the present application;
[0107] Figure 4b A schematic diagram of a result of a simple linear regression prediction in an embodiment of the present application;
[0108] Figure 4c A schematic diagram of a result of linear regression prediction after segmentation of the curve water level difference and slope in the embodiment of the present application;
[0109] Figure 4d A schematic diagram of a result of mad denoising in an embodiment of the present application;
[0110] Figure 4e A schematic diagram showing a comparison of the results of a simple linear regression prediction and a linear regression prediction after segmentation of the curve water level difference and slope and mad denoising in an embodiment of the present application;
[0111] Figure 4f It is a result comparison diagram of simple linear regression prediction, linear regression prediction after segmentation of curve water level difference and slope, and linear regression prediction after segmentation of curve water level difference and slope and mad denoising in the embodiment of the present application;
[0112] Figure 4g A schematic diagram of normal fitting correction and calculation of threshold range in an embodiment of the present application;
[0113] Figure 4h This is a schematic diagram of the effect of normal fitting correction and year-on-year data monitoring in the embodiment of the present application;
[0114] Figure 4i This is a schematic diagram of the effect of calculating the upper and lower water levels in an embodiment of the present application;
[0115] Figure 4j It is a schematic diagram of the year-on-year data monitoring effect of the data monitoring model after calculating the threshold range and the upper and lower water levels in the embodiment of the present application;
[0116] Figure 4k A schematic diagram of a data monitoring effect of a data monitoring model after calculating a threshold range and upper and lower water levels in an embodiment of the present application;
[0117] Figure 4l A schematic diagram showing a sudden drop in the data curve in the embodiment of the present application;
[0118] Figure 5a A schematic diagram of a data curve showing that historical business data includes multiple data features in an embodiment of the present application;
[0119] Figure 5b A schematic diagram of a process for time segmenting historical business data in an embodiment of the present application;
[0120] Figure 5c In the embodiments of this application Figure 5a A schematic diagram of the data obtained after smoothing the data shown;
[0121] Figure 5d In the embodiments of this application Figure 5c A schematic diagram of the data obtained after panel data processing is performed on the data shown;
[0122] Figure 5e In the embodiments of this application Figure 5d A schematic diagram of the data obtained after the data shown is processed by removing outliers;
[0123] Figure 5f In the embodiments of this application Figure 5d A schematic diagram of the data after time segmentation;
[0124] Figure 5g In the embodiments of this application, based on Figure 5d The time segments shown are Figure 5a A schematic diagram of segmenting the historical business data shown;
[0125] Figure 6 This is another schematic diagram of an embodiment of the data processing method in the embodiment of the present application;
[0126] Figure 7 This is a schematic diagram of an embodiment of a data processing device in an embodiment of the present application;
[0127] Figure 8 This is another schematic diagram of an embodiment of a data processing device in an embodiment of the present application;
[0128] Fig. 9 FIG. 2 is a schematic diagram of another embodiment of a data processing device in an embodiment of the present application. DETAILED DESCRIPTION
[0129] The embodiments of the present application provide a data processing method, apparatus, device, and storage medium for realizing accurate monitoring and early warning of data while saving manpower.
[0130] The terms "first", "second", "third", "fourth", etc. (if any) in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein, for example. In addition, the terms "including" and "corresponding to" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.
[0131] As Internet technology (IT) systems become increasingly complex, and their stability and reliability have an increasing impact on corporate business and even social life, more efficient and reliable operation and maintenance tools are needed to ensure the stability and security of the system. As the business continues to expand, data monitoring capabilities are extremely challenging: different business data require different monitoring indicators and monitoring dimensions. At the same time, different business data have different data characteristics, which leads to problems such as poor results of traditional monitoring methods, a huge workload for manually configuring thresholds, and long-term manpower optimization required to set thresholds. Therefore, there is an urgent need to provide a solution that can improve data monitoring results while saving manpower.
[0132] In order to solve the above problems, the present application provides the following technical solutions: obtaining historical business data; extracting data features corresponding to the historical business data, the data features are used to indicate the data distribution of the historical business data; determining the data monitoring model corresponding to the historical business data according to the data features; using the historical business data to train the model parameters and early warning parameters of the data monitoring model, the early warning parameters are evaluation thresholds for determining abnormalities in business data, and the model parameters include but are not limited to model types and monitoring methods; using the model parameters and the early warning parameters to perform anomaly detection on the business data. In this way, different monitoring models are used for different data distributions. At the same time, the data monitoring model can obtain model parameters and early warning parameters based on historical data training, so that the model parameters and early warning parameters can be adjusted accordingly according to the real data, and there is no need to manually set the parameters, so that the data can be monitored and alarmed more accurately.
[0133] For ease of understanding, some of the terms used in this application are described below:
[0134] Artificial Intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines so that machines have the functions of perception, reasoning and decision-making.
[0135] Artificial Intelligence for IT Operations (AIOps): Use artificial intelligence (AI) and automation technology to optimize and improve the IT operations (Ops) process. The main goal of AIOps is to improve efficiency, reduce failures and reduce costs by monitoring, analyzing and automating the processing of IT operations data. Apply artificial intelligence and machine learning to the field of IT operations to improve the efficiency and quality of IT operations. AIOps can automatically and intelligently collect, analyze and process data from IT systems, and identify problems, predict trends and make autonomous decisions, thereby helping enterprises better manage and maintain their IT systems. The main goal of AIOps is to improve the efficiency, reliability and security of IT operations through automation and intelligence.
[0136] Time Series Data: refers to a sequence of data organized in chronological order. For time series data, time is its most important attribute, and changes in time will affect the value of the data. Common time series data include meteorological data, stock prices, sales data, sensor data, etc.
[0137] Anomaly detection: refers to finding data points in a data set that are different from the rest of the data. Anomaly detection has applications in many fields, such as fault detection, fraud detection, security detection, and medical diagnosis. The goal of anomaly detection is to find behaviors or data points that are different from normal behaviors or data points. These anomalies may be caused by equipment failure, malicious attacks, user errors, or other unknown reasons. Anomaly detection usually involves data analysis, statistics, and machine learning techniques to automatically identify and label abnormal data points from the data. The results of anomaly detection can be used to guide further investigation or action to better understand or solve the problem.
[0138] Kernel Density Estimation (KDE) algorithm: an algorithm that infers the distribution of population data based on a limited sample. Therefore, the result of the kernel density estimation algorithm is the probability density function estimate of the sample. According to the estimated probability density function, we can obtain some properties of the data distribution, such as the data aggregation area. At the same time, kernel density estimation uses the information of the data itself to approximate the sample data to the greatest extent (relative to parameter estimation), which can avoid the influence of human subjective prior knowledge on the approximate results. And a variety of kernel functions also provide options for practical applications.
[0139] Gaussian mixture model (GMM): A model that accurately quantifies things based on Gaussian probability density function (normal distribution curve) and decomposes one thing into several models based on Gaussian probability density function (normal distribution curve). The basic idea is to use multiple Gaussian distribution functions to approximate probability distributions of any shape, so GMM is composed of multiple single Gaussian density distributions (Gaussians), each Gaussian is called a "Component", and the linear sum of these "Components" is the probability density function of GMM. The data points to be clustered are regarded as sampling points of the distribution, and the parameters of the Gaussian distribution are estimated through the sampling points using a method similar to maximum likelihood estimation. The parameters are obtained to obtain the membership function of the data points for classification.
[0140] MAD denoising: MAD is a robust measure of sample deviation of univariate numerical data. n The mean absolute deviation formula is:
[0141]
[0142] MAD is often used for outlier detection. In an exemplary scheme, assuming that the experimental data follows a normal distribution, the outliers are allowed to fall within the 50% area on both sides, and the normal values are allowed to fall within the middle 50% area, that is,
[0143]
[0144] in
[0145]
[0146] And by
[0147] Φ(-a)=1-Φ(a)
[0148] Can
[0149]
[0150] From the table we can see
[0151]
[0152] Panel data: also called "parallel data", refers to the sample data formed by taking multiple sections in the time series and selecting sample observations on these sections at the same time. In other words, it is an m*n data matrix, which records a certain data indicator of m objects at n time nodes. In an exemplary scheme, suppose the GDP of the four cities to be counted in 2000, 2001, 2002, 2003, and 2004 are:
[0153] City A’s numbers are 8, 9, 10, 11, and 12;
[0154] City B is 9, 10, 11, 12, and 13 respectively;
[0155] City C is 5, 6, 7, 8, and 9 respectively;
[0156] The values of City D are 7, 8, 9, 10, and 11 respectively (in 100 million yuan). This overall description can be understood as a panel data.
[0157] The data processing method, device, equipment and storage medium provided in the embodiments of the present application can realize accurate monitoring and early warning of data while saving manpower. The exemplary application of the electronic device provided in the embodiments of the present application is described below. The electronic device provided in the embodiments of the present application can be implemented as various types of user terminals or as servers.
[0158] Upper and lower water levels: Water level usually refers to the elevation of the free water surface relative to a certain base surface (elevation refers to the height of a certain point relative to a reference surface). The base surface used to calculate the water level can be a characteristic sea level elevation at a certain location as the zero-point leveling base surface, which is called an absolute base surface; or the elevation of a specific point can be used as a reference to calculate the zero point of the water level, which is called a station base surface. The water level is the most intuitive factor reflecting the water conditions of a water body, and its changes are mainly caused by the increase or decrease in the amount of water in the water body. The water level process line is a curve showing the change of water level at a certain location over time, with the horizontal axis marked as time and the vertical axis marked as water level. In this application, the value of the data point is 0 as the absolute base surface of the data curve, the horizontal axis is time, and the vertical axis is the value of the data point. Therefore, the data curve can also be called a water level process line in another sense. Therefore, data points at different elevations can be divided into upper and lower water levels.
[0159] Normal fitting: It can also be called normality test. Normality test is a non-parametric test. There are many test methods, and the basic principles and applicable conditions of each test method are different. Among all the normality test methods, its basic principles can be explained from the following three aspects: (1) From the perspective of descriptive statistics, to test whether the data obeys the normal distribution, the goodness of fit between the "data distribution" and the "standard normal distribution model" can be calculated. If the fit is poor, it means that the data does not obey the normal distribution. (2) From the perspective of probability statistics, that is, from the perspective of hypothesis testing, whether the data obeys the normal distribution can be calculated by performing hypothesis testing with the null hypothesis of "the data obeys the normal distribution" and constructing relevant statistics to calculate the test results. (3) From the perspective of Bayesian statistics, this method itself does not test its normality, but calculates whether this batch of data is more likely to come from a population with a normal distribution that obeys the mean and standard deviation, or from a population with other distributions.
[0160] Sudden rise and fall: Sudden rise and fall of time series data refers to the data showing a clear upward or downward trend over a period of time. This trend can be cyclical or non-cyclical. This phenomenon is usually due to the influence of certain external factors, such as changes in the economic environment and social policies. In addition, the inherent changes in the data itself may also cause sudden rise and fall in time series data, such as changes in market demand, natural disasters, etc. When comparing the data month-on-month in this application, it is usually necessary to consider whether there is a large difference in data trends in the adjacent or preset time window. For example, at a certain point in time, the value of the data point has a sudden rise or fall compared to the value of the adjacent point. This trend is referred to as a sudden rise or fall in this application.
[0161] By running the data processing solution provided by the embodiment of the present application, the electronic device can realize accurate monitoring and early warning of data while saving manpower. That is, the electronic device itself can realize accurate monitoring and early warning of data while saving manpower. It is applicable to a variety of application scenarios. For example, intelligent operation and maintenance in scenarios such as intelligent fault analysis, abnormal access detection, link monitoring and planning, IT asset combing, business profiling, end-to-end visualization of applications, firewall policy convergence, and fault self-healing in clear scenarios, such as transactions of financial securities companies or fund transactions of bank customers, process monitoring of industrial manufacturing, abnormal game data detection, e-commerce transaction data monitoring, live broadcast traffic monitoring, and other application scenarios.
[0162] See also Figure 1 , Figure 1It is an optional architecture diagram under an application scenario of the data processing solution provided by the embodiment of the present application. In order to realize and support a data processing solution, the terminal device 100 is connected to the server 300 through the network 200, and the server 300 is connected to the database 400. The network 200 can be a wide area network or a local area network, or a combination of the two. The client for implementing the data processing solution is deployed on the terminal device 100, wherein the client can be run on the terminal device 100 in the form of a browser, or can be run on the terminal device 100 in the form of an independent application (application, APP), etc. The specific presentation form of the client is not limited here. The server 300 involved in this application can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (Content Delivery Network, CDN), and basic cloud computing services such as big data and artificial intelligence platforms. The terminal device 100 can be a smart phone, a tablet computer, a laptop, a PDA, a personal computer, a smart TV, a smart watch, a vehicle-mounted device, a wearable device, etc., but is not limited thereto. The terminal device 100 and the server 300 can be directly or indirectly connected through the network 200 by wired or wireless communication, and this application does not limit this. The number of servers 300 and terminal devices 100 is also not limited. The solution provided in this application can be completed independently by the terminal device 100, or by the server 300, or by the terminal device 100 and the server 300 in cooperation, and this application does not make specific limitations. Among them, the database 400, in short, can be regarded as an electronic filing cabinet-a place for storing electronic files, and users can add, query, update, delete, and other operations on the data in the file. The so-called "database" is a collection of data that is stored together in a certain way, can be shared with multiple users, has as little redundancy as possible, and is independent of the application program. A database management system (DBMS) is a computer software system designed for managing databases, generally with basic functions such as storage, interception, security, and backup.Database management systems can be classified according to the database models they support, such as relational, Extensible Markup Language (XML); or according to the types of computers they support, such as server clusters, mobile phones; or according to the query language used, such as Structured Query Language (SQL), XQuery; or according to performance impulse focus, such as maximum scale, maximum operating speed; or other classification methods. Regardless of the classification method used, some DBMSs can cross categories, for example, supporting multiple query languages at the same time. In the present application, the database 400 can be used to store historical data and data monitoring models. Of course, the storage location of historical data and data monitoring models is not limited to the database. For example, they can also be stored in the distributed file system of the terminal device 100, blockchain or server 300.
[0163] In the present application, the data processing device can be divided into two parts when performing business data monitoring, namely online application and offline training. Among them, the offline training refers to determining the corresponding data monitoring model according to the data characteristics of the historical business data, and training the data monitoring model according to the historical business data to obtain the model parameters and early warning parameters of the data monitoring model, wherein the model parameters can be the corresponding parameters set according to the business characteristics corresponding to the historical business data, and the early warning parameters are used to determine the evaluation threshold when the business data is abnormal. The online application refers to deploying the trained data monitoring model on the corresponding data monitoring platform, and then calling different data monitoring models for different business data, and finally monitoring and warning the business data based on the data monitoring model.
[0164] Based on the above description, the business characteristics can be subdivided into: Figure 2a In the scenario shown, the business feature may include three parts: abnormal type, year-on-year data source, and abnormal sensitivity. Among them, the abnormal type can be a sudden increase in data, a sudden decrease in data, a year-on-year increase in data, a year-on-year decrease in data, or severe data jitter, etc. (Among them, the abnormal types can be combined in any two, that is, for the same business data, you may pay attention to the sudden increase or decrease in month-on-month data, or the increase or decrease in year-on-year data). The year-on-year data source mainly focuses on whether there is an important comparison period, and the important comparison period also needs to determine whether there is a holiday effect. The abnormal sensitivity can be set to warn any abnormality or set to reduce the warning for abnormalities with a small online impact radiation range, or relax the warning threshold, or set different abnormalities with different warning thresholds.
[0165] The data characteristics of the historical business data can be determined based on the distribution trend of the data. In an exemplary solution, Figure 2b As shown in the figure, historical business data can be divided into four data types according to the distribution trend of the data. Figure 2b As shown in the upper left of the middle, the distribution trend of each data is unpredictable, indicating Figure 2b The data type shown in the upper left is a pulse type; Figure 2b As shown in the upper right of the middle, the trend within a segment of data can be predicted, but there will be a large change in the water level, forming a new predictable trend data segment, and there will be jumps between the data segments, indicating Figure 2b The data type shown in the upper right corner is a sawtooth type; Figure 2b As shown in the lower left and lower right of the figure, the trend within a period of data can be continuously predicted, so Figure 2b The data types shown in the lower left and lower right are called continuous data. For continuous data, the data distribution characteristics can also be determined based on the difference between the predicted value and the true value. When the distribution of the difference conforms to the same normal distribution, the continuous data can also be called continuous stable data, such as Figure 2b As shown in the lower left of the figure; when the distribution of the difference conforms to multiple different normal distributions, the continuous data can also be called continuous fluctuation data, such as Figure 2b Shown in lower right center.
[0166] Based on the above classification of historical business data, the processing framework of the data processing method of the present application can be described as follows: Figure 3 As shown:
[0167] For historical business data, determine whether it is continuous data. If not, determine whether it is pulse data or sawtooth data. If it is pulse data, determine that its data monitoring model is a single-point threshold comparison model (also called a single-point threshold algorithm). During its offline training process, it is necessary to train to obtain a reasonable threshold range (also called upper and lower thresholds) corresponding to the historical business data. When it is applied online, it is necessary to take the value of the current data point and the reasonable threshold range for abnormal judgment. If it is sawtooth data, determine that its data monitoring model is a single-point threshold comparison model (also called a single-point threshold algorithm). During its offline training process, it is necessary to train to obtain different water levels corresponding to the historical business data and reasonable threshold ranges corresponding to different water levels. When it is applied online, it is necessary to take the value of the current data point and the reasonable threshold range corresponding to the water level of the current data point for abnormal judgment. If it is continuous data, determine that its data monitoring model is a sudden rise and fall comparison model (also called a sudden rise and fall algorithm). During its offline training process, it is necessary to train to obtain a reasonable sudden rise and fall threshold range (also called a sudden rise and fall upper score and lower score) corresponding to the historical business data. At the same time, different smoothing windows are trained for continuous fluctuation type and continuous stability type respectively for data prediction. After determining that it is continuous data, the continuous data can also be judged for periodicity. If there is no periodicity, when applied online, the previous N points of the current data point can be directly taken for sudden rise and fall judgment (that is, to judge whether there is a sudden rise or sudden fall). If there is periodicity, it is judged whether there is a holiday effect. If there is no holiday effect, it is necessary to train to obtain the year-on-year rise and fall thresholds during offline training; when applied online, select the previous N data of the current data point for sudden rise and fall judgment, and also take the previous M points and the last M points of the current data point for year-on-year rise and fall judgment. If there is a holiday effect, it is necessary to train the year-on-year increase or decrease thresholds and the holiday year-on-year increase or decrease thresholds during offline training; when applied online, select the previous N data points of the current data point compared with the previous month to make a sudden increase or decrease judgment, and also take the previous M points and the last M points of the current data point compared with the previous year to make a year-on-year increase or decrease judgment, and take the holiday data points to make a year-on-year increase or decrease judgment before the previous M points and the last M points of the holiday data points.
[0168] It can be understood that in the specific implementation of the present application, related data such as historical business data or real-time business data are involved. When the above embodiments of the present application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of relevant data need to comply with relevant laws, regulations and standards of relevant countries and regions.
[0169] Combined with the above introduction, the data processing method in this application will be introduced below. Please refer to Figure 4, an embodiment of the data processing method in the embodiment of the present application includes:
[0170] 401. Obtain historical business data.
[0171] In this embodiment, the data processing device can obtain historical service data corresponding to the service according to the service characteristics.
[0172] In an exemplary solution, if the business feature focuses on year-on-year increases and decreases, and also focuses on year-on-year increases and decreases on important dates, the historical business data should include multiple historical data within the same time period, and should also include historical data within the same time period on important dates. For example, when monitoring the transaction data of a shopping platform, it is necessary to obtain year-on-year increases and decreases. The historical data can obtain the transaction data from 0:00 to 23:59 on the seven days before the monitoring day. If it is also necessary to pay attention to the historical data of important dates, the transaction data from 0:00 to 23:59 on the promotion day closest to the monitoring day can be obtained, or the transaction data from 0:00 to 23:59 on the historical promotion days of the shopping platform can be directly obtained.
[0173] If the business feature only pays attention to the comparison of sudden rise and fall, the historical business data may only include the historical data of the previous day.
[0174] It should be understood that the above two application scenarios are only used for illustration and do not limit the specific source of the historical business data in this application.
[0175] 402. Extract data features corresponding to the historical business data, where the data features are used to indicate data distribution of the historical business data.
[0176] The data processing device analyzes the data distribution of the historical business data and obtains the data type corresponding to the historical business data.
[0177] In an exemplary scheme, the data processing device can obtain the predicted value of the historical business data, and then perform difference processing between the predicted value and the actual value of the historical business data to obtain a difference set; then use the KDE algorithm and the GMM algorithm to determine the data distribution information of the differences in the difference set; finally, determine the data features corresponding to the historical business data based on the data distribution information.
[0178] In this embodiment, the difference set can be understood as the absolute difference between the predicted value and the true value, that is, the difference in the difference set is equal to the predicted value minus the true value. For example, the historical business data is the transaction amount of a convenience store in 24 hours a day, calculated in hours. Assume that its true value (recorded from 0:00, in RMB) is (0, 0, 0, 0, 0, 30, 70, 80, 100, 70, 50, 40, 90, 80, 56, 36, 90, 80, 60, 80, 30, 50, 0, 0), and its corresponding The predicted values are (0, 0, 0, 0, 27, 68, 76, 99, 67, 55, 47, 89, 88, 67, 40, 88, 82, 56, 89, 33, 42, 5, 10), and the difference set is (0, 0, 0, 0, 0, -3, -2, -4, -1, -3, 5, 7, -1, 8, 11, 4, -2, 2, -4, 9, 3, -8, 5, 10).
[0179] After obtaining the data distribution information, the data features corresponding to the historical business data may be determined according to the data distribution information as follows:
[0180] When the data distribution information indicates that the difference in the difference set conforms to the white noise distribution, determining that the data feature of the historical service data is pulse data;
[0181] When the data distribution information indicates that the difference values in the difference value set satisfy at least two distribution types and the at least two distribution types are arranged at intervals, and the duration of each distribution type satisfies a threshold, it is determined that the data feature of the historical business data is sawtooth data;
[0182] If the data distribution information indicates that the differences in the difference set are at least two normal distributions, it is determined that the data feature of the historical business data is continuous fluctuation data;
[0183] When the data distribution information indicates that the differences in the difference set are normally distributed, it is determined that the data feature of the historical business data is continuous and stable data.
[0184] In mathematical statistics, linear regression is usually used to predict the trend of historical business data and obtain the predicted value. The univariate linear model can be expressed as follows:
[0185] y i =α+βx i +∈ i ,i=1,…,n.
[0186] Where α, β are constants. The random perturbation term ∈ iis a random variable that cannot be directly observed. For regression analysis, it is usually assumed that
[0187] Using the least squares method, that is, the residual sum of squares (RSS) of the observed value and the corresponding regression estimate converges to the minimum value, it can be derived that
[0188]
[0189] For equally spaced time series data, we can consider it as independent variable x is {1,…,n} and dependent variable y is {y1,…,y n}, it can be simplified to
[0190]
[0191] Then the trend estimate for the next point in the time series is
[0192]
[0193] In an exemplary embodiment, Figure 4a The broken line shown represents single-dimensional time series data, the straight line represents the predicted value after linear regression processing of the single-dimensional time series data, and the rightmost point is the predicted value of the next point based on the current trend.
[0194] However, in the actual production process, data will have the following important characteristics: the timeliness and estimated value of data will gradually decrease over time; the data has large noise, linear regression is not robust, and is easily disturbed by burrs. At the same time, linear regression is only a point estimate, and a legal interval needs to be calculated to determine whether the true value appears outside the legal interval and whether there is an anomaly.
[0195] In order to solve the above problem and obtain the predicted value of the historical business data more accurately, this embodiment provides the following technical solutions:
[0196] In an exemplary scheme, a change point set corresponding to N consecutive data points in the historical business data is obtained, where the N consecutive data points are reference data points when predicting a target data point, and the target data point is the first data point after the Nth data point in the N consecutive data points, where N is a positive integer; the N consecutive data points are segmented according to each change point in the change point set to obtain at least one data segment; and linear regression processing is performed on the last data segment in the at least one data segment to obtain a predicted value of the target data point.
[0197] The above process can also be called segmentation using the water level difference and slope of the curve. Figure 4bis a predicted value obtained by common linear regression processing, and Figure 4c It is a predicted value obtained by segmenting the curve water level difference and slope and then performing linear regression processing on the segmented values. From the comparison of the two figures, it can be seen that in the technical solution provided by this embodiment, the predicted value is closer to the true value. Figure 4c As shown, using the water level segmentation algorithm, it is calculated that the data shape changes at the 109th point (the thick black line in the figure). Linear regression is performed using the 109th point and the second half of the data points thereafter. The estimated value of the last point is at the five-pointed star in the figure. It can be seen that the estimated value is significantly close to the true value.
[0198] In another exemplary scheme, a change point set corresponding to N consecutive data points in the historical business data is obtained, and the N consecutive data points are reference data points when predicting the target data point. The target data point is the first data point after the Nth data point in the N consecutive data points, and N is a positive integer; the N consecutive data points are segmented according to each change point in the change point set to obtain at least one data segment; the last data segment in the at least one data segment is denoised, and the abnormal data points in the last data segment are deleted to obtain a smooth data segment. In the embodiment of the present application, the denoising process can be mad denoising, or other denoising processes that can delete abnormal data points from the data segment to obtain a smooth data segment; the smooth data segment is subjected to linear regression processing to obtain the predicted value of the target data point. In this scheme, the mad denoising process can generally be understood as using the absolute median difference to determine the abnormal value, and then eliminating the abnormal value to obtain a smooth data. For example, Figure 4d As shown, the broken line corresponds to the single-dimensional time series data, and the point is used to indicate the normal data obtained after the MAD denoising process. In this technical solution, the comparison between the simple linear regression and the linear regression after MAD denoising can be as follows: Figure 4e What you want, Figure 4e It can be seen that obtaining Figure 4c The black thick line is followed by a simple linear regression prediction of the data points to obtain Figure 4e As can be seen from the results on the left, if linear regression is used directly, the predicted point (five-pointed star) is underestimated because of the two downward burrs interfering with the calculation. Figure 4e In the middle image, mad denoising is used, the black solid points are retained, and the two points marked with crosses are deleted as burr points. Figure 4e In the right figure, the denoised data is subjected to linear regression, and the predicted point (five-pointed star) is closer to the actual data trend.
[0199] As Figure 4c The predicted values obtained by performing different linear regression processing on the overall data distribution can be as follows Figure 4f As shown, Figure 4fThere are three five-pointed stars in the middle. 1 represents linear regression prediction using all data. 2 represents segmentation using the water level difference and slope of the curve, and linear regression prediction after retaining the recent water level trend. 3 represents linear regression prediction after segmentation and MAD denoising on the second half of the data. It is obvious that the prediction value of linear regression prediction after MAD denoising on the second half of the data after segmentation is the most accurate.
[0200] At the same time, if there are multiple points that drop continuously, MAD can delete the most recent abnormal points and retain the data change trend of normal points to ensure that multiple sudden drop points can be captured as abnormalities and will not be disturbed by outliers.
[0201] In this embodiment, the specific operation of the data processing device for segmenting data using the water level difference and slope of the curve can be as follows:
[0202] First, traverse each inspection point, capture each point where there is a possibility of sudden increase or decrease compared to the previous data, and make a round of records. In this process, the specific operations can be as follows:
[0203] For each data point
[0204] 1. If the current segment length is less than 2, the score is reset to zero and the segment length is reset to zero.
[0205] 2. Otherwise, use linear regression analysis to predict the current value and residual standard deviation. The specific method of predicting the current value can refer to the above linear regression solution, which will not be repeated here. The processing process of the residual standard value can be as follows: the real value of the historical business data is expressed as {y1,…,y n}, the prediction scheme described above is used to predict each data point of the historical business data to obtain the predicted value, which is expressed as Then the residual standard deviation std calculated based on the true value and the predicted value can be expressed as For example, suppose the historical business data is the transaction amount of a convenience store within 24 hours a day, counted in hours (recorded from 0:00, recorded once every hour, in RMB, yuan), and its true value is (0, 0, 0, 0, 0, 30, 70, 80, 100, 70, 50, 40, 90, 80, 56, 36, 90, 80, 60, 80, 30, 50, 0, 0), and The corresponding predicted values are (0, 0, 0, 0, 27, 68, 76, 99, 67, 55, 47, 89, 88, 67, 40, 88, 82, 56, 89, 33, 42, 5, 10), and the difference set is (0, 0, 0, 0, 0, -3, -2, -4, -1, -3, 5, 7, -1, 8, 11, 4, -2, 2, -4, 9, 3, -8, 5, 10). Therefore, the residual standard deviation of this set of data is
[0206] 3. Calculate the data after MAD denoising and smoothing and after removing the maximum and minimum values.
[0207] 4. Calculate the sudden rise and fall score. It is understandable that the sudden rise and fall score can be calculated using the following technical solution:
[0208] The true value of historical business data is expressed as {y1,…,y n}, the prediction scheme described above is used to predict each data point of the historical business data to obtain the predicted value, which is expressed as Then the residual standard deviation std calculated based on the true value and the predicted value can be expressed as
[0209]
[0210] The true value y for the next data point n+1 , and its corresponding predicted value is Then the sudden rise and fall score can be expressed as
[0211]
[0212] After calculating the sudden increase or decrease score, a reasonable threshold range can be determined based on the sudden increase or decrease score data of each data point in the historical business data. In an exemplary solution, if the threshold range of the sudden increase or decrease score is set to (-3, 3), when the sudden increase or decrease score is greater than 3, it is considered that there is a sudden increase abnormality, when the sudden increase or decrease score is less than -3, it is considered that there is a sudden decrease abnormality, and when the sudden increase or decrease classification is less than or equal to 3 and greater than or equal to -3, the data is considered normal.
[0213] 5. Assuming that the reasonable threshold range of the sudden rise and fall score is (-3, 3), if the sudden rise and fall score is greater than or equal to 3 or less than or equal to -3, perform the following judgment:
[0214] If there is no change point before or the score after the change point is 0, then record the current change point position and score, the segment length is 1, and do not add a change point temporarily; if the previously recorded score and the current score have the same sign, then increase the segment length by 1; if the previously recorded score and the current score have different signs, then record the current change point position and score, the segment length is 1, and do not add a change point. For example, assuming there is no change point from the first data point to the third data point or the score after the change point is 0, and at this time the sudden increase or decrease score of the fourth data point is greater than or equal to 3 or less than or equal to -3, then record the position and sudden increase or decrease score of the fourth data point, the segment length is updated to 1, and do not add a change point temporarily; then calculate the sudden increase or decrease score of the fifth data point (it can be understood that at this time, the sudden increase or decrease score of the fifth data point compared to the first data point to the third data point should be calculated), and then compare. If the sudden increase or decrease score of the fifth data point is greater than or equal to 3 or less than or equal to -3, since the fourth data point may be a change point, it does not meet the "no change point or sudden increase or decrease" condition. In the case where the score after the change point is "0", the sudden rise and fall score of the fifth data point is compared with the sudden rise and fall score of the fourth data point. If the sudden rise and fall score of the fifth data point has the same sign as the sudden rise and fall score of the fourth data point (that is, both are positive or negative), the current segment length is increased by 1 (the segment length at this time is a variable statistical number, which is updated to 1 at the fourth data point and should be updated to 2 at this time); if the sudden rise and fall score of the fifth data point has different signs from the sudden rise and fall score of the fourth data point (that is, one is positive and the other is negative), the position and sudden rise and fall score of the fifth data point are recorded, and the segment length is recorded again as 1. When the sudden rise and fall score of the fifth data point is less than 3 and greater than -3, the fourth data point is determined to be a burr point.
[0215] 6. If the score is less than 3 and greater than -3, perform the following judgment: if the previously recorded score is greater than or equal to 3 and the current value is greater than the maximum value of the data after removing the maximum and minimum values, the segment length is increased by 1; if the previously recorded score is less than or equal to -3 and the current value is less than the minimum value of the data after removing the maximum and minimum values, the segment length is increased by 1; otherwise, the score is cleared and the segment length is cleared.
[0216] 7. If the segment length is greater than or equal to 2 and the starting change point of the segment has not been recorded yet, then add the current change point position and score to the change point set, and the starting point of the next data segment starts from this newly recorded change point, and repeat the above cycle. Finally, return the candidate change point set C: {c1,…,c k} and a set of scores S: {s1,…,s k}.
[0217] After obtaining the above candidate change point set, in order to achieve overall segmentation, the above candidate change point set can be used to segment the historical business data into multiple data segments, and by calculating the water level difference and slope change information of each segment, it is determined whether the candidate change points can be merged to obtain a more final set of change points. At the same time, in order to reduce the amount of calculation, the historical business data can also be compressed to obtain a compressed series, and then the compression slope and mean of the compressed data series are calculated, and finally the merging of the candidate change points is determined based on the compression slope and mean.
[0218] The following is an explanation of the operations at different stages:
[0219] When performing data compression processing, the specific process can be as follows:
[0220] 1. Calculate the standard statistics of data spacing. For each data segment {y1, ..., y n}, calculate the gap between adjacent true values to obtain interval data I: {i1,...,i n-1},in
[0221] i t =y t+1 -y t .
[0222] For example, assuming that the historical business data is the transaction amount of a convenience store within 24 hours a day, counted in hours (recorded starting from 0:00, recorded once at every hour, in RMB), its true value is (0, 0, 0, 0, 0, 30, 70, 80, 100, 70, 50, 40, 90, 80, 56, 36, 90, 80, 60, 80, 30, 50, 0, 0), and its corresponding interval data is (0, 0, 0, 0, 30, 40, 10, 20, -30, -20, -10, 50, -10, -24, -20, 54, -10, -20, 20, -50, 20, -50, 0).
[0223] For interval data {I1, ..., I n-1} Perform mad data denoising to prevent extreme values from interfering with statistical calculations and improve robustness, and obtain denoised interval data I′: {i′1, ..., i′ k}, and calculate its mean μ I , standard deviation σ I . Assume I′: {i′1,...,i′ k} obeys normal distribution, then the threshold range can be set as (μ I -3σ I , μ I +3σI ), so that I +3σ I or less than μ I -3σ I The data is considered as an outlier, and each point of I′ is traversed, and the outliers are removed. The absolute value of the non-outlier point is added to I″, and then I″ is obtained: {i″1, ..., i″ t}, to further improve the robustness, the data spacing standard statistic Isb can be expressed as follows:
[0224]
[0225] This statistic is used to express the basic magnitude of the interval between the data segments. When the interval is greater than Isb, it means that from the perspective of the entire data segment, the data change of this data point is larger than that of the previous data point; when the interval is greater than Isb, it means that from the perspective of the entire data segment, the data change of this data point is within an acceptable range compared to the previous data point.
[0226] 2. Based on the data spacing standard statistic Isb, the data segment is compressed and transformed based on the slope characteristics to generate a compressed sequence. The specific operation can be as follows:
[0227] a. Initialize the compressed sequence to have only one element, 0, as the starting point.
[0228] b. Traverse all elements except the first element in the data segment, calculate the difference between the element and the previous element and divide it by Isb to get the compressed interval change, add the interval change and the last element of the compressed sequence to get the compressed data value, and then add it to the compressed sequence. For example, the original y values are y1, y2, and y3, and then initialized to 0 in the S list. When calculating again, divide the difference between y2 and y1 by LSB to get a value Z1, and add Z1 to 0 to get a new value Z1 in the S list. Then calculate the difference between y3 and y2 and divide it by LSB to get a value Z2, and add Z1 to Z2 to get a new value Z1+Z2 in the S list.
[0229] When calculating the water level drop and slope change information, the specific process can be as follows:
[0230] 1. Perform linear regression on the compressed series obtained by the above method to obtain the slope β s , when β s >1 indicates that the data changes steeply upward from the perspective of the data spacing standard statistic Isb evaluation standard, β s <-1 indicates that the data changes downward sharply.
[0231] 2. The historical business data is divided into the change point set C: {c1,…,ck} is divided into small data segments, based on the compression slope β after the above data spacing is unified s Calculation method, calculate the compression slope of each small data segment And calculate the mean of each segment {m0,…,m k}.
[0232] When candidate change points are merged based on water level drop and slope change information, the specific process can be as follows:
[0233] Traverse the first compression slope to the kth compression slope in the compression slope set, and when the absolute value of the ith compression slope is greater than or equal to 1 and i is equal to k, the kth candidate change point is attributed to the change point set. That is, when the absolute value of the last compression slope is greater than or equal to 1, the last candidate change point is directly determined to be a significant change point, and it is attributed to the change point set.
[0234] When the absolute value of the i-th compression slope is greater than or equal to 1, i is not equal to k, the i-th compression slope is greater than 0 and the i-1-th mean is less than the i+1-th mean, the i-th candidate change point is included in the change point set. For example, when the absolute value of the second compression slope is greater than or equal to 1 (i.e., the third data segment changes more steeply upward or the third data segment changes more sharply downward), and the value of k is 5, if the second compression slope is greater than 0 (i.e., the third data segment changes more steeply upward), the first mean is less than the third mean (i.e., the overall value of the fourth data segment is greater than the second data segment), it means that the two data segments divided by the second candidate change point have a water level difference, and the second candidate change point can be included in the change point set.
[0235] When the absolute value of the i-th compression slope is greater than or equal to 1, i is not equal to k, the i-th compression slope is less than 0, and the i-1-th mean is greater than the i+1-th mean, the i-th candidate change point is included in the change point set. For example, when the absolute value of the second compression slope is greater than or equal to 1 (i.e., the third data segment changes more steeply upward or the third data segment changes more sharply downward), and the value of k is 5, if the second compression slope is less than 0 (i.e., the third data segment changes more sharply downward), the first mean is greater than the third mean (i.e., the overall value of the fourth data segment is less than the second data segment), then it means that the two data segments divided by the second candidate change point have a water level difference, and the second candidate change point can be included in the change point set.
[0236] When the absolute value of the i-th compression slope is less than 1 and i is equal to k, obtaining a first data segment of a k-1-th candidate change point and a k-th candidate change point, and a second data segment after the k-th change point;
[0237] Obtain a first maximum value and a first minimum value in the first data segment after removing the extreme value, and a second maximum value and a second minimum value in the second data segment after removing the extreme value;
[0238] When the first maximum value is less than the second minimum value or the first minimum value is greater than the second maximum value, the kth candidate change point is attributed to the change point set. For example, when the absolute value of the fifth compression slope is less than 1 (i.e., the sixth data segment changes more slowly), and the value of k is 5, the data segment between the fourth change point and the fifth change point (the fifth data) and the sixth data segment are selected; then the extreme values in the fifth data segment are removed (i.e., mad denoising is performed), and the maximum and minimum values are recorded, and the extreme values in the sixth data segment are removed, and the maximum and minimum values are recorded. Assume that the maximum value of the fifth data segment is 90 and the minimum value is 60; the maximum value of the sixth data segment is 140 and the minimum value is 100. At this time, the fifth data segment has a significant value drop compared to the data of the sixth data segment. Therefore, it can also be considered that there is a significant water level drop between the two data segments divided by the fifth candidate change point, and the fifth candidate change point is attributed to the change point set. Similarly, assume that the maximum value of the fifth data segment is 90 and the minimum value is 60; the maximum value of the sixth data segment is 50 and the minimum value is 20. At this time, the fifth data segment has a significant value difference compared to the data of the sixth data segment, so it can also be considered that there is a significant water level difference between the two data segments divided by the fifth candidate change point, and the fifth candidate change point is attributed to the change point set.
[0239] 403. Determine a data monitoring model corresponding to the historical business data according to the data feature.
[0240] In this embodiment, after the data features corresponding to the historical business data are determined, a corresponding data monitoring model is selected according to the corresponding data features. In this embodiment, the data monitoring model can be understood as a data monitoring strategy.
[0241] In an exemplary scheme, when the data feature of the historical business data is pulse data, the data monitoring model is determined to be a single-point threshold comparison model, the warning parameter is a threshold range, and the monitoring method of the single-point threshold comparison model is to take the value of the current data point and make a judgment based on the threshold range.
[0242] When the data characteristics of the historical business data are sawtooth data, the data monitoring model is determined to be a single-point threshold comparison model, and the warning parameter is the threshold range of different water levels. The monitoring method of the single-point threshold comparison model is to take the value of the current data point and the threshold range corresponding to the water level at which it is located for judgment. It can be understood that the duration of each water level can also be counted at the same time. For example, in a data curve of sawtooth data, its data distribution is an alternating distribution of high and low water levels, in which the high water level will be a low water level distribution after 10 data points, and after the low water level lasts for 15 data points, it will be a high water level distribution again. Therefore, when monitoring sawtooth data, in addition to judging whether the value of the current data point exceeds the threshold range of the current water level, it can also be judged whether the current data point exceeds the duration of the current water level.
[0243] When the data characteristics of the historical business data are continuously fluctuating data or continuously stable data, the data monitoring model is determined to be a sudden increase or decrease score comparison model, the warning parameter is the threshold range of the sudden increase or decrease score, and the monitoring method of the sudden increase or decrease score comparison model is to take the previous N points of the current data point compared with the previous month to judge the sudden increase or decrease score.
[0244] Optionally, when the data feature of the historical business data is continuous fluctuating data or continuous stable data, the warning parameter also includes a year-on-year threshold range parameter, and the monitoring method of the sudden increase and decrease score comparison model is to take the previous X points of the current data point compared with the previous month to judge the sudden increase and decrease score, and take the previous Y points of the current data point compared with the previous month to judge the year-on-year threshold range;
[0245] or,
[0246] When the data characteristics of the historical business data are continuously fluctuating data or continuously stable data, the warning parameters also include a year-on-year threshold range parameter and a holiday threshold range parameter. The monitoring method of the sudden increase or decrease score comparison model is to take the previous X points of the current data point compared with the previous month to judge the sudden increase or decrease score, and take the year-on-year Y points of the current data point to judge the year-on-year threshold range, and take the year-on-year Z points of the holidays to judge the holiday threshold range, where X, Y and Z are positive integers.
[0247] 404. The model parameters and warning parameters of the data monitoring model are obtained by training the historical business data. The warning parameters are evaluation thresholds for determining whether the business data is abnormal. The model parameters include but are not limited to model type and monitoring method.
[0248] In this embodiment, after determining the data monitoring model of the historical business data, the model parameters and early warning parameters that need to be trained for the data monitoring model can also be determined. In this embodiment, the model parameters can generally be understood as information such as the model type, monitoring method, and monitoring target of the data monitoring model. The early warning parameter can be understood as the evaluation threshold for determining that the business data is abnormal when the data monitoring model is applied online.
[0249] The following is an explanation of the training of different warning parameters corresponding to different data monitoring models:
[0250] 1. Training of threshold range.
[0251] When the data feature indicates that the historical service data is pulse data or sawtooth data, the training process of the threshold range may be as follows:
[0252] 1. Obtain the real value set of the historical business data.
[0253] 2. Calculate the population mean μ0 and standard deviation σ0 of the overall true value in the true value set, perform the first normal fitting, calculate the skewness and divergence to evaluate the normal fitting effect, and obtain the first threshold range. In an exemplary scheme, the first threshold range can be calculated based on the μ0 and the σ0. For example, it is set to [μ0-3σ0, μ0+3σ0].
[0254] 3. After eliminating the abnormal points outside [μ0-3σ0, μ0+3σ0], repeat 2 and 3 to get μ i and σ i ;
[0255] 4. After the data convergence is confirmed in 3, that is, there are no more abnormal points to be removed, the final fitted μ is obtained. n and σ n , then the final threshold ranges are [μ n -3σ n , μ n +3σ n ].
[0256] In this embodiment, the real values of the pulse data or sawtooth data can also be distinguished according to the high and low water levels, and then the threshold ranges are trained for the real values of different water levels. The specific training method is as follows, which will not be described in detail here.
[0257] When the data feature indicates that the historical business data is continuous data and a year-on-year increase or decrease comparison is performed, the training process of the threshold range may be as follows:
[0258] 1. Obtain the predicted value set and the true value set corresponding to the historical business data, and calculate the difference value set based on the predicted value set and the true value set. The predicted value of the historical business data is obtained by predicting the year-on-year historical data.
[0259] In this embodiment, the year-on-year forecast of historical business data can adopt the following technical solutions:
[0260] Get the historical data series at the same time; sort the data of the historical data series in ascending order to obtain the middle series; then use the xth data point in the middle series as the dividing point to divide the historical data series into data series 1 and data series 2, where x is a positive integer greater than 1; perform variance processing on data series 1 and data series 2 respectively to obtain variance value 1 and variance value 2; sum variance value 1 and variance value 2 to obtain the statistics of the xth data point; repeat the above operations to obtain the statistical set corresponding to the middle series; divide the middle series based on the minimum statistic in the statistical set, and determine the upper and lower water levels of the historical data series at the same time; calculate the mean of y data points of the target data point on a year-on-year basis, and use the mean as the water level mark of the target data point; based on which water level the water level mark is close to, use the year-on-year historical data mean of the water level as the predicted value of the target data point. For example, suppose the year-on-year historical data series is the number of transactions at convenience stores at 7 o'clock every morning in a week, specifically (5, 4, 6, 23, 20, 8, 7). Then sort them from small to large, and the middle series is (4, 5, 6, 7, 8, 20, 23). Calculate its statistics and get five data (i.e., the data points with a value of 8 in the series) as the dividing points of the upper and lower water levels. At this time, the values corresponding to the high water level are (20, 23), and the values corresponding to the low water level are (4, 5, 6, 7, 8). Assuming that the three year-on-year values of the current target data point are (4, 6, 8), then their mean is 6. At this time, 6 will be used as the current water level mark. Since 6 is closer to the low water level, the predicted value of the target data point is the mean of the low water level 6. If the three year-on-year values of the target data point are (4, 16, 18), then their mean is 19. At this time, 19 will be used as the current water level mark. Since 19 is closer to the high water mark, the predicted value for the target data point is the mean of the low water mark, 21.5.
[0261] 2. Calculate the overall mean μ0 and standard deviation σ0 of the overall difference in the difference set, perform the first normal fit, calculate the skewness and divergence to evaluate the normal fitting effect, and obtain the first threshold range. In an exemplary scheme, the first threshold range can be calculated based on the μ0 and the σ0. For example, it is set to [μ0-3σ0, μ0+3σ0].
[0262] 3. After eliminating the abnormal points outside [μ0-3σ0, μ0+3σ0], repeat 2 and 3 to get μi and σ i ;
[0263] 4. After the data convergence is confirmed in 3, that is, there are no more abnormal points to be removed, the final fitted μ is obtained. n and σ n , then the final threshold ranges are [μ n -3σ n , μ n +3σ n ].
[0264] It should be understood that, since the training process of the year-on-year increase and decrease values is carried out using the difference, the threshold range should be understood as the reasonable fluctuation range of the predicted value. In practical applications, when judging whether the true value is abnormal, the value obtained after calculating the predicted value and the reasonable fluctuation range should be compared with the true value. In an exemplary scheme, if the threshold range is trained to be (-5, +5), and the predicted value of the target data point is 100, the final value used for comparison is (95, 105). If the true value is 100, the true value is determined to be a normal value; if the true value is 106, the true value is determined to be an abnormal value.
[0265] When the data feature indicates that the historical service data is continuous data and a month-on-month sudden increase or decrease score comparison is performed, the training process of the upper score and the lower score of the sudden increase or decrease score may be as follows:
[0266] 1. Obtain a set of sudden increase and decrease scores of the historical business data.
[0267] 2. Calculate the overall mean μ0 and standard deviation σ0 of the sudden rise and fall scores in the sudden rise and fall score set, perform the first normal fitting, calculate the skewness and divergence to evaluate the normal fitting effect, and obtain the first threshold range. In an exemplary scheme, the first threshold range can be calculated based on the μ0 and the σ0. For example, it is set to [μ0-3σ0, μ0+3σ0].
[0268] 3. After eliminating the abnormal points outside [μ0-3σ0, μ0+3σ0], repeat 2 and 3 to get μ i and σ i ;
[0269] 4. After the data convergence is confirmed in 3, that is, there are no more abnormal points to be removed, the final fitted μ is obtained. n and σ n , then the upper and lower scores of the final sudden rise and fall scores are [μ n -3σ n , μ n +3σ n ].
[0270] It can be understood that the above threshold range can also be understood as the upper and lower scores of sudden rise and fall scores, and the rising and falling thresholds of year-on-year data.
[0271] In actual applications, there will be an adjustment factor to cope with the situation where the tolerance for data increase / decrease may be wider on important dates (such as promotional days such as Double Eleven on shopping platforms). When the adjustment factor is greater than 0, the calculated threshold range will be relatively loose. When the adjustment factor is less than 0, the calculated threshold range will be more stringent.
[0272] For example, in Figure 4g In the normal fitting process shown, the gray dotted lines (the leftmost dotted line and the rightmost dotted line) are the normal fitting after calculating the mean and standard deviation of all data. In this normal fitting, the divergence and skewness are extremely unbalanced; after gradually eliminating outliers, the corrected normal fitting indicated by the black dotted line is obtained, and the divergence and skewness are in a relatively balanced state.
[0273] After the threshold range is calculated by this embodiment, Figure 4h In the data monitoring process shown, the data under the gray background exceeds the threshold range, thereby providing a continuous warning prompt.
[0274] 2. Upper and lower water level training.
[0275] The historical business data is sorted in ascending order according to the real value to obtain a first data column;
[0276] Taking the Kth data point in the first data column as a dividing point, the first data column is divided into a second data column and a third data column, where K is a positive integer greater than 1;
[0277] Performing variance processing on the second data column and the third data column respectively to obtain a first variance value and a second variance value;
[0278] Summing the first variance value and the second variance value to obtain the statistic of the K-th data point;
[0279] Repeat the above operation to obtain a set of statistics corresponding to the first data column;
[0280] The first data column is split based on the data point corresponding to the minimum statistic in the statistic set to obtain a first data column and a second data column corresponding to the historical business data, wherein the minimum value of the first data column is greater than the maximum value of the second data column; the water level of the first data column is determined as the upper water level of the historical business data, and the water level of the second data column is determined as the lower water level of the historical business data.
[0281] In an exemplary scheme, for a set of data {x1, x2, ..., xn-1 , x n}, first sort them from small to large to form a new sequence {x′1, x′2, ..., x′ n-1 , x′ n}, traverse from the second data to the last one, calculate it one by one as the cutting point, and calculate the sum of the discrete degrees of the two segments before and after as the statistic θ, that is, for the tth data, its statistic θ t for
[0282]
[0283] in,
[0284]
[0285] Get the statistical series {θ2,…,θ n-1 ,θ n}, take the statistic with the smallest value such as θ p , then the array is transformed into {x′1,…,x′ p-1} and {x′ p ,…,x′ n}After the division, the overall discreteness of the two groups is the smallest. Figure 4i As shown, Figure 4i The above figure is the original data. After segmentation based on the discrete degree, two sets of data are obtained. The data below the dotted line has a smaller value and can be classified as a low water level, while the data above the dotted line has a larger value and can be classified as a high water level. Figure 4i The following figure is a schematic diagram of the calculation of statistics. Figure 4i From the figure below, we can see that in the data arrangement from small to large, the statistic of the fifth data is the smallest, and the discrete degree of the two data segments divided by it is the smallest, so in Figure 4i In the above figure, according to the size of the data, 5 smaller data are selected as the low water level, and the remaining 2 larger data are divided into the high water level.
[0286] In an exemplary embodiment, Figure 4j The monitoring of the year-on-year data shown in the figure illustrates the beneficial effect of dividing the upper and lower water levels in this embodiment:
[0287] exist Figure 4j Take the turnover data of a certain day as an example, where the year-on-year monitoring data includes the data of the day and the data of the previous seven days. At the same time, the data of the day is predicted to obtain a set of predicted values, and the data of the previous seven days is averaged. Figure 4j As can be seen in the box on the right, historical data is distributed at two different water levels. By dividing the upper and lower water levels, today's data can be automatically classified as a low water level, and the predicted value will be closer to the current true value distribution than the seven-day average.
[0288] Below Figure 4k The scenario shown illustrates the beneficial effects of calculating the threshold range and dividing the upper and lower water level scheme in this embodiment:
[0289] exist Figure 4k In the above figure, the two horizontal lines are the threshold ranges calculated based on the training scheme of this embodiment, and the solid points in the figure are the abnormalities obtained by monitoring. It can be seen that the threshold range calculated by this embodiment can tolerate data fluctuations to the greatest extent and obtain a reasonable data fluctuation range. On the contrary, the change rate graph below is the change rate of the data of the day and the data of the previous day at each moment, with a minimum of -1143.21% and a maximum of 1410.15%, and in the case of a real abnormality ( Figure 4k The maximum change rate of the time period (the second color background) is 989.34%, and the time periods ( Figure 4k In the figure, from left to right, the first, third and fourth color backgrounds are shown. In fact, no abnormality occurs. Therefore, the threshold range calculated by this embodiment can greatly improve the accuracy of alarms and reduce false alarms compared with the traditional solution.
[0290] In some businesses involving multiple links or many machines, when some machines or links have problems, it will affect the monitoring indicators to a certain extent, causing a small drop or rise. However, the abnormality may not be detected by only using year-on-year data, such as Figure 4l As shown in the box, the data has an obvious sudden drop at this time. Since the data fluctuates up and down in the historical year-on-year data, it cannot reach the year-on-year abnormal threshold to generate an alarm. Therefore, a solution is provided to detect sudden increase or decrease anomalies from a month-on-month perspective.
[0291] It is understandable that the detection of sudden rise and fall from a month-on-month perspective has been described in the aforementioned step 402 and will not be repeated here.
[0292] 405. Perform anomaly detection on the business data using the model parameters and the warning parameters.
[0293] In this embodiment, after the model parameters and early warning parameters of the data monitoring model are obtained through training, the data monitoring model is deployed online and then applied online. At this time, in the online application, the specific model type of the data monitoring model and the specific monitoring method for abnormal detection of business data can be determined according to the model parameters, and the business data can be compared according to the early warning parameters. If the business data has data values that do not meet the early warning parameters, the business data is prompted as abnormal data.
[0294] In actual application scenarios, historical business data of the same business may have different data distributions in different time periods. Figure 5aAs shown in the figure, the minute-level time series data of the periodic indicator A for seven days (1440 points per day), with different lines representing different days. Figure 5a It can be seen that this curve fluctuates greatly in the first half and relatively less in the second half. At the same time, there are large abnormal fluctuations in the second half, which are easy to interfere with the judgment. Therefore, in another embodiment of the present application, the data processing device can also segment the historical business data in the time dimension, so that different data distributions of the historical business data in different time periods use different data monitoring models, such as Figure 5b As shown, the processing flow of the data processing device for segmenting the historical business data may be as follows:
[0295] 501. Perform data smoothing processing on the historical business data to eliminate trend characteristics of the historical business data to obtain first data, and the first data retains the fluctuation characteristics of the historical business data.
[0296] Data smoothing is performed on historical business data including different data features, so that the trend features of the historical business data are deleted to obtain a piece of data that only retains the fluctuation features of the historical business data.
[0297] It should be understood that the data smoothing process can be understood as performing trend feature removal processing on the historical business data. Figure 5a Taking the data shown in the figure as an example, after data smoothing, we can get the following Figure 5c A piece of data is shown.
[0298] 502. Perform panel data processing on the first data to obtain second data.
[0299] The data retaining the fluctuation characteristics is subjected to panel data processing to obtain second data. The panel data can be understood as displaying the data of multiple days according to time points.
[0300] by Figure 5c Taking the data shown in the figure as an example, after panel data processing, we get the following Figure 5d The panel data are shown.
[0301] 503. Process the second data based on variance segmentation to remove abnormal points to obtain third data, where the third data is one-dimensional time series data.
[0302] The data processing device uses the above-mentioned variance segmentation to perform abnormal denoising on the data at each time point, thereby eliminating abnormal values; at the same time, the extreme difference value is calculated for the data at each time point, thereby obtaining the data drop value at each time point, thereby obtaining the one-dimensional time series data. Figure 5e As shown, a one-dimensional time series data with outliers removed is obtained.
[0303] 504. Use even-odd group cross-validation to time segment the third data to obtain at least two data segments of the historical service data.
[0304] The data processing device uses an odd-even group cross-validation method to time segment the third data to obtain an optimal change point position; then the historical business data is segmented based on the change point position to obtain at least two data segments.
[0305] like Figure 5f As shown, in Figure 5a In the historical business data shown in FIG. 1 , the 400th point can be determined as the optimal change point position, and then the 400th point in the historical business data is selected based on the 400th point to segment the historical business data to obtain the following: Figure 5g The two data segments are shown.
[0306] It is understandable that after the historical business data is segmented, the above-mentioned Figure 4 The data processing method shown is used for processing, and the details are not repeated here.
[0307] Based on the above description, the online application of the data processing method in this application is described, as follows Figure 6 As shown:
[0308] 601. Obtain real-time business data.
[0309] The data processing device deploys the trained data monitoring model to the corresponding data monitoring platform, and then obtains the real-time business data corresponding to each business.
[0310] 602. Call a data monitoring model corresponding to the real-time business data to monitor the real-time business data.
[0311] For different services, the data processing device calls corresponding data monitoring models to monitor the real-time service data.
[0312] In this embodiment, when determining the data threshold, the specific strategy can refer to the aforementioned Figure 4 The scheme described in the illustrated embodiment will not be described in detail here.
[0313] 603. Issue early warning when abnormal data appears.
[0314] When it is determined that the real-time business data has abnormal data, the data processing device sends an early warning prompt to the corresponding user, such as sending an email, a text message, or directly sounding an alarm.
[0315] The data processing device in this application is described in detail below. Figure 7 , Figure 7 This is a schematic diagram of an embodiment of a data processing device in an embodiment of the present application. The data processing device 20 includes:
[0316] Acquisition module 201, used to acquire historical business data;
[0317] Processing module 202, used to extract data features corresponding to the historical business data, where the data features are used to indicate data distribution of the historical business data; and determine a data monitoring model corresponding to the historical business data according to the data features;
[0318] The training module 203 is used to use the historical business data to train the model parameters and warning parameters of the data monitoring model. The warning parameters are evaluation thresholds for determining whether the business data is abnormal. The model parameters include but are not limited to model types and monitoring methods. The business data is detected for anomalies using the model parameters and the warning parameters.
[0319] In the embodiment of the present application, a data processing device is provided. With the above device, different monitoring models are used for different data distributions. At the same time, the data monitoring model can be trained based on historical data to obtain model parameters and warning parameters, so that the model parameters and warning parameters can be adjusted accordingly based on real data, and manual parameter setting is no longer required, so that data can be monitored and alarmed more accurately;
[0320] The processing module 202 is used to perform anomaly detection on the business data using the model parameters and the warning parameters.
[0321] Optionally, in the above Figure 7 On the basis of the corresponding embodiment, in another embodiment of the data processing device 20 provided in the embodiment of the present application,
[0322] The data characteristics of the historical business data at least include pulse data, sawtooth data, continuous fluctuation data and continuous stable data. The processing module 202 is specifically used to determine that the data monitoring model is a single-point threshold comparison model when the data characteristics of the historical business data are pulse data, and the early warning parameter is the overall upper and lower threshold range of the pulse data. The monitoring method of the single-point threshold comparison model is to take the value of the current data point and the overall threshold range of the pulse data for judgment;
[0323] When the data feature of the historical business data is sawtooth data, the data monitoring model is determined to be a single-point threshold comparison model, the warning parameter is a threshold range of different water levels, and the monitoring method of the single-point threshold comparison model is to take the value of the current data point and the threshold range corresponding to the water level at which it is located for judgment;
[0324] When the data characteristics of the historical business data are continuously fluctuating data or continuously stable data, the data monitoring model is determined to be a sudden increase or decrease score comparison model, the warning parameter is the threshold range of the sudden increase or decrease score, and the monitoring method of the sudden increase or decrease score comparison model is to take the previous N points of the current data point compared with the previous month to judge the sudden increase or decrease score.
[0325] In an embodiment of the present application, a data processing device is provided. With the above device, a data monitoring model can be trained according to historical business data to obtain customized model parameters and warning parameters, so that different data can be monitored differently, thereby more accurately monitoring and warning the data.
[0326] Optionally, in the above Figure 7 On the basis of the corresponding embodiment, in another embodiment of the data processing device 20 provided in the embodiment of the present application, the training module 203 is specifically used to use the historical business data to train to obtain the threshold range and / or upper and lower water levels and the model parameters of the data monitoring model, and use the threshold range and / or upper and lower water levels as the warning parameters.
[0327] In an embodiment of the present application, a data processing device is provided. By using the above device, different early warning parameters are trained for different data monitoring models, so that data can be monitored and warned more accurately.
[0328] Optionally, in the above Figure 7 On the basis of the corresponding embodiment, in another embodiment of the data processing device 20 provided in the embodiment of the present application,
[0329] The training module 203 is specifically used to pre-process the historical business data to obtain an intermediate data set; perform average processing on the intermediate data to obtain a first overall mean, and perform standard deviation processing on the intermediate data set to obtain a first standard deviation;
[0330] Performing normal fitting on the first population mean and the first standard deviation to obtain a first threshold range;
[0331] Deleting the outliers beyond the first threshold to obtain a second difference value set;
[0332] Performing an average process on the second difference set to obtain a second population mean, and performing a standard deviation process on the second difference set to obtain a second standard deviation;
[0333] Performing normal fitting on the second population mean and the second standard deviation to obtain a second threshold range;
[0334] Deleting the outliers beyond the second threshold to obtain a third difference value set;
[0335] Repeat the above operation until there are no abnormal points to obtain the target threshold range, and use the target threshold range as the threshold range.
[0336] In an embodiment of the present application, a data processing device is provided. Using the above device, a normal convergence constant threshold algorithm is used to train different threshold ranges corresponding to different data features. Different data monitoring models are obtained through customized training, which can monitor and warn data more accurately. At the same time, the threshold range is trained according to historical data, so that the threshold range can be automatically and dynamically adjusted, thereby saving labor.
[0337] Optionally, in the above Figure 7 On the basis of the corresponding embodiment, in another embodiment of the data processing device 20 provided in the embodiment of the present application,
[0338] The training module 203 is used to obtain a set of predicted values and a set of true values corresponding to the historical business data;
[0339] Performing difference processing on the predicted value set and the true value set to obtain a first difference value set, and using the first difference value set as the intermediate data set;
[0340] or,
[0341] The training module 203 is used to take the real value set corresponding to the historical business data as the intermediate data set;
[0342] or,
[0343] The training module 203 is used to obtain a set of sudden rise and fall scores corresponding to the historical business data, and use the set of sudden rise and fall scores as the intermediate data set. In an embodiment of the present application, a data processing device is provided. Using the above device, a normal convergence constant threshold algorithm is used for training to obtain different threshold ranges corresponding to different data features. Different data monitoring models are obtained through customized training, which can monitor and warn the data more accurately. At the same time, the threshold range is trained according to the historical data, so that the threshold range can be automatically and dynamically adjusted, thereby saving manpower.
[0344] Optionally, in the above Figure 7 On the basis of the corresponding embodiment, in another embodiment of the data processing device 20 provided in the embodiment of the present application,
[0345] The training module 203 is specifically used to sort the historical business data according to the order of real values from small to large to obtain a first data column;
[0346] Taking the Kth data point in the first data column as a dividing point, the first data column is divided into a second data column and a third data column, where K is a positive integer greater than 1;
[0347] Performing variance processing on the second data column and the third data column respectively to obtain a first variance value and a second variance value;
[0348] Summing the first variance value and the second variance value to obtain the statistic of the K-th data point;
[0349] Repeat the above operation to obtain a set of statistics corresponding to the first data column;
[0350] The first data column is split based on the data point corresponding to the minimum statistic in the statistic set to obtain a first data column and a second data column corresponding to the historical business data, wherein the minimum value of the first data column is greater than the maximum value of the second data column; the water level of the first data column is determined as the upper water level of the historical business data, and the water level of the second data column is determined as the lower water level of the historical business data.
[0351] In the embodiment of the present application, a data processing device is provided. The above device is used to describe the discrete degree of historical data by using variance segmentation, so that the historical data is segmented according to the water level, and in the actual early warning process, the accuracy of the predicted value is improved, so that data monitoring and early warning can be achieved more accurately. At the same time, the upper and lower water levels are trained according to the historical data, and the upper and lower water levels can be automatically adjusted according to the actual situation, thereby saving labor.
[0352] Optionally, in the above Figure 7 On the basis of the corresponding embodiment, in another embodiment of the data processing device 20 provided in the embodiment of the present application, the processing module 202 is further used to perform data smoothing processing on the historical business data to remove the trend characteristics of the historical business data to obtain the first data, and the first data retains the fluctuation characteristics of the historical business data;
[0353] Performing panel data processing on the first data to obtain second data;
[0354] The second data is processed based on variance segmentation to remove abnormal points to obtain third data, where the third data is one-dimensional time series data;
[0355] Time segmenting the third data by using odd-even group cross validation to obtain at least two data segments of the historical service data;
[0356] The training module 203 is also used to use the at least two data segments to respectively train and obtain at least two groups of model parameters and warning parameters of the data monitoring model.
[0357] In an embodiment of the present application, a data processing device is provided. By using the above device, the distribution of historical data in different time periods is processed in segments using an even-odd group cross-validation method, and the corresponding model parameters and warning parameters are trained, so that data monitoring and warning can be achieved more accurately.
[0358] Optionally, in the above Figure 7 On the basis of the corresponding embodiment, in another embodiment of the data processing device 20 provided by the embodiment of the present application, when the data feature of the historical business data is continuous fluctuation data or continuous stable data, the warning parameter also includes a year-on-year threshold range parameter, and the monitoring method of the sudden rise and fall score comparison model is to take the previous X points of the current data point compared with the previous month to judge the sudden rise and fall score, and take the year-on-year Y points of the current data point to judge the year-on-year threshold range;
[0359] or,
[0360] When the data characteristics of the historical business data are continuously fluctuating data or continuously stable data, the warning parameters also include a year-on-year threshold range parameter and a holiday threshold range parameter. The monitoring method of the sudden increase or decrease score comparison model is to take the previous X points of the current data point compared with the previous month to judge the sudden increase or decrease score, and take the year-on-year Y points of the current data point to judge the year-on-year threshold range, and take the year-on-year Z points of the holidays to judge the holiday threshold range, where X, Y and Z are positive integers.
[0361] In an embodiment of the present application, a data processing device is provided. The device is used to perform multi-dimensional monitoring and comparison on continuous data, reduce the situation of false warnings and missed warnings, and thus achieve more accurate data monitoring and warnings.
[0362] Optionally, in the above Figure 7 On the basis of the corresponding embodiment, in another embodiment of the data processing device 20 provided in the embodiment of the present application,
[0363] The processing module 202 is specifically used to obtain a set of change points corresponding to N consecutive data points in the historical business data, where the N consecutive data points are reference data points when predicting a target data point, and the target data point is the first data point after the Nth data point in the N consecutive data points, where N is a positive integer;
[0364] Segmenting the N consecutive data points according to each change point in the change point set to obtain at least one data segment;
[0365] Performing linear regression processing on the last data segment of the at least one data segment to obtain a predicted value of the target data point;
[0366] Obtain a difference between the predicted value of the target data point and the true value of the target data point;
[0367] Repeat the above operation to obtain the difference value set corresponding to the historical business data;
[0368] The difference set is processed by kernel density estimation algorithm and Gaussian mixture model to obtain corresponding data distribution information;
[0369] The data characteristics corresponding to the historical business data are determined according to the data distribution information.
[0370] In the embodiment of the present application, a data processing device is provided. The above device is used to predict the historical data using a sliding window and water level segmentation, and then the data distribution of the historical data is determined according to the difference between the predicted value and the true value, so that data feature extraction can be accurately realized, so that the customization of the data monitoring model can be more accurately realized, and then more accurate data monitoring and early warning can be realized.
[0371] Optionally, in the above Figure 7 On the basis of the corresponding embodiment, in another embodiment of the data processing device 20 provided in the embodiment of the present application,
[0372] The processing module 202 is specifically configured to determine that the data feature of the historical service data is pulse data when the data distribution information indicates that the difference in the difference set conforms to the white noise distribution;
[0373] If the data distribution information indicates that the difference in the difference set satisfies at least two distribution types and the at least two distribution types are arranged at intervals and the duration exceeds a threshold, it is determined that the data feature of the historical business data is sawtooth data;
[0374] If the data distribution information indicates that the differences in the difference set are at least two normal distributions, it is determined that the data feature of the historical business data is continuous fluctuation data;
[0375] When the data distribution information indicates that the differences in the difference set are normally distributed, it is determined that the data feature of the historical business data is continuous and stable data.
[0376] In an embodiment of the present application, a data processing device is provided. By using the above device, data feature extraction can be accurately implemented, so that the customization of the data monitoring model can be more accurately implemented, thereby achieving more accurate data monitoring and early warning.
[0377] Optionally, in the above Figure 7 On the basis of the corresponding embodiment, in another embodiment of the data processing device 20 provided in the embodiment of the present application,
[0378] The processing module 202 is specifically used to perform denoising on the last data segment of the at least one data segment, and delete abnormal data points in the last data segment to obtain a smooth data segment;
[0379] Perform linear regression on the smoothed data segment to obtain the predicted value of the target data point.
[0380] In an embodiment of the present application, a data processing device is provided. The above device is used to perform denoising on the data and delete abnormal data points, thereby reducing the impact of abnormal data points on the prediction, improving the accuracy of the prediction value, and further more accurately extracting data features.
[0381] Optionally, in the above Figure 7 On the basis of the corresponding embodiment, in another embodiment of the data processing device 20 provided in the embodiment of the present application,
[0382] The processing module 202 is specifically configured to obtain a first score of the Ith data point among the N consecutive data points, wherein the first score is an evaluation score indicating whether the Ith data point has a sudden rise or fall trend, wherein if the first score is greater than 0, it indicates that the Ith data point has a sudden rise trend, and if the first score is less than 0, it indicates that the Ith data point has a sudden fall trend;
[0383] Determine a classification result of the I-th data point according to the first score, where the classification result is used to indicate whether the I-th data point is a change point;
[0384] When the classification result indicates that the I-th data point is a change point, record the I-th data point as a change point;
[0385] Repeat the above operation, traverse the N consecutive data points to obtain the candidate change point set corresponding to the N consecutive data points;
[0386] Using the candidate change point set to segment the N consecutive data points to obtain at least one data segment;
[0387] Determining water level drop and slope change information of the at least one data segment;
[0388] The change point set is obtained by merging each change point in the candidate change point set according to the water level drop and the slope change information.
[0389] In an embodiment of the present application, a data processing device is provided. The above device is used to determine the change points of historical data using the month-on-month sudden increase and decrease scores, thereby realizing segmented processing of historical data, reducing the impact of fluctuations on predictions, improving the accuracy of prediction values, and further extracting data features more accurately.
[0390] Optionally, in the above Figure 7On the basis of the corresponding embodiment, in another embodiment of the data processing device 20 provided in the embodiment of the present application,
[0391] The processing module 202 is specifically used to use denoising to delete abnormal values from the first data point to the (I-1)th data point in the continuous N data points to obtain smoothed data and to delete the first data after the maximum value and the minimum value from the first data point to the (I-1)th data point;
[0392] Using linear regression processing to obtain a set of predicted values corresponding to the smoothed data and a predicted value of the I-th data point;
[0393] Perform residual processing on the predicted value set of the smoothed data and the true value set of the smoothed data to obtain the residual standard deviation;
[0394] The first score is determined according to the residual standard deviation, the true value of the I-th data point, and the predicted value of the I-th data point.
[0395] In an embodiment of the present application, a data processing device is provided. The above device is used to calculate the sudden rise and fall trend of the current data point relative to the month-on-month data using the residual standard deviation, true value and predicted value of the month-on-month data, so as to determine whether the current data point is a change point, so that the trend direction of different data points can be truly displayed, so that data features can be extracted more accurately.
[0396] Optionally, in the above Figure 7 On the basis of the corresponding embodiment, in another embodiment of the data processing device 20 provided in the embodiment of the present application, the processing module 202 is specifically used to obtain the current segment length corresponding to the I-th data point;
[0397] When the current segment length is less than 2, the sudden rise and fall score and the current segment length are cleared;
[0398] If the current segment length is greater than or equal to 2, the first score is greater than or equal to the upper limit, or less than or equal to the lower limit, and there is no change point before the I-th data point or the sudden increase or decrease score after the change point is 0, then the position of the I-th data point and the first score are recorded, and the current segment length is updated to 1;
[0399] When the current segment length is greater than or equal to 2, the first score is greater than or equal to the upper limit, or less than or equal to the lower limit, and the sign of the sudden increase or decrease score recorded before the I data point is the same as the sign of the first score, the current segment length is increased by 1 and the I-1 data point is determined as a change point;
[0400] When the current segment length is greater than or equal to 2, the first score is greater than or equal to the upper limit, or less than or equal to the lower limit, and the sign of the sudden increase or decrease score recorded before the first data point is different from the sign of the first score, the position of the first data point and the first score are recorded, and the current segment length is updated to 1;
[0401] If the current segment length is greater than or equal to 2, the first score is less than the upper limit and greater than the lower limit, the sudden rise and fall score recorded before the first data point is greater than or equal to the upper limit, and the value of the first data point is greater than the maximum value of the first data, then the current segment length is updated to 1;
[0402] If the current segment length is greater than or equal to 2, the first score is less than the upper limit and greater than the lower limit, the sudden increase or decrease score recorded before the first data point is less than or equal to the lower limit, and the value of the first data point is less than the minimum value of the first data, then the current segment length is updated to 1;
[0403] If the current segment length is greater than or equal to 2, the first score is less than the upper limit and greater than the lower limit, the sudden rise and fall score recorded before the first data point is less than the upper limit, or the value of the first data point is less than or equal to the maximum value of the first data, then the sudden rise and fall score and the current segment length are updated to 0;
[0404] If the current segment length is greater than or equal to 2, the first score is less than the upper limit value and greater than the lower limit value, the sudden increase or decrease score recorded before the I data point is greater than the lower limit value, or the value of the I data point is greater than or equal to the minimum value of the first data, then the sudden increase or decrease score and the current segment length are updated to 0.
[0405] In an embodiment of the present application, a data processing device is provided. The above device is used to detect each data point using the sudden rise and fall score data, thereby determining the set of change points in the historical data, and implementing segmented processing of the historical data, so that the predicted value obtained during linear regression processing is closer to the direct value, thereby better extracting data features.
[0406] Optionally, in the above Figure 7 On the basis of the corresponding embodiment, in another embodiment of the data processing device 20 provided in the embodiment of the present application, the processing module 202 is specifically used to perform difference processing on the real values of adjacent data points of the first data segment in the at least one data segment to obtain the first interval data;
[0407] Performing denoising on the first interval data to obtain second interval data;
[0408] Performing an average process on the second interval data to obtain a third overall mean, and performing a difference process on each interval data in the second interval data set to obtain a third standard deviation;
[0409] Performing normal fitting on the third population mean and the third standard deviation to obtain a third threshold range;
[0410] Deleting abnormal points in the second interval data according to the third threshold range to obtain third interval data;
[0411] Obtaining a data spacing standard statistic according to the third interval data, wherein the data spacing standard statistic is used to indicate a basic magnitude of a data segment interval;
[0412] Compressing the first to the first data segment according to the data interval standard statistic to obtain a first compressed number sequence;
[0413] Repeat the above operation on other data segments in the at least one data segment to obtain at least one compressed sequence;
[0414] The at least one compressed data column is subjected to linear regression processing to obtain a compressed slope set, and the at least one compressed data column is subjected to mean processing to obtain a mean set, and the compressed slope set is used as the slope change information, and the mean set is used as the water level drop.
[0415] In an embodiment of the present application, a data processing device is provided. The above device is used to compress historical data, thereby reducing the amount of calculation of water level drop and slope change information, and at the same time, the candidate change points are merged through the slope change information, so that the segmentation of historical data is more holistic and the accuracy of data feature extraction is improved.
[0416] Optionally, in the above Figure 7 On the basis of the corresponding embodiment, in another embodiment of the data processing device 20 provided by the embodiment of the present application, the candidate change point set includes k candidate change points, the compression slope set includes k+1 compression slopes, the mean set includes k+1 means, and the processing module 202 is specifically used to traverse the first compression slope to the kth compression slope in the compression slope set, and when the absolute value of the i-th compression slope is greater than or equal to 1 and i is equal to k, the k-th candidate change point is attributed to the change point set;
[0417] When the absolute value of the i-th compression slope is greater than or equal to 1, i is not equal to k, the i-th compression slope is greater than 0, and the i-1-th mean is less than the i+1-th mean, the i-th candidate change point is included in the change point set;
[0418] When the absolute value of the i-th compression slope is greater than or equal to 1, i is not equal to k, the i-th compression slope is less than 0, and the i-1-th mean is greater than the i+1-th mean, the i-th candidate change point is included in the change point set;
[0419] When the absolute value of the i-th compression slope is less than 1 and i is equal to k, obtaining a first data segment of a k-1-th candidate change point and a k-th candidate change point, and a second data segment after the k-th change point;
[0420] Obtain a first maximum value and a first minimum value in the first data segment after removing the extreme value, and a second maximum value and a second minimum value in the second data segment after removing the extreme value;
[0421] When the first maximum value is smaller than the second minimum value or the first minimum value is larger than the second maximum value, the kth candidate change point is assigned to the change point set.
[0422] In an embodiment of the present application, a data processing device is provided. By using the above device, multiple change points are merged according to the slope change, so that the segmentation of historical data is more integrated and the accuracy of data feature extraction is improved.
[0423] Optionally, in the above Figure 7 On the basis of the corresponding embodiment, in another embodiment of the data processing device 20 provided in the embodiment of the present application, the acquisition module 201 is further used to acquire real-time business data;
[0424] The processing module 202 is further used to call the data monitoring model corresponding to the real-time business data to monitor the real-time business data;
[0425] When abnormal data occurs, an early warning will be issued.
[0426] In the embodiment of the present application, a data processing device is provided. With the above device, different monitoring models are used for different data distributions. At the same time, the data monitoring model can be trained based on historical data to obtain model parameters and warning parameters, so that the model parameters and warning parameters can be adjusted accordingly according to the real data, and there is no need to manually set the parameters, so that the data can be monitored and alarmed more accurately.
[0427] The data processing device provided in this application can be used in a server, see Figure 8 , Figure 8: is a schematic diagram of a server structure provided in an embodiment of the present application. The server 300 may have relatively large differences due to different configurations or performances, and may include one or more central processing units (CPU) 322 (for example, one or more processors) and memory 332, and one or more storage media 330 (for example, one or more mass storage devices) storing application programs 342 or data 344. Among them, the memory 332 and the storage medium 330 can be short-term storage or permanent storage. The program stored in the storage medium 330 may include one or more modules (not shown in the figure), and each module may include a series of instruction operations on the server. Furthermore, the central processing unit 322 can be configured to communicate with the storage medium 330 to execute a series of instruction operations in the storage medium 330 on the server 300.
[0428] The server 300 may also include one or more power supplies 326, one or more wired or wireless network interfaces 350, one or more input and output interfaces 358, and / or one or more operating systems 341, such as Windows Server 2000. TM , Mac OS X TM , Unix TM ,Linux TM , FreeBSD TM etc.
[0429] The steps performed by the data processing device in the above embodiment can be based on the Figure 8 The server structure shown.
[0430] The data processing device provided in this application can be used in terminal equipment. Fig. 9 For the sake of convenience, only the parts related to the embodiments of the present application are shown. For specific technical details not disclosed, please refer to the method part of the embodiments of the present application. In the embodiments of the present application, the terminal device is a smart phone as an example for explanation:
[0431] Fig. 9 The block diagram shows a partial structure of a smart phone related to the terminal device provided in the embodiment of the present application. Fig. 9 The smart phone includes: a radio frequency (RF) circuit 410, a memory 420, an input unit 430, a display unit 440, a sensor 450, an audio circuit 460, a wireless fidelity (WiFi) module 470, a processor 480, and a power supply 490. Those skilled in the art will appreciate that Fig. 9The structure of the smartphone shown in the figure does not constitute a limitation of the smartphone, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.
[0432] Combine the following Fig. 9 A detailed introduction to the various components of a smartphone:
[0433] The RF circuit 410 can be used for receiving and sending signals during the process of sending and receiving information or making calls. In particular, after receiving the downlink information of the base station, it is sent to the processor 480 for processing; in addition, the designed uplink data is sent to the base station. Generally, the RF circuit 410 includes but is not limited to an antenna, at least one amplifier, a transceiver, a coupler, a low noise amplifier (LNA), a duplexer, etc. In addition, the RF circuit 410 can also communicate with the network and other devices through wireless communication. The above-mentioned wireless communication can use any communication standard or protocol, including but not limited to the global system of mobile communication (GSM), general packet radio service (GPRS), code division multiple access (CDMA), wideband code division multiple access (WCDMA), long term evolution (LTE), email, short messaging service (SMS), etc.
[0434] The memory 420 can be used to store software programs and modules. The processor 480 executes various functional applications and data processing of the smartphone by running the software programs and modules stored in the memory 420. The memory 420 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area may store data created according to the use of the smartphone (such as audio data, a phone book, etc.), etc. In addition, the memory 420 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage devices.
[0435] The input unit 430 can be used to receive input digital or character information, and to generate key signal input related to the user settings and function control of the smart phone. Specifically, the input unit 430 may include a touch panel 431 and other input devices 432. The touch panel 431, also known as a touch screen, can collect the user's touch operation on or near it (such as the user's operation on the touch panel 431 or near the touch panel 431 using any suitable object or accessory such as a finger, stylus, etc.), and drive the corresponding connection device according to a pre-set program. Optionally, the touch panel 431 may include two parts: a touch detection device and a touch controller. Among them, the touch detection device detects the user's touch orientation, detects the signal brought by the touch operation, and transmits the signal to the touch controller; the touch controller receives the touch information from the touch detection device, converts it into contact coordinates, and then sends it to the processor 480, and can receive and execute commands sent by the processor 480. In addition, the touch panel 431 can be implemented in various types such as resistive, capacitive, infrared, and surface acoustic waves. In addition to the touch panel 431, the input unit 430 may also include other input devices 432. Specifically, other input devices 432 may include but are not limited to one or more of a physical keyboard, function keys (such as volume control keys, switch keys, etc.), a trackball, a mouse, a joystick, etc.
[0436] The display unit 440 can be used to display information input by the user or information provided to the user and various menus of the smartphone. The display unit 440 may include a display panel 441. Optionally, the display panel 441 may be configured in the form of a liquid crystal display (LCD), an organic light-emitting diode (OLED), or the like. Further, the touch panel 431 may cover the display panel 441. When the touch panel 431 detects a touch operation on or near it, it is transmitted to the processor 480 to determine the type of touch event. Subsequently, the processor 480 provides a corresponding visual output on the display panel 441 according to the type of touch event. Although in Fig. 9 In the embodiment, the touch panel 431 and the display panel 441 are used as two independent components to implement the input and output functions of the smartphone, but in some embodiments, the touch panel 431 and the display panel 441 can be integrated to implement the input and output functions of the smartphone.
[0437] The smartphone may also include at least one sensor 450, such as a light sensor, a motion sensor, and other sensors. Specifically, the light sensor may include an ambient light sensor and a proximity sensor, wherein the ambient light sensor may adjust the brightness of the display panel 441 according to the brightness of the ambient light, and the proximity sensor may turn off the display panel 441 and / or the backlight when the smartphone is moved to the ear. As a type of motion sensor, the accelerometer sensor can detect the magnitude of acceleration in all directions (generally three axes), and can detect the magnitude and direction of gravity when stationary. It can be used for applications that identify the posture of the smartphone (such as horizontal and vertical screen switching, related games, magnetometer posture calibration), vibration recognition related functions (such as pedometer, tapping), etc.; as for other sensors that can be configured in smartphones, such as gyroscopes, barometers, hygrometers, thermometers, infrared sensors, etc., they will not be repeated here.
[0438] The audio circuit 460, the speaker 461, and the microphone 462 can provide an audio interface between the user and the smartphone. The audio circuit 460 can transmit the received audio data to the speaker 461 after converting the received audio data into an electrical signal, which is converted into a sound signal for output; on the other hand, the microphone 462 converts the collected sound signal into an electrical signal, which is received by the audio circuit 460 and converted into audio data, and then the audio data is output to the processor 480 for processing, and then sent to another smartphone through the RF circuit 410, or the audio data is output to the memory 420 for further processing.
[0439] WiFi is a short-range wireless transmission technology. Smartphones can help users send and receive emails, browse web pages, and access streaming media through WiFi module 470, which provides users with wireless broadband Internet access. Fig. 9 A WiFi module 470 is shown, but it is understandable that it is not an essential component of the smartphone and can be omitted as needed without changing the essence of the invention.
[0440] Processor 480 is the control center of the smartphone, which uses various interfaces and lines to connect various parts of the entire smartphone, and executes various functions of the smartphone and processes data by running or executing software programs and / or modules stored in memory 420, and calling data stored in memory 420, thereby monitoring the smartphone as a whole. Optionally, processor 480 may include one or more processing units; optionally, processor 480 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface, and application programs, and the modem processor mainly processes wireless communications. It is understandable that the above-mentioned modem processor may not be integrated into processor 480.
[0441] The smart phone also includes a power source 490 (such as a battery) for supplying power to various components. Optionally, the power source can be logically connected to the processor 480 through a power management system, thereby managing functions such as charging, discharging, and power consumption management through the power management system.
[0442] Although not shown, the smartphone may also include a camera, a Bluetooth module, etc., which will not be described in detail here.
[0443] The steps performed by the data processing device in the above embodiment can be based on the Fig. 9 The terminal device structure shown.
[0444] A computer-readable storage medium is also provided in an embodiment of the present application. The computer-readable storage medium stores a computer program, which, when executed on a computer, enables the computer to execute the methods described in the aforementioned embodiments.
[0445] The present application also provides a computer program product including a program, which, when executed on a computer, enables the computer to execute the methods described in the aforementioned embodiments.
[0446] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.
[0447] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.
[0448] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
[0449] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.
[0450] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk and other media that can store program code.
[0451] As described above, the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.
Claims
1. A data processing method, characterized in that: include: Obtain historical business data; Extracting data features corresponding to the historical business data, where the data features are used to indicate data distribution of the historical business data; Determining a data monitoring model corresponding to the historical business data according to the data characteristics; The model parameters and early warning parameters of the data monitoring model are obtained by training the historical business data, wherein the early warning parameters are evaluation thresholds for determining whether the business data is abnormal, and the model parameters include but are not limited to model type and monitoring method; The business data is subjected to anomaly detection using the model parameters and the warning parameters.
2. The method according to claim 1, characterized in that The data characteristics of the historical business data at least include pulse data, sawtooth data, continuous fluctuation data and continuous stable data, and the data monitoring model corresponding to the historical business data is determined according to the data characteristics, including: When the data feature of the historical business data is pulse-type data, the data monitoring model is determined to be a single-point threshold comparison model, the early warning parameter is the overall threshold range of the pulse-type data, and the monitoring method of the single-point threshold comparison model is to take the value of the current data point and the overall threshold range of the pulse-type data for judgment; When the data feature of the historical business data is sawtooth data, the data monitoring model is determined to be a single-point threshold comparison model, the warning parameters are the threshold ranges and durations of different water levels of the sawtooth data, and the monitoring method of the single-point threshold comparison model is to take the value of the current data point and the threshold range corresponding to the water level at which it is located for judgment; When the data characteristics of the historical business data are continuously fluctuating data or continuously stable data, the data monitoring model is determined to be a sudden increase or decrease score comparison model, the warning parameter is a threshold range of the sudden increase or decrease score, and the monitoring method of the sudden increase or decrease score comparison model is to take the previous N points of the current data point compared with the previous data point to determine the sudden increase or decrease score.
3. The method according to claim 2, characterized in that The model parameters and warning parameters of the data monitoring model obtained by training the historical business data include: The threshold range and / or upper and lower water levels of the data monitoring model and the model parameters are obtained by training with the historical business data, and the threshold range and / or upper and lower water levels are used as the early warning parameters.
4. The method according to claim 3, characterized in that The threshold range of the data monitoring model obtained by training the historical business data includes: Preprocessing the historical business data to obtain an intermediate data set; Performing an average process on the intermediate data set to obtain a first population mean, and performing a standard deviation process on the intermediate data set to obtain a first standard deviation; Performing normal fitting on the first population mean and the first standard deviation to obtain a first threshold range; Deleting outliers beyond the first threshold to obtain a second difference value set; Performing an average process on the second difference value set to obtain a second population mean, and performing a standard deviation process on the second difference value set to obtain a second standard deviation; Performing normal fitting on the second population mean and the second standard deviation to obtain a second threshold range; Deleting outliers beyond the second threshold to obtain a third difference value set; The above operation is repeated until there are no abnormal points, so as to obtain a target threshold range, and the target threshold range is used as the threshold range.
5. The method according to claim 4, characterized in that The preprocessing of the historical business data to obtain an intermediate data set includes: Obtaining a set of predicted values and a set of true values corresponding to the historical business data; Performing difference processing on the predicted value set and the true value set to obtain a first difference value set, and using the first difference value set as the intermediate data set; or, Taking the real value set corresponding to the historical business data as the intermediate data set; or, A sudden increase or decrease score set corresponding to the historical business data is obtained, and the sudden increase or decrease score set is used as the intermediate data set.
6. The method according to claim 3, characterized in that The step of using the historical business data to train and obtain the upper and lower water levels of the data monitoring model includes: Sorting the historical business data according to the real value from small to large to obtain a first data column; The first data column is divided into a second data column and a third data column by taking the Kth data point in the first data column as a dividing point, where K is a positive integer greater than 1; Performing variance processing on the second data column and the third data column respectively to obtain a first variance value and a second variance value; Summing the first variance value and the second variance value to obtain the statistic of the K-th data point; Repeat the above operation to obtain a set of statistics corresponding to the first data column; The first data column is segmented based on the data point corresponding to the minimum statistic in the statistic set to obtain a first data column and a second data column corresponding to the historical business data, wherein the minimum value of the first data column is greater than the maximum value of the second data column; The water level of the first data column is determined as the upper water level of the historical business data, and the water level of the second data column is determined as the lower water level of the historical business data.
7. The method according to claim 1, characterized in that Before obtaining the model parameters and early warning parameters of the data monitoring model by training with the historical business data, the method further includes: Performing data smoothing processing on the historical business data to eliminate trend characteristics of the historical business data to obtain first data, wherein the first data retains fluctuation characteristics of the historical business data; Performing panel data processing on the first data to obtain second data; Processing the second data based on variance segmentation to remove abnormal points to obtain third data, wherein the third data is one-dimensional time series data; Time segmenting the third data using even-odd group cross validation to obtain at least two data segments of the historical service data; The model parameters and warning parameters of the data monitoring model obtained by training the historical business data include: At least two groups of model parameters and early warning parameters of the data monitoring model are obtained by training respectively using the at least two data segments.
8. The method according to claim 2, characterized in that: When the data characteristics of the historical business data are continuous fluctuating data or continuous stable data, the warning parameters also include a year-on-year threshold range parameter, and the monitoring method of the sudden increase and decrease score comparison model is to take the previous X points of the current data point compared with the previous month to judge the sudden increase and decrease score, and take the year-on-year Y points of the current data point to judge the year-on-year threshold range; or, When the data characteristics of the historical business data are continuously fluctuating data or continuously stable data, the warning parameters also include a year-on-year threshold range parameter and a holiday threshold range parameter, and the monitoring method of the sudden increase or decrease score comparison model is to take the previous X points of the current data point compared with the previous month to judge the sudden increase or decrease score, and take the year-on-year Y points of the current data point to judge the year-on-year threshold range, and take the year-on-year Z points of the holidays to judge the holiday threshold range, and X, Y and Z are positive integers.
9. The method according to any one of claims 1 to 8, characterized in that Determining the data features corresponding to the historical business data includes: Obtain a set of change points corresponding to N consecutive data points in the historical business data, where the N consecutive data points are reference data points when predicting a target data point, and the target data point is the first data point after the Nth data point in the N consecutive data points, where N is a positive integer; Segmenting the N consecutive data points according to each change point in the change point set to obtain at least one data segment; Performing linear regression processing on the last data segment of the at least one data segment to obtain a predicted value of the target data point; Obtaining a difference value based on the predicted value of the target data point and the true value of the target data point; Repeat the above operation to obtain a difference value set corresponding to the historical business data; Performing kernel density estimation algorithm and Gaussian mixture model processing on the difference set to obtain corresponding data distribution information; Data features corresponding to the historical business data are determined according to the data distribution information.
10. The method according to claim 9, characterized in that Determining the data features corresponding to the historical business data according to the data distribution information includes: When the data distribution information indicates that the difference in the difference set conforms to a white noise distribution, determining that the data feature of the historical service data is pulse data; If the data distribution information indicates that the difference in the difference set satisfies at least two distribution types, and the at least two distribution types are arranged at intervals and the duration thereof satisfies a threshold, it is determined that the data feature of the historical business data is sawtooth data; If the data distribution information indicates that the differences in the difference set are at least two normal distributions, it is determined that the data feature of the historical business data is continuous fluctuation data; If the data distribution information indicates that the difference values in the difference set are normally distributed, it is determined that the data feature of the historical business data is continuous and stable data.
11. The method according to claim 9, characterized in that The performing linear regression processing on the last data segment of the at least one data segment to obtain the predicted value of the target data point comprises: Performing denoising processing on the last data segment of the at least one data segment, and deleting abnormal data points in the last data segment to obtain a smooth data segment; Linear regression is performed on the smoothed data segment to obtain a predicted value of the target data point.
12. The method according to claim 9, characterized in that The obtaining of a set of change points corresponding to N consecutive data points in the historical business data comprises: Obtaining a first score of the I-th data point among the N consecutive data points, the first score being an evaluation score indicating whether the I-th data point has a sudden rise or fall trend, wherein if the first score is greater than 0, it indicates that the I-th data point has a sudden rise trend, and if the first score is less than 0, it indicates that the I-th data point has a sudden fall trend, and I is a positive integer; Determine a classification result of the I-th data point according to the first score, wherein the classification result is used to indicate whether the I-th data point is a change point; When the classification result indicates that the I-th data point is a change point, record the I-th data point as a change point; Repeat the above operation to traverse the N consecutive data points to obtain a set of candidate change points corresponding to the N consecutive data points; Segmenting the N consecutive data points using the candidate change point set to obtain at least one data segment; Determining water level drop and slope change information of the at least one data segment; The change point set is obtained by merging each change point in the candidate change point set according to the water level difference and the slope change information.
13. The method according to claim 12, characterized in that The obtaining a first score of the Ith data point among the N consecutive data points comprises: Deleting abnormal values from the first data point to the (I-1)th data point among the N consecutive data points by using a denoising process to obtain smoothed data and deleting the first data after the maximum value and the minimum value from the first data point to the (I-1)th data point; Obtaining a set of predicted values corresponding to the smoothed data and a predicted value of the I-th data point by using linear regression processing; Performing residual processing on the predicted value set of the smoothed data and the true value set of the smoothed data to obtain a residual standard deviation; The first score is determined based on the residual standard deviation, the true value of the Ith data point, and the predicted value of the Ith data point.
14. The method according to claim 13, characterized in that Determining the classification result of the I-th data point according to the first score includes: Obtaining the current segment length corresponding to the I-th data point; When the current segment length is less than 2, clear the sudden rise and fall score and the current segment length; If the current segment length is greater than or equal to 2, the first score is greater than or equal to the upper limit, or less than or equal to the lower limit, and there is no change point before the I-th data point or the sudden increase or decrease score after the change point is 0, then the position of the I-th data point and the first score are recorded, and the current segment length is updated to 1; When the current segment length is greater than or equal to 2, the first score is greater than or equal to the upper limit, or less than or equal to the lower limit, and the sign of the sudden increase or decrease score recorded before the I data point is the same as the sign of the first score, the current segment length is increased by 1 and the I-1 data point is determined as a change point; When the current segment length is greater than or equal to 2, the first score is greater than or equal to the upper limit, or less than or equal to the lower limit, and the sign of the sudden increase or decrease score recorded before the first data point is different from the sign of the first score, the position of the first data point and the first score are recorded, and the current segment length is updated to 1; If the current segment length is greater than or equal to 2, the first score is less than the upper limit and greater than the lower limit, the sudden increase or decrease score recorded before the first data point is greater than or equal to the upper limit, and the value of the first data point is greater than the maximum value of the first data, then the current segment length is updated to 1; If the current segment length is greater than or equal to 2, the first score is less than the upper limit and greater than the lower limit, the sudden increase or decrease score recorded before the first data point is less than or equal to the lower limit, and the value of the first data point is less than the minimum value of the first data, then the current segment length is updated to 1; If the current segment length is greater than or equal to 2, the first score is less than the upper limit and greater than the lower limit, the sudden rise and fall score recorded before the first data point is less than the upper limit, or the value of the first data point is less than or equal to the maximum value of the first data, then the sudden rise and fall score and the current segment length are updated to 0; If the current segment length is greater than or equal to 2, the first score is less than the upper limit value and greater than the lower limit value, the sudden increase or decrease score recorded before the I-th data point is greater than the lower limit value, or the value of the I-th data point is greater than or equal to the minimum value of the first data, then the sudden increase or decrease score and the current segment length are updated to 0.
15. The method according to claim 12, characterized in that Determining the water level drop and slope change information of the at least one data segment includes: Performing difference processing on real values of adjacent data points of a first data segment in the at least one data segment to obtain first interval data; Performing denoising on the first interval data to obtain second interval data; Performing an average process on the second interval data to obtain a third overall mean, and performing a difference process on each interval data in the second interval data set to obtain a third standard deviation; Performing normal fitting on the third population mean and the third standard deviation to obtain a third threshold range; Deleting abnormal points in the second interval data according to the third threshold range to obtain third interval data; Obtaining a data spacing standard statistic according to the third interval data, wherein the data spacing standard statistic is used to indicate a basic magnitude of a data segment interval; Compressing the first to the first data segment according to the data interval standard statistic to obtain a first compressed number sequence; Repeat the above operation on other data segments in the at least one data segment to obtain at least one compressed sequence; Linear regression processing is performed on the at least one compressed data column to obtain a compressed slope set, and mean processing is performed on the at least one compressed data column to obtain a mean set, the compressed slope set is used as the slope change information, and the mean set is used as the water level drop.
16. The method according to claim 11, characterized in that The candidate change point set includes k candidate change points, the compressed slope set includes k+1 compressed slopes, the mean value set includes k+1 means, where k is a positive integer, and the change point set is obtained by merging the change points in the candidate change point set according to the water level drop and the slope change information, including: Traversing the first compression slope to the kth compression slope in the compression slope set, and when the absolute value of the ith compression slope is greater than or equal to 1 and i is equal to k, assigning the kth candidate change point to the change point set; When the absolute value of the i-th compression slope is greater than or equal to 1, i is not equal to k, the i-th compression slope is greater than 0, and the i-1-th mean is less than the i+1-th mean, the i-th candidate change point is included in the change point set; When the absolute value of the i-th compression slope is greater than or equal to 1, i is not equal to k, the i-th compression slope is less than 0, and the i-1-th mean is greater than the i+1-th mean, the i-th candidate change point is included in the change point set; When the absolute value of the i-th compression slope is less than 1 and i is equal to k, obtaining a first data segment of a k-1-th candidate change point and a k-th candidate change point, and a second data segment after the k-th change point; Obtain a first maximum value and a first minimum value in the first data segment after removing extreme values, and a second maximum value and a second minimum value in the second data segment after removing extreme values; When the first maximum value is smaller than the second minimum value or the first minimum value is larger than the second maximum value, the kth candidate change point is assigned to the change point set.
17. The method according to any one of claims 1 to 8 or 10 to 16, characterized in that The method further comprises: Obtain real-time business data; Calling a data monitoring model corresponding to the real-time business data to monitor the real-time business data; When abnormal data occurs, an early warning will be issued.
18. A data processing device, characterized in that: include: Acquisition module, used to obtain historical business data; A processing module, used to extract data features corresponding to the historical business data, wherein the data features are used to indicate data distribution of the historical business data; and determine a data monitoring model corresponding to the historical business data according to the data features; A training module, used to train the historical business data to obtain model parameters and early warning parameters of the data monitoring model, wherein the early warning parameter is an evaluation threshold for determining whether the business data is abnormal, and the model parameters include but are not limited to model type and monitoring method; The processing module is used to perform anomaly detection on the business data using the model parameters and the warning parameters.
19. A computer device, characterized in that: include: Memory, processor, and bus system; Wherein, the memory is used to store programs; The processor is used to execute the program in the memory, and the processor is used to perform the method according to any one of claims 1 to 17 according to the instructions in the program code; The bus system is used to connect the memory and the processor so that the memory and the processor can communicate with each other.
20. A computer-readable storage medium comprising instructions, which, when executed on a computer, causes the computer to perform the method according to any one of claims 1 to 17.
Citation Information
Cited By
Abnormal detection and self-adaptive repair reconstruction method for time series data of coal mine sensor
CN121682377A
Coal mine sensor time series data anomaly detection and adaptive repair reconstruction method
CN121682377B
