Data processing method and device
By identifying the type of time series data and selecting a suitable detection algorithm, the problems of calculation complexity and time-consuming of timing data abnormal detection in the prior art are solved, and detection efficiency and resource utilization are improved.
Patent Information
- Application Number
- CN202510125325.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-01-26
- Publication Date
- 2025-05-13
AI Technical Summary
The prior art has problems of computational complexity and time-consuming in timing data abnormality detection, and it is difficult to find a balance between efficiency and computing resources.
By identifying the data type of the time sequence item data to be processed and determining the corresponding target data detection algorithm based on the data type, and using a detection algorithm that is more consistent with the data type for detection, the problems of low efficiency and high computational complexity caused by using one detection method to detect all data types are avoided.
It improves data detection efficiency, realizes a balance between calculation complexity and time-consuming, and is suitable for time-series data processing of different data types.
Smart Images
Figure CN119988461A_ABST
Abstract
Description
Technical Field
[0001] The embodiments of the present specification relate to the field of computer technology, and in particular, to a data processing method. Background Art
[0002] In the operation and maintenance scenario, anomaly detection is an important function. Anomaly detection aims to automatically discover abnormal fluctuations in the time series data of key indicators through algorithms, and provide decision-making basis for subsequent alarms, automatic stop losses, root cause analysis, etc. In data mining, anomaly detection is the identification of items, events, or observations that do not conform to expected patterns or other items in the data set.
[0003] Current anomaly detection methods are usually divided into decomposition detection algorithms and non-decomposition detection algorithms. Decomposition detection algorithms can handle complex seasonality and trends and provide rich statistical information, but their models are complex and require more parameter adjustments and more computing resources. Non-decomposition detection algorithms have good detection effects on some anomalies and high computational efficiency, but they require adjustment of model parameters and are not sensitive enough to certain types of anomalies and cannot be detected well. In the process of anomaly detection, a decomposition detection algorithm is usually used for time series data to decompose signals such as cycles, trends, and noise, and then determine whether it is abnormal based on trends and noise. Decomposition detection algorithms are usually iterative algorithms and usually require O(n 2 )(O(n) represents the time complexity of the algorithm, O(n n ) represents different degrees of time complexity) and above, so the calculation time is very long. However, it is difficult to detect anomalies superimposed on periodic signals without using decomposition detection algorithms. Therefore, in the process of detecting time series data, how to balance the calculation complexity and time consumption has become an urgent problem for technicians to solve. Summary of the invention
[0004] In view of this, an embodiment of this specification provides a data processing method. One or more embodiments of this specification also relate to a data processing device, an electronic device, a computer-readable storage medium and a computer program product to solve the technical defects existing in the prior art.
[0005] According to a first aspect of an embodiment of this specification, a data processing method is provided, including: Get the time series project data to be processed; Identify the data type of the time series project data to be processed, and determine the target data detection algorithm corresponding to the time series project data to be processed according to the data type; The time series project data to be processed is detected based on the target data detection algorithm to obtain a data detection result corresponding to the time series project data to be processed.
[0006] According to a second aspect of an embodiment of this specification, there is provided a data processing device, including: An acquisition module, configured to acquire time series project data to be processed; an identification module, configured to identify a data type of the time series project data to be processed, and determine a target data detection algorithm corresponding to the time series project data to be processed according to the data type; The detection module is configured to detect the time series project data to be processed based on the target data detection algorithm, and obtain a data detection result corresponding to the time series project data to be processed.
[0007] According to a third aspect of the embodiments of this specification, an electronic device is provided, including: Memory and processor; The memory is used to store computer programs / instructions, and the processor is used to execute the computer programs / instructions. When the computer programs / instructions are executed by the processor, the steps of the above method are implemented.
[0008] According to a fourth aspect of the embodiments of this specification, a computer-readable storage medium is provided, which stores a computer program / instruction, and the steps of the above method are implemented when the computer program / instruction is executed by a processor.
[0009] According to a fifth aspect of the embodiments of this specification, a computer program product is provided, comprising a computer program / instruction, which implements the steps of the above method when executed by a processor.
[0010] One or more embodiments of the present specification provide a data processing method, comprising acquiring time series project data to be processed; identifying a data type of the time series project data to be processed, and determining a target data detection algorithm corresponding to the time series project data to be processed based on the data type; detecting the time series project data to be processed based on the target data detection algorithm, and obtaining a data detection result corresponding to the time series project data to be processed.
[0011] Through the method provided in the embodiment of this specification, the data type of the time series project data to be processed is first determined, and the corresponding target data detection algorithm is selected according to the data type. The target data detection algorithm that is more suitable for the data type is used to detect the time series project data to be processed, avoiding the problems of low efficiency and increased computational complexity caused by using one data detection method to detect time series data of all data types. The data detection efficiency is improved, and different data detection algorithms are used for time series data of different data types, achieving a balance between computational complexity and time consumption in the overall data processing dimension. BRIEF DESCRIPTION OF THE DRAWINGS
[0012] Figure 1 is a flow chart of a data processing method provided by an embodiment of this specification; Figure 2 is a processing flow chart of a data processing method provided by an embodiment of this specification; Figure 3 is a structural schematic diagram of a data processing device provided by an embodiment of this specification; Figure 4 It is a structural block diagram of an electronic device provided by an embodiment of this specification. DETAILED DESCRIPTION
[0013] Many specific details are described in the following description to facilitate a full understanding of this specification. However, this specification can be implemented in many other ways than those described herein, and those skilled in the art can make similar generalizations without violating the connotation of this specification, so this specification is not limited to the specific implementation disclosed below.
[0014] The terms used in one or more embodiments of this specification are only for the purpose of describing specific embodiments, and are not intended to limit one or more embodiments of this specification. The singular forms of "a", "said" and "the" used in one or more embodiments of this specification and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used in one or more embodiments of this specification refers to and includes any or all possible combinations of one or more associated listed items.
[0015] It should be understood that although the terms first, second, etc. may be used to describe various information in one or more embodiments of this specification, this information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of one or more embodiments of this specification, the first may also be referred to as the second, and similarly, the second may also be referred to as the first. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".
[0016] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this manual are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of the relevant regions, and provide corresponding operation entrances for users to choose to authorize or refuse.
[0017] First, the terms involved in one or more embodiments of this specification are explained.
[0018] Anomaly detection: refers to the identification of items, events, or observations that do not conform to their patterns or other items in the data set. Anomalies are also called outliers, novelties, noise, deviations, exceptions, etc.
[0019] Periodicity: The periodicity of data refers to the characteristic of data repeating in time. Specifically, if a data f(t) satisfies f(t+T)=f(t), it means that the data is periodic data, and T is a period of the data.
[0020] Stationarity: Data stationarity refers to the property that the statistical properties of data remain unchanged over time. Depending on the degree of this property, the signal can be divided into two types: strictly stationary and wide stationary.
[0021] Strict stationarity means that all statistical characteristics of a random process do not change with time.
[0022] Wide stability: It means that the data expectation and variance in the random process are independent of time, and the autocovariance function is only related to the time interval.
[0023] Time complexity: used to describe the running time of an algorithm. It is a function of the length of the string representing the input value of the algorithm. Time complexity is usually expressed in big O notation, excluding the low-order terms and the leading coefficient of this function. Time complexity is usually expressed in O(n), O(n 2 )、O(n 3 )、O(n logn ) etc.
[0024] Decomposition detection algorithm: such as STL (Seasonal and Trend decomposition using Loess) algorithm and X-12-ARIMA algorithm. Decomposition detection algorithm decomposes time series data into three components, such as trend, seasonality and residual. By analyzing these components, it is easier to identify anomalies. The advantage of decomposition detection algorithm is that it can handle complex seasonality and trends and provide rich statistical information. However, its corresponding data processing model is more complex and requires more parameter adjustments. The computational complexity is O(n 2 ) and above.
[0025] STL algorithm: Seasonal and Trend decomposition using Loess, uses local regression method to decompose time series into trend, seasonality and residual, which is suitable for time series with complex periodicity and trend. The advantage is that it can handle nonlinear trend and periodicity, but the disadvantage is that the calculation complexity is high and it may be slow for large data sets.
[0026] X-12-ARIMA algorithm: Combines the ARIMA model and time series decomposition technology for seasonal adjustment and anomaly detection, and is suitable for the analysis of economic and financial data. The advantage is that it can handle complex seasonality and trends and provide rich statistical information. The disadvantage is that the model is complex and requires more parameter adjustments.
[0027] Non-decomposition detection algorithms: such as 3sigma, Isolation Forest, LSTM, etc. Non-decomposition detection algorithms do not rely on the structural characteristics of time series, but use other methods to detect anomalies. Non-decomposition detection algorithms have good detection effects on anomalies and high computational efficiency. However, the corresponding data processing model needs to adjust hyperparameters and may not be sensitive enough to certain types of anomalies. The computational complexity is O(n logn ).
[0028] 3sigma: Based on the normal distribution theory in statistics, this method assumes that time series data follows a normal distribution. By calculating the mean and standard deviation of the data, a threshold interval is defined, and values exceeding this interval are considered outliers. 3sigma is suitable for normally distributed data, simple and fast detection, and single-variable time series scenarios. The advantages are that it is simple to understand, efficient in calculation, and does not require a large number of parameter adjustments. The disadvantages are that it is sensitive to discrete values and cannot handle complex relationships.
[0029] Isolation Forest: An ensemble learning method based on decision trees that isolates outliers by randomly selecting features and separation points. It is suitable for high-dimensional data and large-scale data sets. Its advantages are good outlier detection and high computational efficiency. Its disadvantages are that it requires adjustment of hyperparameters and may not be sensitive enough to certain types of outliers.
[0030] LSTM: It uses a neural network model (Long Short-Term Memory) to learn the normal pattern of time series. Data points with large reconstruction errors are considered abnormal points. It is suitable for complex time series data, especially data with long dependencies. The advantage is that it can capture complex nonlinear relationships. The disadvantage is that it requires a large amount of training data and consumes a lot of computing resources.
[0031] In the operation and maintenance scenario, anomaly detection is an important function. Anomaly detection aims to automatically discover abnormal fluctuations in the time series data of key indicators through algorithms, and provide decision-making basis for subsequent alarms, automatic stop losses, root cause analysis, etc. In the operation and maintenance scenario, anomaly detection can use minute-level historical data, the length of which may reach 7200 points (5 days, 1440 minutes per day), or even longer. Generally, the minute signal on a daily basis will show a strong periodicity, so a decomposition detection algorithm is usually used to decompose signals such as cycle, trend, and noise. Then, it is judged whether it is an anomaly based on the trend and noise. Decomposition detection algorithms are generally iterative algorithms, generally O(n 2 ) and above, so the calculation time is very long. If the decomposition detection algorithm is not used, it is difficult to detect the anomaly superimposed on the periodic signal.
[0032] Based on this, in this specification, a data processing method is provided. This specification also relates to a data processing device, an electronic device, a computer-readable storage medium and a computer program product, which are described in detail one by one in the following embodiments.
[0033] See also Figure 1 , Figure 1 A flow chart of a data processing method provided according to an embodiment of the present specification is shown, which specifically includes the following steps.
[0034] Step 102: Obtain time series item data to be processed.
[0035] Among them, the time series project data to be processed can be understood as the time series data of a certain project. Time series data is also called time series data, which refers to a data column recorded in chronological order. Time series data is usually used to describe the changes of a phenomenon or thing over time. In the method provided in the embodiment of this specification, obtaining the time series project data to be processed refers to obtaining the time series data of at least one project.
[0036] In actual applications, different project data collection probes can be pre-configured for each project and deployed to the project application of the project. The project data collection probe is a software tool used to obtain various information generated by the target project during the application process, and is used to capture and analyze project-related data. It can run in the application of the target project, collect pre-configured information, and send the collected information to the specified location for subsequent processing.
[0037] Technical personnel can deploy project data collection probes in target projects according to actual project requirements. The project data collection probes can be used to capture the time series project data of the target project, thus improving the flexibility of collecting time series project data.
[0038] The time series project data to be processed can be understood as a time series signal for a target project. For example, the time series project data to be processed can be a time series signal such as CPU utilization, memory utilization, etc. The time series project data to be processed is discrete in time and can be continuous in value.
[0039] Step 104: Identify the data type of the time series project data to be processed, and determine the target data detection algorithm corresponding to the time series project data to be processed according to the data type.
[0040] After obtaining the time series project data to be processed, the data type of the time series project data to be processed can be further identified. The data type specifically refers to whether the time series project data to be processed is a periodic or stationary signal. That is, the periodicity information and stationary information of the time series project data to be processed are obtained. That is, in the method provided in the embodiment of this specification, identifying the data type of the time series project data to be processed specifically refers to identifying the periodicity information and stationary information of the time series project data to be processed.
[0041] The periodicity information of the time series project data to be processed refers to the characteristic of a signal repeating in time. If a signal f(t) satisfies f(t+T)=f(t), then the signal is a periodic signal, and T is a period of the signal. The stationarity information of the time series project data to be processed refers to the characteristic that the statistical properties of a signal remain unchanged over time.
[0042] After determining the data type of the time series project data to be processed, further determine the target data detection algorithm for data detection on the time series project data to be processed according to the specific data type. Among them, the target data detection algorithm can be understood as the data detection algorithm corresponding to the time series project data to be processed. The target data detection algorithm matches the data type of the time series project data to be processed. In practical applications, different data detection algorithms are set for different data types in the method provided in the embodiments of this specification.
[0043] In a specific implementation manner provided in this specification, identifying the data type of the time series item data to be processed includes: Identify whether the time series item data to be processed is periodic data; If yes, it is determined that the time series item data to be processed is periodic data; If not, identifying whether the time series item data to be processed is stationary data; If yes, it is determined that the time series item data to be processed is non-periodic stationary data; If not, it is determined that the time series item data to be processed is non-periodic and non-stationary data.
[0044] In the above embodiment, identifying the data type of the time series project data to be processed is defined as specifically determining the periodicity information and stationarity information of the time series project data to be processed. In this embodiment, a progressive method is used to determine the data type of the time series project data to be processed.
[0045] Specifically, in this embodiment, it is first determined whether the time series item data to be processed is periodic data. If so, no subsequent determination is performed, and the data type of the time series item data to be processed is directly determined to be periodic data.
[0046] If the time series project data to be processed is not periodic data, it is further determined whether it is stationary data. If so, it is determined that the time series project data to be processed is non-periodic stationary data, and if not, it is determined that the time series project data to be processed is non-periodic non-stationary data.
[0047] In a specific implementation manner provided in this specification, identifying whether the time series item data to be processed is periodic data includes: Performing frequency domain conversion on the time series project data to be processed to obtain frequency domain data to be processed, and judging whether the time series project data to be processed is periodic data according to the frequency domain data to be processed; or, An autocorrelation function graph corresponding to the time series item data to be processed is calculated, and whether the time series item data to be processed is periodic data is determined according to the autocorrelation function graph.
[0048] In the process of identifying whether the time series project data to be processed is periodic data, the time series project data to be processed may be subjected to periodic analysis. Specifically, the periodic analysis of the time series project data to be processed may be achieved by Fourier transform or autocorrelation function analysis.
[0049] Fourier transform is a method of converting time domain data into frequency domain data. For a periodic time series, it is very close to a sine wave and therefore contains a significant sine wave. The period of this sine wave can be found through Fourier transform, that is, expanding the time series data into a linear combination of trigonometric functions to obtain the coefficient of each expanded term (i.e., Fourier coefficient). The larger the Fourier coefficient, the more likely it is that the period of the corresponding sine wave is the period of the data.
[0050] In the method provided in the embodiment of this specification, the time series project data to be processed is converted from time domain data to frequency domain data through Fourier transform to obtain the frequency domain data to be processed. The frequency domain data to be processed can be understood as the frequency domain data generated after the time series project data to be processed is Fourier transformed. Whether the time series project data to be processed is periodic data is determined based on the frequency domain data to be processed.
[0051] Furthermore, after obtaining the frequency domain data to be processed, if there is an obvious frequency peak in the frequency domain data to be processed, it can be determined that the time series project data to be processed is periodic data. If there is no obvious frequency peak in the frequency domain data to be processed, the time series project data to be processed is non-periodic data.
[0052] In addition to using Fourier transform, you can also use autocorrelation graphs to analyze data. Specifically, calculate the autocorrelation function of the time series project data to be processed and draw the autocorrelation function graph. If the autocorrelation coefficient at the delay k is significantly not zero, and as k increases, the autocorrelation coefficient shows a periodic change (such as a sine wave shape), then it can be shown that the time series project data to be processed is periodic data.
[0053] Autocorrelation Function (ACF) is one of the important tools for analyzing time series data. The autocorrelation function is used to measure the correlation between a time series and itself at different delays, helping to reveal the periodicity, trend, and randomness of the data. The autocorrelation function can be used to draw its corresponding autocorrelation function graph, in which each point represents the autocorrelation value at the corresponding delay. In the autocorrelation function graph, significant peaks are usually significantly higher than the values of the surrounding points, and the delays k corresponding to these peaks may represent the periodic components in the time series.
[0054] In practical applications, in order to more objectively identify significant peaks, a threshold can be set. Usually, the threshold can be set to twice the standard deviation of the autocorrelation function or a fixed value. Only when the peak exceeds the threshold is it called a significant peak. After the significant peak is determined, its periodic changes can be verified by calculating the periodogram, performing spectrum analysis, etc.
[0055] Through the above-mentioned Fourier transform or autocorrelation function processing method, it is possible to identify whether the time series project data to be processed is periodic data. When it is identified that the time series project data to be processed is periodic data, it is determined that the time series project data to be processed is periodic data. When it is identified that the time series project data to be processed is not periodic data, it is necessary to further identify whether the time series project data to be processed is stationary data.
[0056] In another specific implementation manner provided in this specification, identifying whether the time series project data to be processed is stationary data includes: Collecting statistical characteristic information of the time series project data to be processed; Determining whether the statistical feature information remains constant; If yes, it is determined that the time series item data to be processed is stationary data; If not, it is determined that the time series item data to be processed is non-stationary data.
[0057] Identifying whether the time series project data to be processed is stationary data can be understood as performing a stationary analysis on the time series project data to be processed.
[0058] Stationarity analysis of time series data is an important concept in time series analysis. Stationarity means that the statistical characteristics of time series data do not change over time. The mean of a stationary time series remains unchanged over time, and the variance of the series is also fixed and does not change over time.
[0059] Based on this, in the process of identifying whether the time series project data to be processed is stationary data provided in the embodiment of this specification, statistical feature information of the time series project data to be processed is counted. Furthermore, counting the statistical feature information of the time series project data to be processed includes: counting at least one of the mean and variance of the time series project data to be processed.
[0060] In the method provided in the embodiment of this specification, at least one of the mean and variance of the time series project data to be processed is counted, and it is determined whether the mean and variance remain constant, that is, it is determined that the mean and variance do not change with time. If the statistical feature information remains constant, remains unchanged in time, and does not change with time, it means that the time series project data to be processed is stationary data, otherwise, it is determined that the time series project data to be processed is non-stationary data.
[0061] After determining the periodicity information and stationary information of the time series project data to be processed, the data type of the time series project data to be processed can be further determined based on the information of these two dimensions, that is, the data type of the time series project data to be processed includes periodic data, non-periodic stationary data, and non-periodic non-stationary data.
[0062] Periodic data indicates that the time series project data to be processed is data that satisfies periodicity; non-periodic stationary data refers to the time series project data to be processed that does not satisfy periodicity but satisfies stationarity; non-periodic non-stationary data refers to the time series project data to be processed that satisfies neither periodicity nor stationarity.
[0063] After determining the data type of the time series project data to be processed, the detection algorithm used to detect the time series project data to be processed can be further determined according to the data type, and the detection algorithm corresponding to the data type is used to perform data detection, which is more in line with the data characteristics of the time series project data to be processed. Specifically, in the specific implementation provided in this specification, the data type includes periodic data, non-periodic stationary data, or non-periodic non-stationary data; Determining a target data detection algorithm corresponding to the time series item data to be processed according to the data type includes: In the case where the data type is periodic data, determining that the target data detection algorithm corresponding to the time series item data to be processed is a decomposition type detection algorithm; In the case where the data type is non-periodic stationary data, determining that the target data detection algorithm corresponding to the time series item data to be processed is a statistical detection algorithm; In the case where the data type is non-periodic and non-stationary data, it is determined that the target data detection algorithm corresponding to the time series project data to be processed is a machine learning detection algorithm.
[0064] When the data type is periodic data, it means that the time series project data to be processed meets the periodic characteristics, and the decomposition type detection algorithm corresponding to the periodic data can be used as the detection algorithm for the time series project data to be processed. The decomposition type detection algorithm is used to detect whether there is abnormal data in the time series project data to be processed.
[0065] When the data type is non-periodic stationary data, it means that the time series project data to be processed does not meet the periodic characteristics, but meets the stationary characteristics. A statistical detection algorithm that is more suitable for this data type can be used. Use the data characteristics of the past period of time to detect whether there is abnormal data in the time series project data to be processed.
[0066] When the data type is non-periodic and non-stationary data, it means that the time series project data to be processed does not meet the periodic characteristics or the stationary characteristics. In this case, a machine learning detection algorithm can be collected to detect whether there is abnormal data in the time series project data to be processed through the characteristic information of the standard time series data.
[0067] Step 106: Detect the time series project data to be processed based on the target data detection algorithm to obtain data detection results corresponding to the time series project data to be processed.
[0068] After determining the target data detection algorithm corresponding to the time series project data to be processed, the time series project data to be processed can be detected according to the target data detection algorithm, thereby obtaining the data detection result corresponding to the time series project data to be processed.
[0069] In one or more specific implementations provided in this specification, the data detection result refers to whether there is abnormal project data in the time series project data to be processed. Specifically, the data detection result includes two situations: "abnormal project data exists" and "abnormal project data does not exist".
[0070] In practical applications, different data types correspond to different target data detection algorithms, and different target data detection algorithms have different ways of processing time series project data. The implementation methods provided in this specification explain the specific implementation methods of different data types and target data detection algorithms respectively.
[0071] In a specific implementation provided in this specification, the data type is periodic data, and the target data detection algorithm is a decomposition type detection algorithm; Detecting the time series item data to be processed based on the target data detection algorithm includes: Performing smoothing processing on the time series project data to be processed to obtain a trend component, and removing the trend component from the time series project data to be processed to generate reference time series project data; Performing periodic decomposition on the reference time series project data to obtain seasonal components, and removing the seasonal components from the reference time series project data to generate target time series project data; The target time series project data is detected based on a preset abnormal value threshold, and it is identified whether there is abnormal project data in the target time series project data.
[0072] In this embodiment, the data type is periodic data, and the target data detection algorithm is a decomposition detection algorithm. The classification inspection algorithm can be an STL algorithm, an X-12-ARIMA algorithm, etc. Furthermore, the decomposition detection algorithm is an STL algorithm.
[0073] The STL (Seasonal and Trend decomposition using Loess) algorithm is a time series analysis method that decomposes time series data into three components: seasonality, trend, and residual through the Loess (local weighted regression) technique. The basic principle of the STL algorithm is to represent time series data as a linear combination of trend, seasonality, and residual. Specifically, the time series data Y_t can be expressed as: Y_t = T_t + S_t + R_t. Among them, T_t represents the trend component, which reflects the long-term trend of the data; S_t represents the seasonal component, which reflects the cyclical changes of the data; R_t represents the residual component, which is the data after removing the trend and seasonality, reflecting the random fluctuations of the data.
[0074] In this embodiment, the time series project data to be processed is first smoothed to obtain the trend component. Smoothing is also called filtering, which is a low-frequency enhanced spatial domain filtering technology. The common use of smoothing is to reduce noise or distortion on the image. Smoothing is to fit the original data in mathematical and statistical analysis and generate smoother data. The purpose is to suppress outliers or noise in the original data and make the data more readable and interpretable. By smoothing the time series project data to be processed, its corresponding trend component can be obtained.
[0075] The trend component reflects the long-term trend of the data, that is, the overall direction or development of the time series data over time, which can be manifested as a continuous rise, fall, or stable development of the data. The trend component helps to understand the long-term behavior of time series data and is one of the key elements in time series analysis.
[0076] After obtaining the trend component, remove the trend component from the time series project data to be processed to generate reference time series project data. Remove the trend component from the time series project data to be processed. Since the trend component is a long-term change trend in the time series, it may mask other important information in the data, such as seasonal changes, residual fluctuations, etc. Removing the trend component from the time series project data to be processed can make other components in the time series project data to be processed clearer, prevent the trend component from interfering with other data analyses, and improve the accuracy of seasonal analysis of subsequent data. The reference time series project data can be understood as the time series project data to be processed with the trend component removed.
[0077] In the reference time series project data, the seasonal changes in the time series data can be seen more clearly. In time series analysis, the seasonal component is an important component alongside the trend component, cyclic component, irregular component, etc. The seasonal component refers to the fluctuating part of the time series data that is related to the season or a specific time period and recurs within a certain period. By periodically decomposing the reference time series project data, the seasonal components contained therein can be obtained. By removing the seasonal components from the reference time series project data, the target time series project data corresponding to the time series project data to be processed can be obtained. What is retained in the target time series project data is the residual component in the time series.
[0078] After determining the residual component, the abnormal project data in the residual component can be identified by presetting the abnormal value threshold. The preset abnormal value threshold can be set according to the actual situation. In a specific implementation provided in this specification, 3 times the standard deviation is used as an example for explanation. Count 3 times the standard deviation in the time series project data to be processed, and use the 3 times the standard deviation as the preset abnormal value threshold. Compare each project data in the target time series project data with the preset abnormal value threshold, and determine the project data greater than the preset abnormal value threshold as abnormal project data.
[0079] In another specific implementation provided in this specification, the data type is non-periodic stationary data, and the target data detection algorithm is a statistical detection algorithm; Detecting the time series item data to be processed based on the target data detection algorithm includes: Calculate the mean and standard deviation of the time series project data to be processed; Determine an abnormality detection interval according to the mean and the standard deviation; The time series item data to be processed is detected based on the abnormality detection interval, and it is identified whether there is abnormal item data in the time series item data to be processed.
[0080] In this embodiment, the data period type is non-periodic stationary data and the target data detection algorithm is a statistical detection algorithm. Furthermore, the target data detection algorithm is a nsigma algorithm.
[0081] Nsigma is a term used to describe the distribution or dispersion of the mean of data. Taking n=3 as an example, the 3sigma principle is an important concept in statistics. Sigma stands for standard deviation and is an important indicator to measure the dispersion of a data set. In a normal distribution, data points are distributed around the mean, and the standard deviation describes the degree of deviation between the data points and the mean. The 3sigma principle is based on the fact that in a normal distribution, about 68.27% of the data points fall within one standard deviation of the mean, about 95.45% of the data points fall within two standard deviations of the mean, and almost all (99.73% or 99.74%) of the data points fall within three standard deviations of the mean. Therefore, data points exceeding 3sigma can be considered outliers.
[0082] In the method provided in the embodiment of the present specification, the time series project data to be processed is detected based on the 3sigma principle. Specifically, the corresponding mean and standard deviation of the time series project data to be processed are first counted. And the abnormality detection interval is determined based on the 3sigma principle according to the mean and standard deviation. The time series project data to be processed is detected through the abnormality detection interval, and it is identified whether there is project data outside the abnormality detection interval in the time series project data to be processed. If there is, the project data outside the abnormality detection interval is determined as abnormal project data.
[0083] In another specific implementation manner provided in this specification, the data type is non-periodic non-stationary data, and the target data detection algorithm is a machine learning detection algorithm; Detecting the time series item data to be processed based on the target data detection algorithm includes: Inputting the time series item data to be processed into a data detection model, and obtaining error information between the time series item data to be processed and the standard time series item data output by the data detection model; Based on the error information and a preset error threshold, it is identified whether there is abnormal project data in the time series project data to be processed.
[0084] In this embodiment, the data cycle type is non-periodic non-stationary data, and the target data detection algorithm is a machine learning detection algorithm. Furthermore, the target data detection algorithm is a GBDT anomaly detection process.
[0085] GBDT (Gradient Boosting Decision Tree) is an ensemble learning algorithm that belongs to the Boosting type. It obtains the final prediction result by superimposing the prediction results of multiple decision trees. GBDT achieves accurate prediction of data by iteratively training decision trees to minimize the loss function. Each round of iteration aims to fit the residual of the previous round, that is, the difference between the true value and the predicted value, so as to gradually improve the prediction accuracy of the model. GBDT belongs to the Boosting family, which trains the base classifiers in a serial manner and improves the performance by weighted combination of these models. In addition, the base learner of GBDT is usually a regression tree, while the random forest uses multiple decision trees to vote or average to obtain the final result.
[0086] In the method provided in the embodiments of this specification, historical time series project data can be obtained, and a data detection model can be trained using the historical time series project data. The data detection model is trained to calculate the error information between the input time series project data and the standard time series project data. The error information here can be square error, etc.
[0087] In the model application stage, the time series project data to be processed is input into the pre-trained data detection model. The data detection model will convert the input time series project data to be processed into feature information to be processed, perform data prediction on the feature information to be processed, and calculate the error information between the predicted value and the actual value of the standard time series project data.
[0088] When the error information exceeds the preset error threshold, it can be determined that there is abnormal project data in the time series project data to be processed. If the error information does not exceed the preset error threshold, it can be determined that there is no abnormal project data in the time series project data to be processed.
[0089] Through the target data detection algorithms corresponding to the above different data types, the time series project data to be processed can be detected to detect whether there are abnormal project data in the time series project data to be processed. Determining the target data detection algorithm according to the data type of the time series project data to be processed can match a more suitable target data detection algorithm for the time series project data to be processed, avoiding the problems of high computational complexity and long computational time caused by directly using the decomposition detection algorithm. Improves data processing efficiency.
[0090] In a specific implementation provided in this specification, it also includes: In the case where the data detection result includes abnormal item data, abnormal warning information is generated based on the data detection result.
[0091] In the method provided in the embodiments of the present specification, when the data detection results include abnormal project data, corresponding abnormal alarm information will also be generated based on the data detection results, and the alarm information will be sent to the operation and maintenance personnel, so that the operation and maintenance personnel can promptly understand the abnormal information in the time series project data to be processed, so that they can perform corresponding processing in time and reduce the impact of the abnormal project data.
[0092] The following combination Figure 2 , the data processing method is further described. Figure 2 FIG. 1 shows a process flow chart of a data processing method provided by an embodiment of the present specification. Figure 2 As shown, after obtaining the time series project data to be processed, a periodicity analysis is first performed on the time series project data to be processed to determine whether it is periodic data.
[0093] If the time series project data to be processed is periodic data, the STL algorithm can be used to perform data detection on the time series project data to be processed to obtain data detection results corresponding to the time series project data to be processed.
[0094] If the time series project data to be processed is not periodic data, it is necessary to further perform a stationary analysis on the time series project data to be processed, that is, to determine whether the time series project data to be processed is stationary data.
[0095] If the time series project data to be processed is stationary data, the 3sigma algorithm can be used to perform data detection on the time series project data to be processed to obtain a data detection result corresponding to the time series project data to be processed.
[0096] If the time series project data to be processed is not stationary data, it is necessary to use the GBDT algorithm to perform data detection on the time series project data to be processed to obtain the data detection results corresponding to the time series project data to be processed.
[0097] Corresponding to the above method embodiment, this specification also provides a data processing device embodiment, Figure 3 FIG. 1 is a schematic diagram showing the structure of a data processing device provided by an embodiment of the present specification. Figure 3 As shown, the device comprises: An acquisition module 302 is configured to acquire time series project data to be processed; The identification module 304 is configured to identify the data type of the time series project data to be processed, and determine the target data detection algorithm corresponding to the time series project data to be processed according to the data type; The detection module 306 is configured to detect the time series project data to be processed based on the target data detection algorithm, and obtain a data detection result corresponding to the time series project data to be processed.
[0098] Optionally, the identification module 304 is further configured to: Identify whether the time series item data to be processed is periodic data; If yes, it is determined that the time series item data to be processed is periodic data; If not, identifying whether the time series item data to be processed is stationary data; If yes, it is determined that the time series item data to be processed is non-periodic stationary data; If not, it is determined that the time series item data to be processed is non-periodic and non-stationary data.
[0099] Optionally, the identification module 304 is further configured to: Performing frequency domain conversion on the time series project data to be processed to obtain frequency domain data to be processed, and judging whether the time series project data to be processed is periodic data according to the frequency domain data to be processed; or, An autocorrelation function graph corresponding to the time series item data to be processed is calculated, and whether the time series item data to be processed is periodic data is determined according to the autocorrelation function graph.
[0100] Optionally, the identification module 304 is further configured to: Collecting statistical characteristic information of the time series project data to be processed; Determining whether the statistical feature information remains constant; If yes, it is determined that the time series item data to be processed is stationary data; If not, it is determined that the time series item data to be processed is non-stationary data.
[0101] Optionally, the identification module 304 is further configured to: At least one of the mean and variance of the time series item data to be processed is counted.
[0102] Optionally, the data type includes periodic data, non-periodic stationary data, or non-periodic non-stationary data; The identification module 304 is further configured to: In the case where the data type is periodic data, determining that the target data detection algorithm corresponding to the time series item data to be processed is a decomposition type detection algorithm; In the case where the data type is non-periodic stationary data, determining that the target data detection algorithm corresponding to the time series item data to be processed is a statistical detection algorithm; In the case where the data type is non-periodic and non-stationary data, it is determined that the target data detection algorithm corresponding to the time series project data to be processed is a machine learning detection algorithm.
[0103] Optionally, the data type is periodic data, and the target data detection algorithm is a decomposition detection algorithm; The detection module 306 is further configured to: Performing smoothing processing on the time series project data to be processed to obtain a trend component, and removing the trend component from the time series project data to be processed to generate reference time series project data; Performing periodic decomposition on the reference time series project data to obtain seasonal components, and removing the seasonal components from the reference time series project data to generate target time series project data; The target time series project data is detected based on a preset abnormal value threshold, and it is identified whether there is abnormal project data in the target time series project data.
[0104] Optionally, the data type is non-periodic stationary data, and the target data detection algorithm is a statistical detection algorithm; The detection module 306 is further configured to: Calculate the mean and standard deviation of the time series project data to be processed; Determine an abnormality detection interval according to the mean and the standard deviation; The time series item data to be processed is detected based on the abnormality detection interval, and it is identified whether there is abnormal item data in the time series item data to be processed.
[0105] Optionally, the data type is non-periodic and non-stationary data, and the target data detection algorithm is a machine learning detection algorithm; The detection module 306 is further configured to: Inputting the time series item data to be processed into a data detection model, and obtaining error information between the time series item data to be processed and the standard time series item data output by the data detection model; Based on the error information and a preset error threshold, it is identified whether there is abnormal project data in the time series project data to be processed.
[0106] Optionally, the device further includes an alarm module configured to: In the case where the data detection result includes abnormal item data, abnormal warning information is generated based on the data detection result.
[0107] Through the target data detection algorithms corresponding to the above different data types, the time series project data to be processed can be detected to detect whether there are abnormal project data in the time series project data to be processed. Determining the target data detection algorithm according to the data type of the time series project data to be processed can match a more suitable target data detection algorithm for the time series project data to be processed, avoiding the problems of high computational complexity and long computational time caused by directly using the decomposition detection algorithm. Improves data processing efficiency.
[0108] The above is a schematic scheme of a data processing device of this embodiment. It should be noted that the technical scheme of the data processing device and the technical scheme of the above data processing method belong to the same concept, and the details of the technical scheme of the data processing device that are not described in detail can be referred to the description of the technical scheme of the above data processing method.
[0109] Figure 4 The structure block diagram of an electronic device 400 provided according to an embodiment of the present application is shown. The components of the electronic device 400 include but are not limited to a memory 410 and a processor 420. The processor 420 is connected to the memory 410 via a bus 430, and the database 450 is used to store data.
[0110] The electronic device 400 also includes an access device 440, which enables the electronic device 400 to communicate via one or more networks 460. Examples of these networks include a public switched telephone network (PSTN), a local area network (LAN), a wide area network (WAN), a personal area network (PAN), or a combination of communication networks such as the Internet. The access device 440 may include one or more of any type of network interface (e.g., a network interface card (NIC)) of wired or wireless, such as an IEEE 802.11 wireless local area network (WLAN) wireless interface, a world-wide interoperability for microwave access (Wi-MAX) interface, an Ethernet interface, a universal serial bus (USB) interface, a cellular network interface, a Bluetooth interface, a near field communication (NFC) interface, and the like.
[0111] In one embodiment of the present application, the above components of the electronic device 400 and Figure 4 Other components not shown in the figure may also be connected to each other, for example, via a bus. It should be understood that Figure 4 The electronic device structure block diagram shown is only for the purpose of illustration, and is not intended to limit the scope of the present application. Those skilled in the art may add or replace other components as needed.
[0112] The electronic device 400 may be any type of stationary or mobile electronic device, including a mobile computer or mobile electronic device (e.g., a tablet computer, a personal digital assistant, a laptop computer, a notebook computer, a netbook, etc.), a mobile phone (e.g., a smart phone), a wearable electronic device (e.g., a smart watch, smart glasses, etc.), or other types of mobile devices, or a stationary electronic device such as a desktop computer or a personal computer (PC). The electronic device 400 may also be a mobile or stationary server.
[0113] The processor 420 is used to execute the following computer program / instructions, which implement the steps of the above-mentioned data processing method when executed by the processor.
[0114] The above is a schematic scheme of an electronic device of this embodiment. It should be noted that the technical scheme of the electronic device and the technical scheme of the above data processing method belong to the same concept, and the details not described in detail in the technical scheme of the electronic device can be referred to the description of the technical scheme of the above data processing method.
[0115] An embodiment of the present specification further provides a computer-readable storage medium storing a computer program / instruction, which implements the steps of the above-mentioned data processing method when executed by a processor.
[0116] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the computer-readable storage medium embodiment, since it is basically similar to the data processing method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the data processing method embodiment.
[0117] An embodiment of the present specification also provides a computer program product, including a computer program / instruction, which implements the steps of the above data processing method when executed by a processor.
[0118] The above is a schematic scheme of a computer program product of this embodiment. It should be noted that the technical scheme of the computer program product and the technical scheme of the above data processing method belong to the same concept, and the details not described in detail in the technical scheme of the computer program product can be referred to the description of the technical scheme of the above data processing method.
[0119] The above is a description of a specific embodiment of the specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.
[0120] The computer instructions include computer program codes, which may be in source code form, object code form, executable files or some intermediate forms, etc. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium, etc. It should be noted that the content contained in the computer-readable medium may be appropriately increased or decreased according to the requirements of patent practice. For example, in some regions, according to patent practice, computer-readable media do not include electric carrier signals and telecommunication signals.
[0121] It should be noted that the above is a description of a specific embodiment of the present specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the accompanying drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the embodiments of the present specification.
[0122] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.
[0123] The preferred embodiments of this specification disclosed above are only used to help explain this specification. The optional embodiments do not describe all the details in detail, nor do they limit the invention to only the specific implementation methods described. Obviously, many modifications and changes can be made according to the content of the embodiments of this specification. This specification selects and specifically describes these embodiments in order to better explain the principles and practical applications of the embodiments of this specification, so that technicians in the relevant technical field can well understand and use this specification. This specification is only limited by the claims and their full scope and equivalents.
Claims
1. A data processing method, comprising: Get the time series project data to be processed; Identify the data type of the time series project data to be processed, and determine the target data detection algorithm corresponding to the time series project data to be processed according to the data type; The time series project data to be processed is detected based on the target data detection algorithm to obtain a data detection result corresponding to the time series project data to be processed.
2. The method according to claim 1, identifying the data type of the time series project data to be processed, comprising: Identify whether the time series item data to be processed is periodic data; If yes, it is determined that the time series item data to be processed is periodic data; If not, identifying whether the time series item data to be processed is stationary data; If yes, it is determined that the time series item data to be processed is non-periodic stationary data; If not, it is determined that the time series item data to be processed is non-periodic and non-stationary data.
3. The method according to claim 2, identifying whether the time series item data to be processed is periodic data, comprising: Performing frequency domain conversion on the time series project data to be processed to obtain frequency domain data to be processed, and judging whether the time series project data to be processed is periodic data according to the frequency domain data to be processed; or, An autocorrelation function graph corresponding to the time series item data to be processed is calculated, and whether the time series item data to be processed is periodic data is determined according to the autocorrelation function graph.
4. The method according to claim 2, wherein the step of identifying whether the time series project data to be processed is stationary data comprises: Collecting statistical characteristic information of the time series project data to be processed; Determining whether the statistical feature information remains constant; If yes, it is determined that the time series item data to be processed is stationary data; If not, it is determined that the time series item data to be processed is non-stationary data.
5. The method according to claim 4, wherein the statistical characteristic information of the time series project data to be processed is collected, comprising: At least one of the mean and variance of the time series item data to be processed is counted.
6. The method of claim 1, wherein the data type comprises periodic data, non-periodic stationary data, or non-periodic non-stationary data; Determining a target data detection algorithm corresponding to the time series item data to be processed according to the data type includes: In the case where the data type is periodic data, determining that the target data detection algorithm corresponding to the time series item data to be processed is a decomposition type detection algorithm; In the case where the data type is non-periodic stationary data, determining that the target data detection algorithm corresponding to the time series item data to be processed is a statistical detection algorithm; In the case where the data type is non-periodic and non-stationary data, it is determined that the target data detection algorithm corresponding to the time series project data to be processed is a machine learning detection algorithm.
7. The method according to claim 1, wherein the data type is periodic data, and the target data detection algorithm is a decomposition type detection algorithm; Detecting the time series item data to be processed based on the target data detection algorithm includes: Performing smoothing processing on the time series project data to be processed to obtain a trend component, and removing the trend component from the time series project data to be processed to generate reference time series project data; Performing periodic decomposition on the reference time series project data to obtain seasonal components, and removing the seasonal components from the reference time series project data to generate target time series project data; The target time series project data is detected based on a preset abnormal value threshold, and it is identified whether there is abnormal project data in the target time series project data.
8. The method according to claim 1, wherein the data type is non-periodic stationary data, and the target data detection algorithm is a statistical detection algorithm; Detecting the time series item data to be processed based on the target data detection algorithm includes: Calculate the mean and standard deviation of the time series project data to be processed; Determine an abnormality detection interval according to the mean and the standard deviation; The time series item data to be processed is detected based on the abnormality detection interval, and it is identified whether there is abnormal item data in the time series item data to be processed.
9. The method according to claim 1, wherein the data type is non-periodic and non-stationary data, and the target data detection algorithm is a machine learning detection algorithm; Detecting the time series item data to be processed based on the target data detection algorithm includes: Inputting the time series item data to be processed into a data detection model, and obtaining error information between the time series item data to be processed and the standard time series item data output by the data detection model; Based on the error information and a preset error threshold, it is identified whether there is abnormal project data in the time series project data to be processed.
10. The method according to any one of claims 1 to 9, further comprising: In the case where the data detection result includes abnormal item data, abnormal warning information is generated based on the data detection result.
11. A data processing device, comprising: An acquisition module, configured to acquire time series project data to be processed; an identification module, configured to identify a data type of the time series project data to be processed, and determine a target data detection algorithm corresponding to the time series project data to be processed according to the data type; The detection module is configured to detect the time series project data to be processed based on the target data detection algorithm, and obtain a data detection result corresponding to the time series project data to be processed.
12. An electronic device comprising: Memory and processor; The memory is used to store computer programs / instructions, and the processor is used to execute the computer programs / instructions. When the computer programs / instructions are executed by the processor, the steps of the method described in any one of claims 1 to 10 are implemented.
13. A computer-readable storage medium storing a computer program / instruction, wherein the computer program / instruction, when executed by a processor, implements the steps of the method according to any one of claims 1 to 10.
14. A computer program product, comprising a computer program / instruction, which, when executed by a processor, implements the steps of the method according to any one of claims 1 to 10.