Data mining method and system based on isolated forest and time sequence analysis

Abstract

The invention discloses a data mining method and system based on an isolated forest and time sequence analysis, and the method comprises the steps: carrying out the standardization processing of obtained original renewable energy data, and obtaining standard renewable energy data; segmenting the standard renewable energy source data into a plurality of data samples according to a preset sliding window; performing anomaly detection on each data sample according to a pre-constructed isolated forest to obtain a plurality of abnormal data samples and a plurality of normal data samples; performing data restoration on the standard renewable energy source data according to the abnormal data sample to obtain restored renewable energy source data; and performing time sequence decomposition on the repaired renewable energy source data to obtain time sequence characteristic data corresponding to the original renewable energy source data, so as to improve the accuracy of data mining.

Description

Technical Field

[0001] The present invention relates to the technical field of power grid data detection, and in particular to a data mining method and system based on isolation forest and time series analysis. Background Art

[0002] As time series data, renewable energy data has multidimensional characteristics and significant randomness and volatility, which brings great challenges to anomaly detection of renewable energy data and also restricts the large-scale application of renewable energy.

[0003] Traditional anomaly detection methods often have problems such as insufficient accuracy, poor real-time performance, and high computational cost when processing high-dimensional, nonlinear, and highly random time series data. It is difficult to effectively capture the complex time series characteristics of data, and it is impossible to achieve efficient and accurate anomaly detection in large-scale application scenarios. In addition, when faced with data noise and high volatility, it is easy to miss or misdetect, which limits the data mining depth and operation optimization capabilities of renewable energy systems. Summary of the invention

[0004] In order to solve the above technical problems, the present invention discloses a data mining method and system based on isolation forest and time series analysis, which are used to improve the accuracy of renewable energy data decomposition.

[0005] In order to achieve the above object, the present invention discloses a data mining method based on isolation forest and time series analysis, comprising:

[0006] Standardize the acquired raw renewable energy data to obtain standard renewable energy data;

[0007] Dividing the standard renewable energy data into a plurality of data samples according to a preset sliding window;

[0008] Performing anomaly detection on each of the data samples according to the pre-constructed isolation forest to obtain a number of abnormal data samples and a number of normal data samples;

[0009] Performing data repair on the standard renewable energy data according to the abnormal data sample to obtain repaired renewable energy data;

[0010] The repaired renewable energy data is subjected to time series decomposition to obtain time series characteristic data corresponding to the original renewable energy data.

[0011] The present invention discloses a data mining method based on isolation forest and time series analysis. First, the original renewable energy data obtained is standardized to eliminate the influence between data of different dimensions, thereby improving the accuracy and efficiency of subsequent detection of abnormal data. Then, the standard renewable energy data is divided into multiple data samples using a sliding window to ensure that the data samples not only retain the time series information corresponding to the original data but also do not destroy the correlation between the features in the original data, so as to improve the accuracy of detection and the efficiency of subsequent abnormal detection. Furthermore, the abnormal data is detected using isolation forest to improve the accuracy and efficiency of detection. Finally, the standard renewable energy data is repaired for abnormal data according to the results of abnormal detection to ensure the accuracy of the data used for mining, and then the renewable energy data obtained after repair is decomposed into time series to improve the accuracy of the decomposition.

[0012] As a preferred example, the standardization process of the acquired raw renewable energy data includes:

[0013] Performing data decomposition on the original renewable energy data to obtain a plurality of dimensional feature data corresponding to the original renewable energy data;

[0014] The maximum value and the minimum value corresponding to each dimension of the characteristic data are obtained, so as to perform standardization processing on the characteristic data of each dimension through a preset standardization formula.

[0015] The present invention performs standardization processing on each dimension of feature data in the original renewable energy data to eliminate the influence between different feature data, thereby improving the efficiency and accuracy of subsequent anomaly detection.

[0016] As a preferred example, the standard renewable energy data is divided into a plurality of data samples according to a preset sliding window, including:

[0017] Obtaining the window width and sliding step size of the sliding window;

[0018] According to the window width and the sliding step length, the standard renewable energy data is divided into a plurality of data samples along a time axis direction corresponding to the original renewable energy data.

[0019] The present invention utilizes a sliding window to segment the data, so that each data sample after segmentation contains not only the time series information in the original data, but also the feature data of different dimensions in the original data. Therefore, when performing anomaly detection based on the processed data samples later, not only the detection accuracy is improved, but also the tedious process of one-to-one detection of each dimensional feature is avoided, thereby greatly improving the detection efficiency.

[0020] As a preferred example, performing anomaly detection on each of the data samples according to the pre-constructed isolation forest to obtain a number of abnormal data samples includes:

[0021] Obtaining the path length traversed by each of the data samples in each isolated tree; wherein the isolation forest includes a plurality of isolated trees;

[0022] According to the path length and the number of isolated trees, obtaining a total path length corresponding to each of the data samples;

[0023] An abnormal score corresponding to each of the data samples is determined according to the total path length and an abnormal value calculation method preset in the isolation forest.

[0024] The present invention uses isolation forest to detect outliers to ensure the efficiency and accuracy of anomaly detection.

[0025] As a preferred example, performing anomaly detection on each of the data samples according to the pre-constructed isolation forest to obtain a plurality of abnormal data samples and a plurality of normal data samples includes:

[0026] Determining the number of abnormalities of the abnormal data samples according to a preset prior abnormality ratio and the number of the data samples;

[0027] A plurality of data samples corresponding to the maximum anomaly score are screened according to the anomaly quantity to generate a plurality of the anomaly data samples.

[0028] The present invention utilizes the size of the anomaly score and selects a plurality of samples with the largest anomaly index from a plurality of data samples according to a preset prior anomaly ratio to determine them as abnormal data, so as to improve the accuracy of abnormal data detection.

[0029] As a preferred example, the construction process of the isolation forest is:

[0030] Randomly select a feature data from the feature data of a plurality of dimensions, and determine the data sample where the feature data is located;

[0031] Generating a split point between the maximum value and the minimum value corresponding to the characteristic data in the data sample to segment the data sample;

[0032] Acquire each node generated after segmentation in real time to construct an isolated tree according to a preset segmentation stop condition and a tree structure generated by the current segmentation; wherein the segmentation stop condition includes that the feature data on the current node have the same feature or the currently formed isolated tree reaches a preset depth threshold;

[0033] The process of constructing the isolated tree is repeated until an isolated forest consisting of a plurality of the isolated trees is obtained.

[0034] In order to ensure the accuracy of anomaly detection by the isolation forest, the present invention needs to set a certain number of isolated trees to ensure the diversity of classification. Furthermore, in this embodiment, the isolated trees are iteratively constructed by randomly selecting feature data in the renewable energy data, thereby ensuring that the constructed isolation forest has high-precision abnormal data detection capabilities.

[0035] As a preferred example, the performing data repair on the standard renewable energy data according to the abnormal data sample includes:

[0036] According to the time axis direction, acquiring adjacent data samples whose tails are connected to the beginning of the abnormal data sample;

[0037] Calculating the similarity index between the adjacent data sample and each normal data sample according to a preset similarity index calculation formula;

[0038] The abnormal data sample is replaced by the normal data sample corresponding to the time when the similarity index is the largest.

[0039] For the detected abnormal data sample, the present invention first finds the data sample whose tail is connected to its beginning, and then finds the data sample most similar to the connected data sample in the normal data samples, and ensures the accuracy of abnormal data repair by replacement.

[0040] As a preferred example, the time series decomposition of the repaired renewable energy data to obtain the time series characteristic data corresponding to the original renewable energy data includes:

[0041] Performing data decomposition on the repaired renewable energy data to obtain a plurality of dimensional feature data corresponding to the repaired renewable energy data;

[0042] Perform STL decomposition on the characteristic data of each dimension to obtain time series characteristic data corresponding to the characteristic data of each dimension; wherein the time series characteristic data includes a trend component, a period component and a remainder.

[0043] The present invention performs STL decomposition on each dimensional feature in the renewable energy data after anomaly detection and repair, so as to improve the accuracy of the time series decomposition through the STL decomposition and the anomaly detection and repair.

[0044] On the other hand, the present invention discloses a data mining system based on isolation forest and time series analysis, including a standardization module, a data segmentation module, an anomaly detection module, an anomaly repair module and a time series decomposition module;

[0045] The standardization module is used to perform standardization processing on the acquired original renewable energy data to obtain standard renewable energy data;

[0046] The data segmentation module is used to segment the standard renewable energy data into a plurality of data samples according to a preset sliding window;

[0047] The anomaly detection module is used to perform anomaly detection on each of the data samples according to the pre-constructed isolation forest to obtain a plurality of abnormal data samples and a plurality of normal data samples;

[0048] The abnormality repair module is used to perform data repair on the standard renewable energy data according to the abnormal data sample to obtain repaired renewable energy data;

[0049] The time series decomposition module is used to perform time series decomposition on the repaired renewable energy data to obtain time series characteristic data corresponding to the original renewable energy data.

[0050] The present invention discloses a data mining system based on isolation forest and time series analysis. First, the acquired original renewable energy data is standardized to eliminate the influence between data of different dimensions, thereby improving the accuracy and efficiency of subsequent detection of abnormal data. Then, the standard renewable energy data is divided into multiple data samples using a sliding window to ensure that the data samples not only retain the time series information corresponding to the original data but also do not destroy the correlation between the features in the original data, so as to improve the accuracy of detection and the efficiency of subsequent abnormal detection. Furthermore, the abnormal data is detected using isolation forest to improve the accuracy and efficiency of detection. Finally, the abnormal data of the standard renewable energy data is repaired according to the results of the abnormal detection to ensure the accuracy of the data used for mining, and then the renewable energy data obtained after the repair is decomposed into time series to improve the accuracy of the decomposition.

[0051] As a preferred example, the standardization module includes a decomposition unit and a processing unit;

[0052] The decomposition unit is used to perform data decomposition on the original renewable energy data to obtain a plurality of dimensional feature data corresponding to the original renewable energy data;

[0053] The processing unit is used to obtain the maximum value and the minimum value corresponding to each dimension of the characteristic data, so as to perform standardization processing on the characteristic data of each dimension through a preset standardization formula.

[0054] The present invention performs standardization processing on each dimension of feature data in the original renewable energy data to eliminate the influence between different feature data, thereby improving the efficiency and accuracy of subsequent anomaly detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Figure 1 : A flow chart of a data mining method based on isolation forest and time series analysis disclosed in an embodiment of the present invention;

[0056] Figure 2 : A structural diagram of a data mining system based on isolation forest and time series analysis disclosed in an embodiment of the present invention;

[0057] Figure 3 : A flow chart of a data mining method based on isolation forest and time series analysis disclosed in another embodiment of the present invention;

[0058] Figure 4 : A schematic diagram of a construction process of an isolated forest disclosed in another embodiment of the present invention. DETAILED DESCRIPTION

[0059] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0060] Embodiment 1

[0061] In this embodiment, in order to improve the accuracy of renewable energy data mining, a data mining method based on isolation forest and time series analysis is provided. Specifically, the specific implementation process of the mining method can be referred to in Figure 1 , mainly including steps 101 to 105, the steps are:

[0062] Step 101: Standardize the acquired raw renewable energy data to obtain standard renewable energy data.

[0063] In this embodiment, this step mainly includes: performing data decomposition on the original renewable energy data to obtain several dimensions of feature data corresponding to the original renewable energy data; obtaining the maximum and minimum values ​​corresponding to each dimension of the feature data, so as to standardize each dimension of the feature data through a preset standardization formula.

[0064] In this embodiment, this step performs standardization processing on each dimension of feature data in the original renewable energy data to eliminate the influence between different feature data, thereby improving the efficiency and accuracy of subsequent anomaly detection.

[0065] Step 102: Divide the standard renewable energy data into a plurality of data samples according to a preset sliding window.

[0066] In this embodiment, this step mainly includes: obtaining the window width and sliding step of the sliding window; and dividing the standard renewable energy data into a plurality of data samples along the time axis direction corresponding to the original renewable energy data according to the window width and the sliding step.

[0067] In this embodiment, this step uses a sliding window to segment the data, so that each data sample after segmentation contains not only the time series information in the original data, but also the feature data of different dimensions in the original data. Therefore, when anomaly detection is subsequently performed based on the processed data samples, not only the detection accuracy is improved, but also the tedious process of one-to-one detection of each dimensional feature is avoided, thereby greatly improving the detection efficiency.

[0068] Step 103: Perform anomaly detection on each of the data samples according to the pre-constructed isolation forest to obtain a plurality of abnormal data samples and a plurality of normal data samples.

[0069] In this embodiment, this step mainly includes: obtaining the path length traversed by each of the data samples in each isolated tree; wherein the isolated forest includes a plurality of isolated trees; obtaining the total path length corresponding to each of the data samples according to the path length and the number of the isolated trees; and determining the anomaly score corresponding to each of the data samples according to the total path length and the outlier calculation method preset in the isolated forest.

[0070] Furthermore, the number of abnormalities of the abnormal data samples is determined according to a preset prior abnormality ratio and the number of the data samples; and a number of data samples corresponding to the maximum abnormality score are screened according to the number of abnormalities to generate a number of the abnormal data samples.

[0071] The construction process of the isolation forest is as follows: randomly select a feature data from the feature data of several dimensions, and determine the data sample where the feature data is located; generate a split point between the maximum value and the minimum value corresponding to the feature data in the data sample to segment the data sample; obtain each node generated after the segmentation in real time to construct an isolated tree according to a preset segmentation stop condition and the tree structure generated by the current segmentation; wherein the segmentation stop condition includes that the feature data on the current node have the same feature or the currently formed isolated tree reaches a preset depth threshold; repeat the construction process of the isolated tree until an isolated forest consisting of several isolated trees is obtained.

[0072] In this embodiment, this step uses an isolation forest to detect outliers to ensure the efficiency and accuracy of anomaly detection. At the same time, in the anomaly detection process, the size of the anomaly score is used to screen out multiple samples with the largest anomaly index from a number of data samples according to a preset prior anomaly ratio to determine them as abnormal data, so as to improve the accuracy of abnormal data detection. Further, in order to ensure the accuracy of anomaly detection by the isolation forest, a certain number of isolated trees need to be set to ensure the diversity of classification. Further, in this embodiment, the feature data in the renewable energy data is randomly selected to iteratively construct an isolated tree, so as to ensure that the constructed isolation forest has high-precision abnormal data detection capabilities.

[0073] Step 104: Perform data repair on the standard renewable energy data according to the abnormal data sample to obtain repaired renewable energy data.

[0074] In this embodiment, the steps are mainly as follows: according to the time axis direction, obtaining the adjacent data sample whose tail is connected to the beginning of the abnormal data sample; calculating the similarity index between the adjacent data sample and each normal data sample according to a preset similarity index calculation formula; replacing the abnormal data sample with the normal data sample corresponding to the time when the similarity index is the largest.

[0075] In this embodiment, for the detected abnormal data sample, this step first finds the data sample whose tail is connected to its beginning, and then finds the data sample most similar to the connected data sample in the normal data samples, and ensures the accuracy of abnormal data repair by replacement.

[0076] Step 105: performing time series decomposition on the repaired renewable energy data to obtain time series characteristic data corresponding to the original renewable energy data.

[0077] In this embodiment, the step mainly includes: performing data decomposition on the repaired renewable energy data to obtain several dimensions of feature data corresponding to the repaired renewable energy data; performing STL decomposition on each dimension of the feature data to obtain time series feature data corresponding to each dimension of the feature data; wherein the time series feature data includes a trend component, a periodic component and a remainder.

[0078] In this embodiment, this step performs STL decomposition on each dimensional feature in the renewable energy data after anomaly detection and repair, so as to improve the accuracy of the time series decomposition through the STL decomposition and the anomaly detection and repair.

[0079] On the other hand, this embodiment also provides a data mining system based on isolation forest and time series analysis. For the specific structure of the data mining system, please refer to Figure 2 , including a standardization module 201, a data segmentation module 202, an anomaly detection module 203, an anomaly repair module 204 and a time series decomposition module 205.

[0080] The standardization module 201 is used to perform standardization processing on the acquired original renewable energy data to obtain standard renewable energy data.

[0081] The data segmentation module 202 is used to segment the standard renewable energy data into a plurality of data samples according to a preset sliding window.

[0082] The anomaly detection module 203 is used to perform anomaly detection on each of the data samples according to the pre-constructed isolation forest to obtain a plurality of abnormal data samples and a plurality of normal data samples.

[0083] The abnormality repair module 204 is used to perform data repair on the standard renewable energy data according to the abnormal data sample to obtain repaired renewable energy data.

[0084] The time series decomposition module 205 is used to perform time series decomposition on the repaired renewable energy data to obtain time series characteristic data corresponding to the original renewable energy data.

[0085] In this embodiment, the standardization module 201 includes a decomposition unit and a processing unit.

[0086] The decomposition unit is used to perform data decomposition on the original renewable energy data to obtain a plurality of dimensional feature data corresponding to the original renewable energy data.

[0087] The processing unit is used to obtain the maximum value and the minimum value corresponding to each dimension of the characteristic data, so as to perform standardization processing on the characteristic data of each dimension through a preset standardization formula.

[0088] The present embodiment discloses a data mining method and system based on isolation forest and time series analysis. First, the acquired original renewable energy data is standardized to eliminate the influence between data of different dimensions, thereby improving the accuracy and efficiency of subsequent detection of abnormal data. Then, the standard renewable energy data is divided into multiple data samples using a sliding window to ensure that the data samples not only retain the time series information corresponding to the original data but also do not destroy the correlation between the features in the original data, so as to improve the accuracy of detection and the efficiency of subsequent abnormal detection. Furthermore, the detection of abnormal data is performed using an isolation forest to improve the accuracy and efficiency of detection. Finally, the abnormal data of the standard renewable energy data is repaired according to the results of the abnormal detection to ensure the accuracy of the data used for mining, and then the renewable energy data obtained after the repair is decomposed into time series to improve the accuracy of the decomposition.

[0089] Embodiment 2

[0090] In the prior art, based on the multi-dimensional characteristics and significant randomness and volatility of renewable energy data, anomaly detection and time series decomposition of the renewable energy data have become research difficulties, which restricts the large-scale application of renewable energy. Traditional anomaly detection methods often cannot meet high-precision requirements or have too high time complexity. Therefore, it is particularly important to develop efficient anomaly detection technology suitable for renewable energy data. On the basis of accurate and reliable data, the intrinsic characteristics of renewable energy can be deeply explored through the data time series decomposition method, which further supports the operation and management decisions of renewable energy power stations, which is of great significance for promoting the large-scale application of renewable energy.

[0091] In order to solve the above technical difficulties, this embodiment discloses a data mining method based on isolation forest and time series analysis. Specifically, the specific implementation process of the data mining method is shown in FIG. Figure 3 , mainly including steps 301 to 305:

[0092] Step 301: Standardize the acquired original renewable energy data according to a preset data standardization formula to obtain standardized renewable energy data.

[0093] In this embodiment, this step mainly includes: performing data decomposition on the original renewable energy data to perform standardization processing on each dimension of feature data corresponding to the original renewable energy data.

[0094] Specifically, in some implementations of this embodiment, the standardization processing method includes Z-score standardization, Min-Max normalization, etc. Preferably, taking the Min-Max normalization method for data standardization as an example, for renewable energy data with N-dimensional features, in order to improve the accuracy and efficiency of anomaly detection, the maximum and minimum values ​​of each dimensional feature of the renewable energy data are first found, and then a column of feature data is standardized to the [0,1] interval using the following formula:

[0095]

[0096] Among them, D max Indicates the maximum value, the D min Indicates the minimum value.

[0097] Step 302: Divide the standardized renewable energy data into a plurality of data samples according to a preset sliding window.

[0098] In this embodiment, this step mainly includes: dividing the standard renewable energy data into a plurality of data samples along the time axis direction corresponding to the original renewable energy data according to the window width and the sliding step of the sliding window.

[0099] Specifically, in this embodiment, after the feature data of all dimensions are normalized, the window width w and the sliding step t are set first to obtain a window with a shape of N*w. Then, the window slides on the data set along the time axis with a step t, and the segmented data samples are represented as X p ∈R N·w ,p=1,...,T.

[0100] Step 303: construct an isolation forest according to the data samples and the isolation forest algorithm, so as to filter out a plurality of abnormal data samples from a plurality of the data samples through the isolation forest.

[0101] In this embodiment, the steps are mainly as follows: first, construct an isolation forest based on the data sample and the isolation forest algorithm. The construction process of the isolation forest can refer to Figure 4 .like Figure 4 As shown in the figure, based on the isolation forest algorithm, a certain number of random binary trees iTree need to be set first to ensure the diversity of classification. Usually, the detection effect is no longer significantly improved when the number of iTrees exceeds 100.

[0102] To construct an iForest, we randomly select a feature threshold p∈(mind i ,maxd i), i = 1, ..., N, whose size is between the maximum and minimum values ​​of the attribute of the root subsample and is used to split the sample X after sliding slicing p ∈R N·w ,p=1,...,T.

[0103] According to the feature threshold, two subtrees F are obtained each time. l ={X1,...,X j} and F r ={X1,...,X k} are placed on the left and right sides of the node respectively.

[0104] Through the above segmentation method, iTree is iteratively constructed until one of the following conditions is met: a. there is only one data or several identical data in this subtree; b. the tree reaches its maximum height. Thus, the isolation forest is constructed.

[0105] Next, after the isolation forest is constructed, each sample is evaluated in all isolated trees and the path length traversed in each individual isolated tree is recorded. Then, an anomaly score is calculated for each sample using the following formula:

[0106]

[0107]

[0108] H(i)=ln(i)+0.5772156649

[0109] Among them, E(h(X i )) is the total path length h(X i ), c(T) is the average length of isolated trees, and H(i) is the harmonic number.

[0110] Furthermore, since outliers are rare in a data set, they are easier to be isolated than normal data, and the calculated anomaly index S∈(0,1) approaches 1. Normal samples with higher density in the feature space need a large number of splits to be separated from a large amount of data, and their anomaly index S is generally close to 0. If the anomaly index of all data is around 0.5, there are no obvious outliers in the sample.

[0111] After obtaining the anomaly score corresponding to each sample data through the isolation forest, in order to ensure the accuracy of abnormal data detection, the prior anomaly ratio u can be set in advance so that the isolation forest will eventually determine the u*T samples with the largest anomaly index as abnormal data, where T is the number of data samples.

[0112] Step 304: performing anomaly repair on each of the abnormal data samples to obtain repaired renewable energy data according to the anomaly repair.

[0113] In this embodiment, the steps are mainly as follows: according to the time axis direction, obtaining the adjacent data sample whose tail is connected to the beginning of the abnormal data sample; calculating the similarity index between the adjacent data sample and each normal data sample according to a preset similarity index calculation formula; replacing the abnormal data sample with the normal data sample corresponding to the time when the similarity index is the largest.

[0114] Specifically, in this embodiment, for the abnormal slice X detected p , first find the tail and X p Slices that start with X p-w , then in the normal slice sample {X1,X2,...,X (1-u)T} to find the p-w Most Similar X i , through X i+w To replace X p , to achieve the repair of the abnormal data sample. Preferably, in this embodiment, it is set to find X i The similarity index is the Euclidean distance, and the calculation expression of the Euclidean distance is:

[0115]

[0116] Wherein, N is the number of normal slice samples, and DIS() is the Euclidean distance.

[0117] Step 305: performing STL decomposition on the repaired renewable energy data to obtain trend component data, period component data and residual data corresponding to the repaired renewable energy data.

[0118] In this embodiment, the specific step is: after abnormal detection and repair, the renewable energy data needs to be analyzed for each dimension feature d i ∈R k , i=1,...,N perform time series decomposition separately to obtain the decomposed time series feature data; and the time series feature data includes three components, namely, the trend component representing the low-frequency component, the periodic component representing the high-frequency component, and the remainder representing the residual noise component. In this embodiment, the expression of the time series feature data is:

[0119] d i =T i +S i +R i

[0120] Among them, the Ti Represents the trend component of the low-frequency component, S i Represents the periodic component of the high-frequency component, R i The remainder represents the remaining noise component.

[0121] Furthermore, in some implementations of this embodiment, the STL decomposition method is used to obtain the time series feature data. The STL algorithm includes an outer loop and an inner loop. The inner loop uses the Loess algorithm to smooth the trend component and the period component. The specific process is as follows: select a point d away from the point to be fitted. i ,i=1,...,tThe nearest q points are used as the range of the neighborhood, and the distance between q points and the point to be fitted d is calculated respectively. i The maximum distance is λ q (d i ), then for the point to be fitted d i Nearby point d j , whose weight v j (d i ) is calculated as:

[0122]

[0123] Among them, the function W is defined as:

[0124]

[0125] Therefore, according to the definition formula of the weight and the function W, the closer to the point to be fitted, the larger the weight, and the farther away from the point to be fitted, the smaller the weight, until the weight of the farthest qth point decays to 0. The outer loop calculates the robust weight based on the result of the inner loop to reduce the impact of noise on the next inner loop.

[0126] Further, in a certain implementation of this embodiment, it is assumed that S i (k) is the periodic sequence after the kth inner cycle, T i (k) is the trend sequence after the kth inner cycle, and T i The value in (0) is initialized to 0. The inner loop in the STL algorithm is described. The inner loop is divided into the following six steps:

[0127] (1): Remove the trend. Calculate d i -T i (k) As a detrended series.

[0128] (2): Subsequence smoothing. First, the detrended sequence is divided into multiple subsequences, and then each subsequence is smoothed using the Loess algorithm. Then, the subsequences are combined and restored, and the restored sequence is C i (k+1).

[0129] (3): Perform low-pass filtering on the restored sequence. The filtering steps are divided into three steps. First, perform two sliding averages with a step length of the period length, then perform one sliding average with a step length of 3, and finally perform Loess smoothing. The filtered result is recorded as L i (k+1).

[0130] (4): Remove the trend. Use C i (k+1)-L i (k+1) is taken as the detrended sequence and is taken as the new periodic sequence, denoted as S i (k+1).

[0131] (5): Remove the cycle. Use d i -S i (k+1) is the sequence after de-periodization.

[0132] (6): Trend smoothing. Perform Loess smoothing on the de-periodic sequence to obtain a new trend sequence T i (k+1).

[0133] After each inner loop is completed, the original time series is subtracted from the trend component and the period component to obtain the remainder, and the calculation expression of the remainder is:

[0134] R i =d i -T i -S i

[0135] After the inner loop as shown in step (1) to step (6), a large outlier may appear in the residual term. In this case, the robust weight can be assigned through the outer loop for noise reduction. The robust weight reflects the residual term R i extreme degree, if |R i |With a larger value, a smaller weight is assigned.

[0136] Specifically, the calculation method of the robust weight is as follows:

[0137] h=6median(|R i |)

[0138]

[0139] ρ i =B(|R i | / h)

[0140] Wherein, the h represents the value of |R i |6 times the median.

[0141] Therefore, through the calculation formula of the robust weight, when the next inner loop is performed, the weight in Loess is adjusted using the robust weight:

[0142] v j (d i ) new =ρ i v j (d i ) old

[0143] Through the inner loop and the outer loop, the original sequence is decomposed into a trend component representing the trend and direction of the data, a periodic component representing the periodic change law of the data, and a residual term containing the noise in the sequence.

[0144] This embodiment discloses a data mining method based on isolation forest and time series analysis. From the perspective of renewable energy system monitoring, the sliding slice technology significantly improves the accuracy of anomaly detection by virtue of its ability to retain the multidimensional features and time series information of renewable energy data. Furthermore, the sliding slice avoids the cumbersome process of "one-to-one" detection of one feature by one method, thereby improving the detection efficiency. The tree-based isolation forest method not only greatly reduces the time cost, but also ensures the detection accuracy. Among them, STL decomposition helps researchers to grasp the long-term trend of renewable energy data, analyze the unstable factors that may exist in the system, and provide a scientific basis for system optimization and operation. For system reliability, the method provided in this embodiment realizes real-time monitoring of renewable energy systems, can detect potential faults in a timely manner, significantly improves the stability and reliability of the system, and reduces the risk of faults. In terms of renewable energy utilization, time data series decomposition is conducive to in-depth mining of the inherent structure and characteristics of renewable energy data and formulating more reasonable operation and maintenance plans and resource allocation plans based on this, thereby improving the utilization of renewable energy.

[0145] The specific embodiments described above further illustrate the purpose, technical solutions and beneficial effects of the present invention. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. It is particularly pointed out that for those skilled in the art, any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention should be included in the scope of protection of the present invention.

Claims

1. A data mining method based on isolation forest and time series analysis, characterized in that: include: Standardize the acquired raw renewable energy data to obtain standard renewable energy data; Dividing the standard renewable energy data into a plurality of data samples according to a preset sliding window; Performing anomaly detection on each of the data samples according to the pre-constructed isolation forest to obtain a number of abnormal data samples and a number of normal data samples; Performing data repair on the standard renewable energy data according to the abnormal data sample to obtain repaired renewable energy data; The repaired renewable energy data is subjected to time series decomposition to obtain time series characteristic data corresponding to the original renewable energy data.

2. The data mining method based on isolation forest and time series analysis according to claim 1, characterized in that: The standardization process of the acquired raw renewable energy data includes: Performing data decomposition on the original renewable energy data to obtain a plurality of dimensional feature data corresponding to the original renewable energy data; The maximum value and the minimum value corresponding to each dimension of the characteristic data are obtained, so as to perform standardization processing on the characteristic data of each dimension through a preset standardization formula.

3. The data mining method based on isolation forest and time series analysis according to claim 1, characterized in that: The step of dividing the standard renewable energy data into a plurality of data samples according to a preset sliding window includes: Obtaining the window width and sliding step size of the sliding window; According to the window width and the sliding step length, the standard renewable energy data is divided into a plurality of data samples along a time axis direction corresponding to the original renewable energy data.

4. The data mining method based on isolation forest and time series analysis according to claim 1, characterized in that: The method of performing anomaly detection on each of the data samples according to the pre-constructed isolation forest to obtain a plurality of abnormal data samples includes: Obtaining the path length traversed by each of the data samples in each isolated tree; wherein the isolation forest includes a plurality of isolated trees; According to the path length and the number of isolated trees, obtaining a total path length corresponding to each of the data samples; An abnormal score corresponding to each of the data samples is determined according to the total path length and an abnormal value calculation method preset in the isolation forest.

5. The data mining method based on isolation forest and time series analysis according to claim 4 is characterized in that: The method of performing anomaly detection on each of the data samples according to the pre-constructed isolation forest to obtain a plurality of abnormal data samples and a plurality of normal data samples includes: Determining the number of abnormalities of the abnormal data samples according to a preset prior abnormality ratio and the number of the data samples; A plurality of data samples corresponding to the maximum anomaly score are screened according to the anomaly quantity to generate a plurality of the anomaly data samples.

6. The data mining method based on isolation forest and time series analysis according to claim 2, characterized in that: The construction process of the isolation forest is as follows: Randomly select a feature data from the feature data of a plurality of dimensions, and determine the data sample where the feature data is located; Generating a split point between the maximum value and the minimum value corresponding to the characteristic data in the data sample to segment the data sample; Acquire each node generated after segmentation in real time to construct an isolated tree according to a preset segmentation stop condition and a tree structure generated by the current segmentation; wherein the segmentation stop condition includes that the feature data on the current node have the same feature or the currently formed isolated tree reaches a preset depth threshold; The process of constructing the isolated tree is repeated until an isolated forest consisting of a plurality of the isolated trees is obtained.

7. The data mining method based on isolation forest and time series analysis according to claim 3 is characterized in that: The performing data repair on the standard renewable energy data according to the abnormal data sample comprises: According to the time axis direction, acquiring adjacent data samples whose tails are connected to the beginning of the abnormal data sample; Calculating the similarity index between the adjacent data sample and each normal data sample according to a preset similarity index calculation formula; The abnormal data sample is replaced by the normal data sample corresponding to the time when the similarity index is the largest.

8. The data mining method based on isolation forest and time series analysis according to claim 1, characterized in that: The performing time series decomposition on the repaired renewable energy data to obtain time series characteristic data corresponding to the original renewable energy data includes: Performing data decomposition on the repaired renewable energy data to obtain a plurality of dimensional feature data corresponding to the repaired renewable energy data; Perform STL decomposition on the characteristic data of each dimension to obtain time series characteristic data corresponding to the characteristic data of each dimension; wherein the time series characteristic data includes a trend component, a period component and a remainder.

9. A data mining system based on isolation forest and time series analysis, characterized in that: It includes standardization module, data segmentation module, anomaly detection module, anomaly repair module and time series decomposition module; The standardization module is used to perform standardization processing on the acquired original renewable energy data to obtain standard renewable energy data; The data segmentation module is used to segment the standard renewable energy data into a plurality of data samples according to a preset sliding window; The anomaly detection module is used to perform anomaly detection on each of the data samples according to the pre-constructed isolation forest to obtain a plurality of abnormal data samples and a plurality of normal data samples; The abnormality repair module is used to perform data repair on the standard renewable energy data according to the abnormal data sample to obtain repaired renewable energy data; The time series decomposition module is used to perform time series decomposition on the repaired renewable energy data to obtain time series characteristic data corresponding to the original renewable energy data.

10. The data mining system based on isolation forest and time series analysis according to claim 9, characterized in that: The standardization module includes a decomposition unit and a processing unit; The decomposition unit is used to perform data decomposition on the original renewable energy data to obtain a plurality of dimensional feature data corresponding to the original renewable energy data; The processing unit is used to obtain the maximum value and the minimum value corresponding to each dimension of the characteristic data, so as to perform standardization processing on the characteristic data of each dimension through a preset standardization formula.

Images