Mass time sequence distributed clustering algorithm based on autocorrelation function

Abstract

The invention discloses a mass time sequence distributed clustering algorithm based on an autocorrelation function. The method comprises the following steps: step (1), setting clustering parameters; step (2), carrying out distributed clustering processing; step (3), merging clustering results of the subsets; and step (4), automatically selecting a proper self-correlation order. The method has the advantages that the clustering algorithm based on the self-correlation function is adopted, the method for automatically determining the order of the self-correlation function is provided, the problem that the dimension difference of all time sequences is large can be effectively solved in combination with the distributed algorithm for processing mass data, and the mass data are efficiently processed.

Description

Technical Field

[0001] The present invention relates to the technical field of data processing, and in particular to a massive time series distributed clustering algorithm based on autocorrelation function. Background Art

[0002] The time series clustering algorithm is a data analysis method used to process time series data. The specific algorithm flow is as follows: data preprocessing: clean the time series data to eliminate the influence of noise and outliers; select appropriate clustering algorithms and parameters: select appropriate clustering algorithms according to data characteristics and application requirements, and determine the parameters of the algorithm, such as the number of clusters, distance measurement methods, etc.; perform clustering: run the clustering algorithm to assign time series data to different clusters; evaluate and analyze the results: evaluate the clustering results, such as calculating the purity of the clustering, the silhouette coefficient and other indicators to judge the quality of the clustering, and at the same time, analyze the clustering results to explore the potential laws and trends.

[0003] Although existing time series clustering algorithms can classify time series data with similar characteristics into the same category, which can effectively help people better understand the structure and pattern of the data and discover potential laws and trends, they are not very effective when faced with massive time series or when the dimensional differences between time series are large. Summary of the invention

[0004] In order to solve the above problems, especially to address the deficiencies in the prior art, the present invention provides a massive time series distributed clustering algorithm based on autocorrelation function that can solve the above problems.

[0005] To achieve the above purpose, the present invention adopts the following technical means:

[0006] A massive time series distributed clustering algorithm based on autocorrelation function includes the following steps:

[0007] Step (1), set clustering parameters:

[0008] Clustering is performed based on the autocorrelation function of the time series, and the initial autocorrelation order R is given; based on the total number of time series, an appropriate number of clusters is selected to give the final number of clusters K;

[0009] Step (2), distributed clustering processing:

[0010] Divide the n time series into L subsets, perform clustering on the L subsets based on the autocorrelation order R of step (1), and obtain the cluster center {X l,i ,i=1,…,K},l=1,…,L;

[0011] Step (3), merge subset clustering results:

[0012] Based on step (2), the K cluster centers {X l,i , i=1,…,K}, l=1,…,L, a total of L*K cluster centers are re-matched to obtain new K global cluster centers, and each subset is re-clustered according to the global cluster center until the global cluster center is stable, and the clustering result is output;

[0013] Step (4), automatically select the appropriate autocorrelation order:

[0014] Based on the clustering results of step (3), a clustering matrix is ​​obtained, the autocorrelation order R is increased, and steps (2) and (3) are repeated until the clustering matrix is ​​stable.

[0015] A further solution of the present invention is that R in step (1) is not the autocorrelation order used for the final clustering result, and should be selected starting from a smaller autocorrelation order, R=5, that is, initially clustering is performed only based on the autocorrelation functions of the 1st to 5th orders of the time series.

[0016] A further solution of the present invention is that the number of clusters K in step (1) is selected in the range of 3 to 10.

[0017] A further solution of the present invention is that the specific method of distributed clustering processing in step (2) is:

[0018] There are n time series to be clustered, and the n time series are divided into L subsets Y l ,l=1,…,L, each subset has n l samples;

[0019] The R-order autocorrelation function of each time series is calculated on each subset, and the cluster center {X l,i ,i=1,…,K},l=1,…,L, where {X l,i ,i=1,…,K} are the K cluster centers of the lth subset;

[0020] Collect the cluster centers of all nodes {X l,i ,i=1,…,K}.

[0021] A further solution of the present invention is that the specific method of merging the subset clustering results in step (3) is:

[0022] Take the K cluster centers {X 1,i , i=1,…,k} is the benchmark cluster center, and for the cluster centers {X l,i,i=1,…,K},l=2,…,L are matched with the benchmark cluster center. The specific steps are as follows:

[0023] Step 1: Take the K cluster centers {X 1,i ,i=1,…,K} is the benchmark cluster center;

[0024] Step 2: For the cluster centers {X l,i ,i=1,…,K},l=2,…,L, first compare with the first benchmark cluster center X on Y1 1,1 Match them separately, that is, find the matches between each subset and X on each remaining subset. 1,1 The nearest cluster center Right now:

[0025]

[0026] l=2,…,L, and get the first global cluster center:

[0027]

[0028] Step 3: After obtaining k-1 (k ≥ 2) global cluster centers, we need to compare them with the kth benchmark cluster center X on Y1. 1,k Matching is performed, and each subset is found to be consistent with X on the remaining K-1 subsets. 1,k The nearest cluster center Different from Step 2, it is not necessary to search in all cluster centers of each subset at this time. It is only necessary to search in cluster centers that have not been selected before, that is:

[0029]

[0030] l=2,…,L, and get the kth global cluster center:

[0031]

[0032] Step 4: Repeat Step 3 until K global cluster centers {X k , k=1,…,K}, transmit the K global cluster centers to each subset, and each subset uses the K global cluster centers as the initial cluster centers and re-cluster according to step (2);

[0033] Step 5: Repeat Step 1 to Step 4 above until the change of K global cluster centers is less than the threshold.

[0034] A further solution of the present invention is that the specific method for selecting a suitable autocorrelation order in step (4) is:

[0035] After the K global cluster centers are stable, the clustering matrix C is defined as (c ij ) n×n , that is, if the i-th sample and the j-th sample belong to the same category, c ij =0; if the i-th sample and the j-th sample are not classified into the same category, c ij =1; and for the elements on the diagonal of the matrix ci i=0;

[0036] Based on step (3), L clustering matrices are obtained on L subsets respectively. Among them C l is the clustering matrix of the lth subset. The appropriate autocorrelation order is selected through the following steps to obtain the final clustering result:

[0037] Step 1: Increase the autocorrelation order R used for clustering, and obtain L new clustering matrices according to steps (2) and (3) And calculate on each subset separately And calculate

[0038] Step 2: If max l Δ l >10 -6 ,make Repeat Step 1; if max l Δ l <=10 -6 , output the clustering results.

[0039] Beneficial effects of the present invention:

[0040] The present invention adopts a clustering algorithm based on autocorrelation function, proposes a method for automatically determining the order of autocorrelation function, and combines it with a distributed algorithm for processing massive data to effectively solve the problem of large dimensional differences in various time series and efficiently process massive data. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 It is a flow chart of the present invention. DETAILED DESCRIPTION

[0042] The technical solution of the present invention will be described clearly and completely below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0043] Example

[0044] like Figure 1 As shown, a massive time series distributed clustering algorithm based on autocorrelation function includes the following steps:

[0045] Step (1), set clustering parameters:

[0046] The specific method is to give the initial autocorrelation order R and the final number of clusters K;

[0047] Clustering is performed based on the autocorrelation function of the time series instead of directly clustering the original time series. On the one hand, it can cluster time series with large dimensional differences and find time series with similar shapes, which can reduce the amount of calculation;

[0048] Given the final number of clusters K, it is necessary to select the appropriate number of clusters based on experience in advance, so that the clustering algorithm can be extended to a distributed environment;

[0049] Step (2), distributed clustering processing:

[0050] The specific method is to divide n time series into L subsets, and cluster them according to the autocorrelation order R in each L subset to obtain the cluster center {X l,i ,i=1,…,K},I=1,…,L. The significance of this step is that when there are massive time series, clustering on subsets can significantly reduce the consumption of computing resources;

[0051] Step (3), merge subset clustering results:

[0052] The specific method is to convert the K cluster centers {X l,i , i=1,…,K}, l=1,…,L, a total of L*K cluster centers are re-matched to obtain new K global cluster centers. Each subset is re-clustered according to the global cluster center until the global cluster center is stable. The purpose of this step is that the K cluster centers on each subset are different because the results of clustering in step (2) are different. We need to merge the clustering results on all subsets to obtain the overall clustering result.

[0053] Step (4), automatically select the appropriate autocorrelation order:

[0054] The specific method is to increase the autocorrelation order R, repeat steps (2) and (3) until the clustering matrix is ​​stable. The purpose of this step is to automatically select a suitable autocorrelation order so that when the autocorrelation order used is increased, the clustering result will not change greatly, and a stable clustering result is obtained, and the clustering result uses enough information of the autocorrelation order.

[0055] Furthermore, the specific method for setting the clustering parameters in step (1) is:

[0056] Given the initial autocorrelation order R and the final number of clusters K, R is not the autocorrelation order used in the final clustering result, so you can start with a smaller autocorrelation order, such as R = 5, that is, initially clustering is performed based only on the autocorrelation functions of the 1st to 5th order of the time series, rather than directly using the original time series. The final number of clusters K needs to be selected based on the total number of time series, combined with the knowledge and experience of domain experts, and the common selection range is 3 to 10.

[0057] Furthermore, the specific method of distributed clustering processing in step (2) is:

[0058] Assume that there are n time series to be clustered, and divide the n time series into L subsets Y l ,l=1,…,L, each subset has n l The R-order autocorrelation function of each time series is calculated on each subset, and the cluster center {X l,i ,i=1,…,L},l=1,…,L, where {X l,i ,i=1,…,K} are the K cluster centers of the lth subset.

[0059] Collect the cluster centers of all nodes {X l,i ,i=1,…,K}.

[0060] Furthermore, the specific method of merging the subset clustering results in step (3) is as follows:

[0061] Take the K cluster centers {X 1,i , i=1,…,K} is the benchmark cluster center, and for the cluster centers {X l,i ,i=1,…,K},l=2,…,L are matched with the benchmark cluster center. The specific steps are:

[0062] Step 1: Take the K cluster centers {X 1,i ,i=1,…,K} is the benchmark cluster center.

[0063] Step 2: For the cluster centers {X l,i ,i=1,…,K},l=2,…,L, first compare with the first benchmark cluster center X on Y1 1,1 Match them separately, that is, find the matches between each subset and X on each remaining subset. 1,1 The nearest cluster center Right now

[0064]

[0065] l=2,…,L. And get the first global cluster center

[0066]

[0067] Step 3: After obtaining k-1 (k ≥ 2) global cluster centers, we need to compare them with the kth benchmark cluster center X on Y1. 1,k Matching is performed, and each subset is found to be compatible with X on the remaining L-1 subsets. 1,k The nearest cluster center Unlike Step 2, it is not necessary to search in all cluster centers of each subset at this time. It is only necessary to search in cluster centers that have not been selected before, that is,

[0068]

[0069] l=2,…,L. And get the kth global cluster center

[0070]

[0071] Step 4: Repeat Step 3 until K global cluster centers {X k , k=1,…,K}, transmit the K global cluster centers to each subset, and each subset uses these K global cluster centers as the initial cluster centers and re-cluster according to step (2).

[0072] Step 5: Repeat Step 1 to Step 4 above until the change of K global cluster centers is less than the threshold.

[0073] Furthermore, the specific method for selecting a suitable autocorrelation order in step (4) is:

[0074] After the K global cluster centers are stable, in order to intuitively represent the clustering results, the clustering matrix C is defined as (c ij ) n×n , that is, if the i-th sample and the j-th sample belong to the same category, c ij =0; if the i-th sample and the j-th sample are not classified into the same category, c ij =1; and for the elements c on the diagonal of the matrix ii =0.

[0075] According to step (3), L clustering matrices are obtained on L subsets respectively. Among them C l is the clustering matrix of the lth subset. The following steps are used to select the appropriate autocorrelation order and obtain the final clustering result:

[0076] Step 1: Increase the autocorrelation order R used for clustering, and obtain L new clustering matrices according to steps (2) and (3) And calculate on each subset separately And calculate

[0077] Step 2: If max l Δ l >10 -6 ,make Repeat Step 1. If max l Δ l <=10 -6 , output the clustering results.

[0078] The examples given in the present invention are not intended to limit the implementation methods. For those skilled in the art, other different forms of changes or modifications can be made based on the above description. It is not necessary and impossible to list all the implementation methods here, and the obvious changes or modifications derived therefrom are still within the protection scope of the present invention.

Claims

1. A massive time series distributed clustering algorithm based on autocorrelation function, characterized in that: The following steps are involved: Step (1), set clustering parameters: Clustering is performed based on the autocorrelation function of the time series, and the initial autocorrelation order R is given; based on the total number of time series, an appropriate number of clusters is selected to give the final number of clusters K; Step (2), distributed clustering processing: Divide the n time series into L subsets, perform clustering on the L subsets based on the autocorrelation order R of step (1), and obtain the cluster center {X l,i ,i=1,…,K},l=1,…,L; Step (3), merge subset clustering results: Based on step (2), the K cluster centers {X l,i , i=1,…,K}, l=1,…,L, a total of L*K cluster centers are re-matched to obtain new K global cluster centers, and each subset is re-clustered according to the global cluster center until the global cluster center is stable, and the clustering result is output; Step (4), automatically select the appropriate autocorrelation order: Based on the clustering results of step (3), a clustering matrix is ​​obtained, the autocorrelation order R is increased, and steps (2) and (3) are repeated until the clustering matrix is ​​stable.

2. The massive time series distributed clustering algorithm based on autocorrelation function according to claim 1, characterized in that: In step (1), R is not the autocorrelation order used for the final clustering result. It should be selected starting from a smaller autocorrelation order, R=5, that is, initially clustering is performed based only on the autocorrelation functions of the 1st to 5th orders of the time series.

3. The massive time series distributed clustering algorithm based on autocorrelation function according to claim 1, characterized in that: The number of clusters K in step (1) is selected in the range of 3 to 10.

4. The massive time series distributed clustering algorithm based on autocorrelation function according to claim 1, characterized in that: The specific method of distributed clustering processing in step (2) is: There are n time series to be clustered, and the n time series are divided into L subsets Y l ,l=1,…,L, each subset has n l samples; The R-order autocorrelation function of each time series is calculated on each subset, and the cluster center {X l,i ,i=1,…,K},l=1,…,L, where {X l,i ,i=1,…,K} are the K cluster centers of the lth subset; Collect the cluster centers of all nodes {X l,i ,i=1,…,K}.

5. The massive time series distributed clustering algorithm based on autocorrelation function according to claim 1, characterized in that: The specific method of merging the subset clustering results in step (3) is as follows: Take the K cluster centers {X 1,i , i=1,…,K} is the benchmark cluster center, and for the cluster centers {X l,i ,i=1,…,K},l=2,…,L are matched with the benchmark cluster center. The specific steps are as follows: Step 1: Take the K cluster centers {X 1,i ,i=1,…,K} is the benchmark cluster center; Step 2: For the cluster centers {X l,i ,i=1,…,K},l=2,…,L, first compare with the first benchmark cluster center X on Y1 1,1 Match them separately, that is, find the matches between each subset and X on each remaining subset. 1,1 The nearest cluster center Right now: l=2,…,L, and get the first global cluster center: Step 3: After obtaining k-1 (k ≥ 2) global cluster centers, we need to compare them with the kth benchmark cluster center X on Y1. 1,k Matching is performed, and each subset is found to be compatible with X on the remaining L-1 subsets. 1,k The nearest cluster center Different from Step 2, it is not necessary to search in all cluster centers of each subset at this time. It is only necessary to search in cluster centers that have not been selected before, that is: l=2,…,L, and get the kth global cluster center: Step 4: Repeat Step 3 until K global cluster centers {X k , k=1,…,K}, transmit the K global cluster centers to each subset, and each subset uses the K global cluster centers as the initial cluster centers and re-cluster according to step (2); Step 5: Repeat Step 1 to Step 4 above until the change of K global cluster centers is less than the threshold.

6. The massive time series distributed clustering algorithm based on autocorrelation function according to claim 1, characterized in that: The specific method of selecting a suitable autocorrelation order in step (4) is: After the K global cluster centers are stable, the clustering matrix C is defined as (c ij ) n×n , that is, if the i-th sample and the j-th sample belong to the same category, c ij =0; if the i-th sample and the j-th sample are not classified into the same category, c ij =1; and for the elements c on the diagonal of the matrix ii =0; Based on step (3), L clustering matrices are obtained on L subsets respectively. Among them C l is the clustering matrix of the lth subset. The appropriate autocorrelation order is selected through the following steps to obtain the final clustering result: Step 1: Increase the autocorrelation order R used for clustering, and obtain L new clustering matrices according to steps (2) and (3) And calculate on each subset separately And calculate Step 2: If max l Δ l >10 -6 ,make Repeat Step 1; if max l Δ l <=10 -6 , output the clustering results.

Images