Solar activity time sequence data compression method based on principal component analysis

Abstract

The invention discloses a solar activity time sequence data compression method based on principal component analysis. The method comprises the following steps: S1, data preprocessing: preprocessing solar activity time sequence data; s2, constructing a data matrix: constructing the processed time sequence data into a data matrix, each row representing a time point, and each column representing a feature or variable; s3, calculating a covariance matrix: performing covariance matrix calculation on the data matrix, wherein the matrix describes correlation among variables in the data; the covariance matrix is subjected to eigenvalue decomposition, the eigenvalues and the corresponding eigenvectors are obtained, the eigenvectors represent the main change direction in the data, original data can be projected into a low-dimensional subspace through PCA, data dimension reduction is achieved by retaining principal components, and the accuracy of the data dimension reduction is improved. Redundant information in the data is reduced, and the compactness and effectiveness of the data are improved.

Description

Technical Field

[0001] The invention relates to the technical field of solar activity time series data compression, and in particular to a solar activity time series data compression method based on principal component analysis. Background Art

[0002] Solar activity time series data refers to data that records the changes in solar activity over time. These data usually include the number of sunspots, coronal mass emission, solar flare frequency, coronal magnetic field intensity, etc.

[0003] Solar time series data is usually observed through observation techniques such as solar observation telescopes, solar photometers and solar observation satellites. Solar observation telescopes are equipped with special solar filters that can observe details on the solar surface, such as sunspots and flares; solar photometers are used to measure changes in the solar brightness, including measurements of solar radiation in different wavelength ranges, so as to understand the radiation characteristics of the sun; solar observation satellites can provide continuous monitoring of solar activity, such as NASA's SDO (Solar Dynamics Observatory).

[0004] Traditional solar activity time series data compression methods usually include some techniques based on signal processing and statistical analysis. One of the most common methods is data smoothing and downsampling based on moving average, difference and filtering techniques. The moving average method can divide the data into windows and calculate the average value of the data in each window, thereby smoothing the data and reducing the sampling rate. However, moving average may cause the temporal resolution of the data to decrease, especially for sudden events or rapidly changing data, which may cause information loss or blurring. When using the difference method, the difference between adjacent time points is calculated to reduce the redundant information of the data, but the difference method may amplify the noise in the data, especially for the low-frequency part, the difference operation may lead to The noise becomes the main change information, thus affecting the quality and interpretability of the data; when using the filtering method, a low-pass filter or a high-pass filter is used to filter the data to remove high-frequency noise or low-frequency background information, but the design of the filter needs to be selected according to the characteristics of the specific data, and the filter may cause distortion or loss of frequency components, especially for nonlinear and non-stationary data, the filtering may not be accurate enough; the downsampling method is to reduce the sampling rate of the data, by retaining data points at fixed time intervals to reduce the amount of data, but downsampling may cause loss of data details, especially for rapidly changing or high-frequency data, downsampling may not capture important information in the data. Summary of the invention

[0005] The purpose of the present invention is to provide a solar activity time series data compression method based on principal component analysis to solve the problems raised in the above background technology.

[0006] To achieve the above purpose, a solar activity time series data compression method based on principal component analysis includes the following steps:

[0007] S1. Data preprocessing: preprocessing solar activity time series data;

[0008] S2. Construct a data matrix: construct the processed time series data into a data matrix, where each row represents a time point and each column represents a feature or variable;

[0009] S3. Calculate the covariance matrix: Calculate the covariance matrix for the data matrix, which describes the correlation between the variables in the data;

[0010] S4. Calculate eigenvalues ​​and eigenvectors: Perform eigenvalue decomposition on the covariance matrix to obtain eigenvalues ​​and corresponding eigenvectors. The eigenvectors represent the main change directions in the data, and the eigenvalues ​​represent the importance or variance of each principal component.

[0011] S5. Select principal components: According to the size of the eigenvalues, select the eigenvectors corresponding to the first few eigenvalues ​​as principal components, and select the principal components that can explain most of the data variance;

[0012] S6, projection and reconstruction: projecting the original data using the selected principal components, mapping the data into the principal component space, and then selectively retaining the projected data so that the data is compressed;

[0013] S7. Evaluate compression effect: compare the difference between the compressed data and the original data to evaluate the compression effect;

[0014] S8. Apply the compressed data: use the compressed data for subsequent analysis.

[0015] Preferably, the data preprocessing in step S1 includes data cleaning, removal of outliers and data normalization, so that the obtained data is more accurate and reliable;

[0016] Data cleaning refers to the final process of discovering and correcting identifiable errors in data files, including checking data consistency, handling invalid values ​​and missing values, etc.

[0017] Removing data outliers is to use the Raida criterion or the Grubbs criterion to remove outliers in the data;

[0018] ①Laida Principle

[0019] When the number of repeated tests is much greater than 10, the experimental standard deviation s is calculated using the Bessel formula, and a certain suspicious value x a The average value x of n results - The difference (x a-x - ) is greater than or equal to 3s, then x a is an outlier;

[0020] When x a After elimination, continue to calculate and judge using the above method until / x a -x / <3s.

[0021] ② Grubbs criterion

[0022] In a set of n data of repeated tests, the maximum absolute value of the residual v is the suspicious value x b , given a confidence interval of p = 0.99 or p = 0.95, that is, a significant level of α = 1-p = 0.01 or 0.05, if the following formula is satisfied, then it can be determined that x b For outliers: (s is the experimental standard deviation, G(α,n) is the critical value table of the Grubbs criterion, n≤30) b After elimination, continue to calculate and judge using the above method. until;

[0023] Data normalization is to convert all data into numbers between [0,1] or [-1,1]. Its purpose is to eliminate the numerical differences between data of different dimensions and avoid excessive network prediction errors caused by large differences in the order of magnitude of input and output data. Linear normalization is used to perform linear transformation on the original data and map the data values ​​to the range between (0,1). The formula is:

[0024]

[0025] Difference standardization retains the relationships existing in the original data and is the simplest way to eliminate dimensions and data value ranges.

[0026] Preferably, in step S2, in constructing the data matrix, it is assumed that there are m indicator variables for principal component analysis: 1 , x 2 ,...x m , there are n evaluation objects in total, and the value of the jth indicator of the i-th evaluation object is x ij Convert to standardized indicators

[0027] is a standardized indicator variable.

[0028] As a preference, in step S3, the correlation coefficient matrix R=(r ij ) m×m

[0029] Among them, r ii =1, r ij =r ji , r ij is the correlation coefficient between the i-th indicator and the j-th indicator.

[0030] Preferably, in step S4, the eigenvalues ​​and eigenvectors are calculated, the eigenvalues ​​are sorted from large to small, and the eigenvalues ​​λ1≥λ2≥...≥λm≥0 of the correlation coefficient matrix R and the corresponding eigenvector u are calculated. 1 ,u 2 ,...,u m , where u j =(u 1j ,u 2j ,...,u nj ) T , m new indicator variables are composed of feature vectors:

[0031]

[0032] Where y 1 is the first principal component, y 2 is the second principal component, ..., y m is the mth principal component.

[0033]

[0034] The principal component y j Information contribution rate;

[0035] The principal component y 1 ,y 2 , ..., y p The cumulative contribution rate of p Close to 1(α p =0.85,0.90,0.95), then select the first p indicator variables y 1 ,y 2 , ..., y p As p principal components, they replace the original m indicator variables, so that a comprehensive analysis of the p principal components can be performed;

[0036] ②Calculate the comprehensive score

[0037] where b j is the information contribution rate of the jth principal component, which can be evaluated based on the comprehensive score.

[0038] Preferably, in step S7, reconstruction error or information retention rate index is used to evaluate the compression effect;

[0039] Use reconstruction error or information retention rate indicators to evaluate compression effect;

[0040] Reconstruction Error: Reconstruction error refers to the error introduced when remapping data from the principal component space back to the original space. In PCA, data is compressed by retaining some principal components, and then reconstructing the data using these principal components. The reconstruction error is the difference between the original data and the reconstructed data, calculated using the mean square error (MSE) or other appropriate distance metrics. The smaller the reconstruction error, the smaller the difference between the reconstructed data and the original data, indicating that the selected principal components can well capture the changes in the original data, and the better the compression effect.

[0041] Variance Retained: Information retention refers to the variance ratio of the original data that can be retained by retaining some principal components. In PCA, the eigenvalues ​​of the principal components represent the variance information in the data, so the information retention is calculated by the cumulative ratio of the eigenvalues. Usually, the principal components whose cumulative ratio of eigenvalues ​​reaches a certain threshold (such as 95%) can be selected as compressed data. The higher the information retention, the more original data information can be retained by the selected principal components, and the more variability is retained in the compressed data.

[0042] Preferably, in step S8, the compressed data is used for pattern recognition, anomaly detection and prediction.

[0043] Compared with the prior art, the present invention has the following beneficial effects:

[0044] 1. Dimensionality reduction and reduction of redundant information: PCA can project the original data into a low-dimensional subspace. By retaining the principal component, that is, the most significant variance direction in the data, it can achieve data dimensionality reduction, help reduce redundant information in the data, and improve the compactness and effectiveness of the data.

[0045] 2. Retain the main features and structural information: PCA retains the most important variance information in the data, so the compressed data still retains the main features and structural information in the original data, and can maintain the key characteristics of the data even while reducing the dimension.

[0046] 3. Remove noise and reduce interference: PCA can extract the main direction of change in the data, which helps to remove noise and reduce interference. By projecting the data into the principal component space, the noise component can be projected into a dimension with lower variance, thereby filtering out the noise.

[0047] 4. Improve data interpretability: PCA transforms the data into a new coordinate system in which each principal component represents a pattern or structure in the data. Therefore, the compressed data can be more easily interpreted and understood, helping researchers to extract meaningful information from it.

[0048] 5. Accelerate calculation and save storage space: By reducing the dimension of data, PCA can reduce the computational complexity and storage requirements of data, thereby accelerating the speed of data processing and analysis and saving storage space. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 It is a schematic diagram of the steps of the method of the present invention. DETAILED DESCRIPTION

[0050] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0051] See also Figure 1 The present invention provides a technical solution: a method for compressing solar activity time series data based on principal component analysis, characterized in that it comprises the following steps:

[0052] S1. Data preprocessing: preprocessing solar activity time series data;

[0053] It should be noted that data preprocessing includes data cleaning, removal of outliers and data normalization to make the obtained data more accurate and reliable;

[0054] Data cleaning refers to the final process of discovering and correcting identifiable errors in data files, including checking data consistency, handling invalid values ​​and missing values, etc.

[0055] Removing data outliers is to use the Raida criterion or the Grubbs criterion to remove outliers in the data;

[0056] ①Laida Principle

[0057] When the number of repeated tests is much greater than 10, the experimental standard deviation s is calculated using the Bessel formula, and a certain suspicious value x a The average value x of n results - The difference (x a -x - ) is greater than or equal to 3s, then x a is an outlier;

[0058] When xa After elimination, continue to calculate and judge using the above method until / x a -x / <3s.

[0059] ② Grubbs criterion

[0060] In a set of n data of repeated tests, the maximum absolute value of the residual v is the suspicious value x b , given a confidence interval of p = 0.99 or p = 0.95, that is, a significant level of α = 1-p = 0.01 or 0.05, if the following formula is satisfied, then it can be determined that x b For outliers: (s is the experimental standard deviation, G(α,n) is the critical value table of the Grubbs criterion, n≤30) b After elimination, continue to calculate and judge using the above method. until;

[0061] Data normalization is to convert all data into numbers between [0,1] or [-1,1]. Its purpose is to eliminate the numerical differences between data of different dimensions and avoid excessive network prediction errors caused by large differences in the order of magnitude of input and output data. Linear normalization is used to perform linear transformation on the original data and map the data values ​​to the range between (0,1). The formula is:

[0062]

[0063] Difference standardization retains the relationships existing in the original data and is the simplest way to eliminate dimensions and data value ranges.

[0064] S2. Construct a data matrix: construct the processed time series data into a data matrix, where each row represents a time point and each column represents a feature or variable;

[0065] It should be noted that in constructing the data matrix, it is assumed that there are m indicator variables for principal component analysis: x 1 , x 2 ,...x m , there are n evaluation objects in total, and the value of the jth indicator of the i-th evaluation object is x ij Convert to standardized indicators

[0066] is a standardized indicator variable.

[0067] S3. Calculate the covariance matrix: Calculate the covariance matrix for the data matrix, which describes the correlation between the variables in the data;

[0068] It should be noted that the correlation coefficient matrix R = (rij ) m×m

[0069] Among them, r ii =1, r ij =r ji , r ij is the correlation coefficient between the i-th indicator and the j-th indicator

[0070] S4. Calculate eigenvalues ​​and eigenvectors: Perform eigenvalue decomposition on the covariance matrix to obtain eigenvalues ​​and corresponding eigenvectors. The eigenvectors represent the main change directions in the data, and the eigenvalues ​​represent the importance or variance of each principal component.

[0071] It should be noted that the eigenvalues ​​are sorted from large to small, and the eigenvalues ​​λ1≥λ2≥...≥λm≥0 of the correlation coefficient matrix R and the corresponding eigenvector u are calculated. 1 ,u 2 ,...,u m , where u j =(u 1j ,u 2j ,...,u nj ) T , m new indicator variables are composed of feature vectors:

[0072]

[0073] Components, calculate the comprehensive evaluation value:

[0074] ① Calculate the eigenvalue λ j The information contribution rate and cumulative contribution rate of (j=1,2,...,m) are called

[0075] The principal component y j Information contribution rate;

[0076] The principal component y 1 ,y 2 , ..., y p The cumulative contribution rate of p Close to 1(α p =0.85,0.90,0.95), then select the first p indicator variables y 1 ,y 2 , ..., y p As p principal components, they replace the original m indicator variables, so that a comprehensive analysis of the p principal components can be performed;

[0077] ②Calculate the comprehensive score

[0078] where b j is the information contribution rate of the jth principal component, which can be evaluated based on the comprehensive score

[0079] S6, projection and reconstruction: projecting the original data using the selected principal components, mapping the data into the principal component space, and then selectively retaining the projected data so that the data is compressed;

[0080] It should be noted that the projected data can be selectively retained to achieve data compression, or the data can be reconstructed using principal components for subsequent analysis or visualization.

[0081] S7. Evaluate compression effect: compare the difference between the compressed data and the original data to evaluate the compression effect;

[0082] It should be noted that the reconstruction error or information retention rate indicators are used to evaluate the compression effect;

[0083] Reconstruction Error: Reconstruction error refers to the error introduced when remapping data from the principal component space back to the original space. In PCA, data is compressed by retaining some principal components, and then reconstructed using these principal components. The reconstruction error is the difference between the original data and the reconstructed data, calculated using the mean square error (MSE) or other appropriate distance metrics. The smaller the reconstruction error, the smaller the difference between the reconstructed data and the original data, indicating that the selected principal components can well capture the changes in the original data, and the better the compression effect.

[0084] Variance Retained: Information retention refers to the variance ratio of the original data that can be retained by retaining some principal components; in PCA, the eigenvalues ​​of the principal components represent the variance information in the data, so the information retention is calculated by the cumulative ratio of the eigenvalues. Usually, the principal components whose cumulative ratio of eigenvalues ​​reaches a certain threshold (such as 95%) can be selected as compressed data; the higher the information retention, the more original data information can be retained by the selected principal components, and the more variability is retained in the compressed data.

[0085] S8. Apply the compressed data: use the compressed data for subsequent analysis;

[0086] It should be noted that using compressed data for pattern recognition, anomaly detection and prediction is an important application that makes full use of PCA compression method:

[0087] ① Pattern Recognition:

[0088] Feature extraction: Various machine learning or statistical methods can be used on the compressed data to extract features and identify patterns in the data. This can include methods based on statistical features, frequency domain features, or time series features.

[0089] Classifier training: Using the extracted features, classifiers can be trained to recognize different patterns or categories. Common classifiers include support vector machines (SVM), decision trees, random forests, etc.

[0090] Model evaluation: Use techniques such as cross-validation to evaluate the performance of the model to ensure that the pattern recognition system has good generalization capabilities.

[0091] ②Anomaly Detection:

[0092] Based on reconstruction error: PCA reconstruction error is used as an indicator for anomaly detection. Abnormal data usually leads to large reconstruction error, so anomalies can be detected by setting a threshold.

[0093] Based on statistical methods: Use the compressed data for statistical analysis, such as calculating the mean, variance and other statistics of the data, and then identify anomalies based on the statistical distribution.

[0094] Based on machine learning methods: Use compressed data to train anomaly detection models, such as density-based methods (such as LOF, One-Class SVM), clustering-based methods, etc.

[0095] ③Prediction:

[0096] Feature selection and modeling: Select appropriate features and build a prediction model based on the specific problem. You can use compressed data for feature selection and then build a model based on the selected features.

[0097] Time series prediction: If the compressed data is time series data, time series models (such as ARIMA, LSTM, etc.) can be used for prediction.

[0098] Supervised learning prediction: Convert the problem into a supervised learning task, use the compressed data as features, and then use supervised learning algorithms (such as linear regression, random forest, neural network, etc.) for prediction.

[0099] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A solar activity time series data compression method based on principal component analysis, Features: The following steps are involved: S1. Data preprocessing: preprocessing solar activity time series data; S2. Construct a data matrix: construct the processed time series data into a data matrix, where each row represents a time point and each column represents a feature or variable; S3. Calculate the covariance matrix: Calculate the covariance matrix for the data matrix, which describes the correlation between the variables in the data; S4. Calculate eigenvalues ​​and eigenvectors: Perform eigenvalue decomposition on the covariance matrix to obtain eigenvalues ​​and corresponding eigenvectors. The eigenvectors represent the main change directions in the data, and the eigenvalues ​​represent the importance or variance of each principal component. S5. Select principal components: According to the size of the eigenvalues, select the eigenvectors corresponding to the first few eigenvalues ​​as principal components, and select the principal components that can explain most of the data variance; S6, projection and reconstruction: projecting the original data using the selected principal components, mapping the data into the principal component space, and then selectively retaining the projected data so that the data is compressed; S7. Evaluate compression effect: compare the difference between the compressed data and the original data to evaluate the compression effect; S8. Apply the compressed data: use the compressed data for subsequent analysis.

2. According to claim 1, a method for compressing solar activity time series data based on principal component analysis, Features: The data preprocessing in step S1 includes data cleaning, removal of outliers and data normalization, so that the obtained data is more accurate and reliable; Data cleaning refers to the final process of discovering and correcting identifiable errors in data files, including checking data consistency, handling invalid values ​​and missing values, etc. Removing data outliers is to use the Raida criterion or the Grubbs criterion to remove outliers in the data; ①Laida Principle When the number of repeated tests is much greater than 10, the experimental standard deviation s is calculated using the Bessel formula, and a certain suspicious value x a The average value x of n results - The difference (x a -x - ) is greater than or equal to 3s, then x a is an outlier; When x a After elimination, continue to calculate and judge using the above method until / x a -x / <3s. ② Grubbs criterion In a set of n data of repeated tests, the maximum absolute value of the residual v is the suspicious value x b , given a confidence interval of p = 0.99 or p = 0.95, that is, a significant level of α = 1-p = 0.01 or 0.05, if the following formula is satisfied, then it can be determined that x b For outliers: (s is the experimental standard deviation, G(α,n) is the critical value table of the Grubbs criterion, n≤30) b After elimination, continue to calculate and judge using the above method. until; Data normalization is to convert all data into numbers between [0,1] or [-1,1]. Its purpose is to eliminate the numerical differences between data of different dimensions and avoid excessive network prediction errors caused by large differences in the order of magnitude of input and output data. Linear normalization is used to perform linear transformation on the original data and map the data values ​​to the range between (0,1). The formula is: Difference standardization retains the relationships existing in the original data and is the simplest way to eliminate dimensions and data value ranges.

3. According to the method for compressing solar activity time series data based on principal component analysis in claim 1, Features: In step S2, in constructing the data matrix, it is assumed that there are m indicator variables for principal component analysis: 1 , x 2 ,...x m , there are n evaluation objects in total, and the value of the jth indicator of the i-th evaluation object is x ij Convert to standardized indicators (i=1, 2, ..., n; j=1, 2, ..., m) are standardized indicator variables.

4. According to claim 1, a method for compressing solar activity time series data based on principal component analysis, It is characterized in that In step S3, the correlation coefficient matrix R is calculated as follows: ij ) m×m (i, j = 1, 2, ..., m), where r ii =1, r ij =r ji , r ij is the correlation coefficient between the i-th indicator and the j-th indicator.

5. According to claim 1, a method for compressing solar activity time series data based on principal component analysis, Features: In step S4, the eigenvalues ​​and eigenvectors are calculated, the eigenvalues ​​are sorted from large to small, and the eigenvalues ​​λ1≥λ2≥...≥λm≥0 of the correlation coefficient matrix R and the corresponding eigenvector u are calculated. 1 ,u 2 ,...,u m , where u j =(u 1j ,u 2j ,...,u nj ) T , m new indicator variables are composed of feature vectors: Where y 1 is the first principal component, y 2 is the second principal component, ..., y m is the mth principal component.

6. The method for compressing solar activity time series data based on principal component analysis according to claim 1, Features: When selecting the principal components in step S5, p (p≤m) principal components are selected and the comprehensive evaluation value is calculated: ① Calculate the eigenvalue λ j The information contribution rate and cumulative contribution rate of (j=1,2,...,m) are called (j=1,2,...,m) is the principal component y j Information contribution rate; The principal component y 1 ,y 2 , ..., y p The cumulative contribution rate of p Close to 1(α p =0.85,0.90,0.95), then select the first p indicator variables y 1 ,y 2 , ..., y p As p principal components, they replace the original m indicator variables, so that a comprehensive analysis of the p principal components can be performed; ②Calculate the comprehensive score where b j is the information contribution rate of the jth principal component, which can be evaluated based on the comprehensive score.

7. The method for compressing solar activity time series data based on principal component analysis according to claim 1, Features: In step S7, the compression effect is evaluated using reconstruction error or information retention rate indicators; Reconstruction Error: Reconstruction error refers to the error introduced when remapping data from the principal component space back to the original space; Variance Retained: Information retention refers to the proportion of variance in the original data that can be retained by retaining some principal components.

8. The method for compressing solar activity time series data based on principal component analysis according to claim 1, It is characterized in that In step S8, the compressed data is used for pattern recognition, anomaly detection and prediction.

Images