Multimodal F based on image features and time series 10.7 Index prediction methods

Abstract

This invention proposes a multimodal F based on image features and time series analysis. 10.7 The exponential prediction method primarily addresses the issues of low prediction accuracy and insufficient efficiency in existing technologies. Its implementation involves analyzing SDO / HMI solar activity images and F... 10.7 Exponential time series data are preprocessed, time-aligned, and stored sequentially to obtain multimodal data, which is then divided into training and test sets. A multimodal prediction model is constructed, comprising a 3D convolutional neural network, a temporal convolutional network, a self-attention module, and a fully connected network, with mean squared error used as the loss function. The training set is input into the multimodal model, and it is trained using the Adam optimizer. The test set is input into the trained multimodal prediction model, and the output F... 10.7 Exponential prediction value. This invention improves F... 10.7 The index's predictive accuracy reduces computational costs and improves computational efficiency, making it applicable to aerospace, communications, and space weather early warning systems.

Description

Technical fields:

[0001] This invention belongs to the interdisciplinary field of artificial intelligence and space science, and further relates to a multimodal F 10.7 Index-based forecasting methods can be widely applied to aerospace, communications, and space weather early warning systems. Background technology:

[0002] As human research into solar activity and its impact on Earth's space environment continues to deepen, F 10.7 The solar activity index, as an important indicator of solar activity intensity, has been widely used in aerospace, satellite communications, navigation systems, and space weather warning. However, existing F... 10.7 The index forecasting method still has significant shortcomings in terms of accuracy and adaptability.

[0003] The earliest F 10.7 Exponential forecasts primarily rely on physical models that simulate changes in solar activity based on solar physics theories. However, the dependence of these physical models on high-precision observational data limits their practicality. To overcome this problem, statistical methods based on time series analysis have emerged, which make predictions through statistical analysis of large amounts of historical data. However, statistical methods typically assume linear relationships in the data, making it difficult to capture the nonlinear and time-varying characteristics of solar activity, thus affecting prediction accuracy.

[0004] The introduction of machine learning techniques has partially alleviated the aforementioned problems. Compared to statistical methods, machine learning performs better in handling non-linear features. However, traditional machine learning methods rely heavily on manually designed features, especially when dealing with high-dimensional data, making it difficult to efficiently extract effective features, and the prediction accuracy remains limited.

[0005] In recent years, the rapid development of deep learning technology has led to F 10.7 Exponential prediction offers a new approach. Deep learning methods such as Long Short-Term Memory (LSTM) networks have shown great potential in time series modeling; however, most existing deep learning models focus on a single modality of time series or image data, neglecting the correlation between solar activity images and time series, and failing to fully explore multimodal driving characteristics. Furthermore, deep learning models have high computational costs during training and are prone to gradient explosion or vanishing gradient problems, especially under conditions of high solar activity, where there is still significant room for improvement in prediction accuracy.

[0006] Patent document with application number CN202310775667.8 proposes an F method based on solar multi-band observation image data sequences. 10.7 The exponential prediction method, which considers solar observation images at different wavelengths and F... 10.7The correlation between indices overcomes the problem of insufficient short-term prediction accuracy. However, this method has high computational costs during training, and there is still room for optimization in the fusion of image features and time-series features.

[0007] Patent document with application number CN202310134273.4 proposes a method combining SCINet network and Empirical Mode Decomposition (EMD) technology. 10.7 The exponential prediction model reduces the time cost of training existing models and improves prediction efficiency by decomposing signals based on the time-scale characteristics of the data itself. However, this method does not consider the relationship between solar activity images and F... 10.7 The correlation between indices suggests that there is still considerable room for improvement in prediction accuracy. Summary of the Invention:

[0008] The purpose of this invention is to address the shortcomings of the existing technology by proposing a deep learning prediction method based on multimodal data fusion to improve F... 10.7 Improve the accuracy and efficiency of index predictions, and reduce computational costs.

[0009] The technical approach to achieving the objective of this invention is as follows: by combining solar activity images and time-series features, and introducing a self-attention mechanism, the potential correlations between multimodal data are fully explored to improve F... 10.7 The predictive accuracy of the exponent is improved by introducing an expansion factor into the convolution kernel, i.e., inserting a gap between adjacent elements, which effectively expands the receptive field of the convolution kernel and enables it to cover a wider input region. This improves computational efficiency and reduces computational cost while keeping the number of model parameters and computational cost basically unchanged.

[0010] Based on the above technical concept, the technical solution of the present invention includes the following:

[0011] (1) Constructing multimodal data:

[0012] 1a) Download SDO / HMI solar activity image data for different years from the SDO observation website, and obtain the corresponding F data for the year from the NASA website. 10.7 Exponential time series data, and preprocess these two types of data separately;

[0013] 1b) The preprocessed image data and time series data are time-aligned to obtain multimodal data, which is then stored in HDF5 format and divided into training and test sets according to the year.

[0014] (2) Building a multimodal prediction model:

[0015] 2a) Establish a three-dimensional convolutional neural network consisting of multiple convolutional layers, pooling layers and fully connected layers in a cascaded manner. Use this network to extract local and global spatial features of solar activity images and generate highly integrated image feature vectors.

[0016] 2b) Construct a temporal convolutional network consisting of multiple dilated convolutional residual modules and cascaded fully connected layers, and use this network to extract F. 10.7 The temporal features in exponential time series data are used to generate feature vectors containing important temporal information.

[0017] 2c) By fusing image features and time series features through a self-attention module, the weights of different features are dynamically adjusted to highlight key features, establish the correlation between images and time series data, and generate feature vectors that are more interpretable and predictive.

[0018] 2d) The fused feature vectors are input into a two-layer fully connected network for nonlinear processing and regression analysis.

[0019] Map the feature vector to F 10.7 Index value;

[0020] 2e) After connecting the three-dimensional convolutional neural network in parallel with the temporal convolutional network, it is then connected in series with the feature fusion module and the fully connected network to form a multimodal prediction model;

[0021] (3) Training the multimodal prediction model:

[0022] 3a) Use mean squared error (MSE) as the loss function for the multimodal prediction model;

[0023] 3b) Input the training set into the multimodal model and train it using the Adam optimizer until the set number of iterations is reached to obtain the trained multimodal model;

[0024] (4) Input the test set data into the trained multimodal model and output F. 10.7 Index forecast.

[0025] Compared with the prior art, the present invention has the following advantages:

[0026] Firstly, this invention combines solar activity image features with time series features to perform F... 10.7 Exponential prediction, incorporating a self-attention module, dynamically adjusts the importance of image and time series features based on the characteristics of the input data, improving the F-squared accuracy. 10.7 The accuracy of index predictions.

[0027] Secondly, this invention introduces dilated convolution technology into the temporal convolutional network to expand the receptive field of the convolution kernel, enabling it to cover a wider input region. While keeping the number of model parameters and computational cost basically unchanged, it improves computational efficiency and reduces computational cost. Attached image description:

[0028] Figure 1 This is a flowchart illustrating the implementation of the present invention;

[0029] Figure 2 This invention relates to a multimodal F based on image features and time series. 10.7 Index prediction model;

[0030] Figure 3 This is a block diagram of the dilated convolution residual module in the temporal convolutional network of this invention;

[0031] Figure 4 This is a schematic diagram of the multilayer dilated convolutional residual network structure based on different dilation factors in this invention;

[0032] Figure 5 This is a block diagram of the self-attention module in this invention;

[0033] Figure 6 The simulation of the 2020 F using this invention 10.7 Index forecast chart;

[0034] Figure 7 The simulation of F in the first half of 2024 using this invention 10.7 Index prediction chart. Detailed implementation method:

[0035] The embodiments and effects of the present invention will be further described below with reference to the accompanying drawings.

[0036] refer to Figure 1 The implementation steps of this embodiment are as follows:

[0037] Step 1: Construct multimodal data.

[0038] 1.1) Download SDO / HMI solar activity image data for different years from the SDO observation website, and obtain the corresponding F data for the year from the NASA website. 10.7 Exponential time series data;

[0039] 1.2) Preprocess the two types of data obtained separately;

[0040] 1.2.1) Analysis of distorted SDO / HMI solar activity images and anomalies F 10.7 The date data corresponding to the index is removed, and the SDO / HMI solar activity image after removal is converted into a single-channel grayscale image and then cropped to a size of 224×224 pixels;

[0041] 1.2.2) Use the MinMaxScaler function to scale the original F 10.7 Exponential time series data are normalized to the [0,1] interval to obtain the normalized data value X. ′ :

[0042]

[0043] In the formula, X represents the original data. min X is the minimum value in the original data. max The maximum value in the original data;

[0044] 1.3) The preprocessed image data and time series data are time-aligned to obtain multimodal data, which are stored in HDF5 format and divided into training set and test set according to year.

[0045] This example uses data from 2013 to 2019 as the training set and data from 2020 to the first half of 2024 as the test set, ensuring that both the training and test sets cover periods of high and low solar activity.

[0046] Step 2: Build a multimodal prediction model.

[0047] Reference Figure 2 The specific implementation of this step is as follows:

[0048] 2.1) Establish a three-dimensional convolutional neural network consisting of multiple convolutional layers, pooling layers, and fully connected layers cascaded together;

[0049] The multi-layer convolutional layer includes four convolutional layers, each with a kernel size of 3×3×3, a stride of 1, and an activation function of ReLU to enhance the network's nonlinear modeling capability. The image feature extraction process is carried out step by step through the four convolutional layers. The input and output channels of each convolutional layer are {1,64}, {64,128}, {128,256}, and {256,512}, respectively.

[0050] The pooling layer is used to compress the extracted image features;

[0051] The fully connected layer is used to map the compressed image features into a 128-dimensional feature vector, providing effective input for subsequent feature fusion.

[0052] 2.2) Construct a temporal convolutional network consisting of multi-layer dilated convolutional residual modules and cascaded fully connected layers;

[0053] The multi-layer dilated convolutional residual module consists of three layers of dilated convolutional residual modules, where each layer comprises two cascaded dilated convolutional layers and a residual connection module connected in parallel. Figure 3 As shown, each dilated convolutional layer consists of a dilated convolutional module, weight normalization, ReLU activation function, and a Dropout layer. The ratio of the Dropout layer is set to 0.2 to alleviate the overfitting problem during model training. The residual connection module performs dimensionality transformation through a 1×1 convolutional kernel when the number of input and output channels is inconsistent, so that the input is added to the output of the main branch network to achieve residual connection.

[0054] like Figure 4 As shown, the dilation factor of each dilated convolutional residual module is different. The dilation factor of the first layer is set to 1, the dilation factor of the second layer is 2, and the dilation factor of the third layer is 4. These are used to capture long-range dependencies at different time scales, thereby better extracting time series features.

[0055] The fully connected layer is used to map the time-series features extracted by the dilated convolution residual module into a 128-dimensional vector, which facilitates the input for subsequent feature fusion.

[0056] 2.3) Establish a self-attention module for fusing image features and time-series features. The implementation of this self-attention module for fusing image features and time-series features is as follows:

[0057] The 128-dimensional image feature vector and the 128-dimensional time series feature vector are concatenated to form a 256-dimensional fused feature vector. A linear transformation is then performed on the fused feature vector to generate a query vector, a key vector, and a value vector, which serve as input to the self-attention module. In this example, the query vector, key vector, and value vector are all tensor data in the range [5, 5, 256]. Figure 5 The example given is only a tensor data of [3,2,2].

[0058] The dot product of the query vector Query and the key vector Key is calculated to obtain the similarity score between features. The similarity score is then normalized using the Softmax function to generate attention weights, which reflect the degree of influence of each feature on the final prediction result.

[0059] By using attention weights to perform a weighted summation of the value vector, the interaction between image features and time series features is dynamically adjusted, fully exploring the complementary information of multimodal data, generating a fusion vector after passing through the attention module, and improving the model's prediction performance.

[0060] 2.4) A two-layer fully connected network was used to perform regression analysis on the fusion vector:

[0061] The fused vector, after passing through the attention module, is input into a two-layer fully connected network. The first fully connected layer compresses the 256-dimensional fused features into 128 dimensions and applies the ReLU activation function for non-linear processing to enhance the model's expressive power. The second fully connected layer then performs regression analysis on the non-linearly processed 128-dimensional features, mapping the feature vectors to a single scalar and outputting F. 10.7 Index forecast.

[0062] 2.5) After connecting the three-dimensional convolutional neural network in parallel with the temporal convolutional network, it is then connected in series with the feature fusion module and the fully connected network to form a multimodal prediction model.

[0063] Step 3: Train the multimodal prediction model.

[0064] 3.1) The existing mean squared error (MSE) is used as the loss function for the multimodal prediction model, and its formula is as follows:

[0065]

[0066] Where n is the number of samples, y i This represents the true value of the i-th sample. This represents the predicted value of the i-th sample;

[0067] 3.2) Input the training set into the multimodal model and train it using the Adam optimizer;

[0068] 3.2.1) Set the learning rate to 0.001, the training batch size to 5, and the maximum number of iterations to 1000 rounds;

[0069] 3.2.2) Input data using a sliding window method, that is, inputting five consecutive days of historical data into the multimodal model to obtain F. 10.7 Preliminary forecast results for the index;

[0070] 3.2.3) Based on the prediction results and the true values ​​in the training set, calculate the loss value of the multimodal prediction model using the loss function;

[0071] 3.2.4) The gradient descent method is used to backpropagate the loss value and update the network parameters of the multimodal model;

[0072] 3.2.5) Repeat steps 3.2.2 to 3.2.4) until the maximum number of iterations is reached to obtain the trained multimodal model.

[0073] Step 4: Input the test set data into the trained multimodal model and output F.10.7 The predicted value of the index.

[0074] The effects of the present invention will be further described below with reference to simulation experiments.

[0075] 1. Experimental conditions

[0076] The hardware used in the experiment included: an Intel Xeon Gold 6148 CPU @ 2.4GHz processor and two 24GB NVIDIA RTX 3090 GPUs. The operating system was 64-bit Ubuntu 18.04, the simulation program was written in Python, and the deep learning framework was PyTorch 1.9.1.

[0077] The evaluation metrics are Mean Absolute Error (MAE), Root Mean Square Error (RMSE), Mean Relative Error (MRE), and Correlation Coefficient (R). MAE measures the average deviation between predicted and actual values; RMSE emphasizes the impact of larger errors; MRE measures the percentage deviation of predicted values ​​from actual values; and the correlation coefficient (R) assesses the linear correlation between predicted and actual values. Their calculation formulas are as follows:

[0078]

[0079] In the formula, n is the sample size, y i This represents the true value of the i-th sample. This represents the predicted value of the i-th sample.

[0080] The mean of the true values. This represents the mean of the predicted values.

[0081] Among these indicators, the lower the values ​​of MAE, RMSE, and MRE, the higher the prediction accuracy; the higher the correlation coefficient R value, the stronger the correlation of the model prediction.

[0082] The simulation uses self-built multimodal data, covering 4187 days from 2013 to the first half of 2024. SDO / HMI solar activity image data can be downloaded from https: / / sdo.gsfc.nasa.gov / assets / img / browse / . 10.7 The index time series data can be downloaded via the link https: / / cdaweb.gsfc.nasa.gov / .

[0083] 2. Experiment Content:

[0084] Simulation 1: Under the above experimental conditions, the present invention is compared with five existing methods: LSTM, N-BEATS, TCN, CNN-LSTM, and LSTM-VMD. 10.7 The index was predicted and the predictive indicators were evaluated. The results are shown in Table 1.

[0085] Table 1. Comparison of the present invention with five existing methods in F 10.7 Comparative evaluation of index prediction accuracy

[0086]

[0087] The five existing methods in Table 1 are as follows:

[0088] LSTM: A deep learning-based F-mode matrix designed by Zhang's team. 10.7 Index forecasting methods;

[0089] N-BEATS: A proposal from Stevenson's research group applicable to F 10.7 A predictive deep learning framework;

[0090] TCN: A prediction algorithm based on temporal convolutional networks constructed by Wang et al.

[0091] CNN-LSTM: A predictive model developed by Luo et al. that combines convolutional neural networks and long short-term memory networks;

[0092] LSTM-VMD: Hao's team employs an innovative method combining long short-term memory networks with variational mode decomposition;

[0093] As can be seen from Table 1, the mean absolute error (MAE), root mean square error (RMSE), and mean relative error (MRE) of the prediction results of this invention are lower than those of existing methods, and the correlation coefficient (R) is also higher than that of the five existing methods, indicating that this invention has better prediction accuracy.

[0094] Simulation 2: Under the above experimental conditions, the F in 2020 was analyzed using the present invention and the existing TCN method. 10.7 The index was predicted using two different methods, and the prediction results were compared with the actual values. The results are as follows: Figure 6 As shown. From Figure 6 It can be seen that the prediction results of this invention are closer to the true value, achieving a better understanding of F. 10.7 The high accuracy of the index's predictions proves its effectiveness.

[0095] Simulation 3: Under the above experimental conditions, the F-situation in the first half of 2024 was analyzed using the present invention and the existing TCN method. 10.7The index was predicted using two different methods, and the prediction results were compared with the actual values. The results are as follows: Figure 7 As shown. From Figure 7 It can be seen that the prediction results of this invention are closer to the true value, achieving a better understanding of F. 10.7 The high accuracy of the index prediction further demonstrates the effectiveness and applicability of the present invention.

[0096] The above description is merely a specific example of the present invention and does not constitute any limitation on the present invention. Obviously, those skilled in the art, after understanding the content and principles of the present invention, may make various modifications and changes in form and details without departing from the principles and structure of the present invention. However, these modifications and changes based on the ideas of the present invention are still within the scope of protection of the claims of the present invention.

[0097] It should be noted that the step numbers in the specification and claims of this invention are only for the purpose of clearly describing the embodiments of this invention and facilitating understanding, and their order is not limited.

Claims

1. A multimodal F based on image features and time series 10.7 The index prediction method is characterized by, include: (1) Constructing multimodal data: 1a) Download SDO / HMI solar activity image data for different years from the SDO observation website, and obtain the corresponding F data for the year from the NASA website. 10.7 Exponential time series data, and preprocess these two types of data separately; 1b) The preprocessed image data and time series data are time-aligned to obtain multimodal data, which is then stored in HDF5 format and divided into training and test sets according to the year. (2) Building a multimodal prediction model: 2a) Establish a three-dimensional convolutional neural network consisting of multiple convolutional layers, pooling layers and fully connected layers in a cascaded manner. Use this network to extract local and global spatial features of solar activity images and generate highly integrated image feature vectors. 2b) Construct a temporal convolutional network consisting of multiple dilated convolutional residual modules and cascaded fully connected layers, and use this network to extract F. 10.7 The temporal features in exponential time series data are used to generate feature vectors containing important temporal information. 2c) By fusing image features and time series features through a self-attention module, the weights of different features are dynamically adjusted to highlight key features, establish the correlation between images and time series data, and generate feature vectors that are more interpretable and predictive. 2d) The fused feature vectors are input into a two-layer fully connected network for nonlinear processing and regression analysis, mapping the feature vectors to F. 10.7 Index value; 2e) After connecting the three-dimensional convolutional neural network in parallel with the temporal convolutional network, it is then connected in series with the feature fusion module and the fully connected network to form a multimodal prediction model; (3) Training the multimodal prediction model: 3a) Use mean squared error (MSE) as the loss function for the multimodal prediction model; 3b) Input the training set into the multimodal model and train it using the Adam optimizer until the set number of iterations is reached to obtain the trained multimodal model; (4) Input the test set data into the trained multimodal model and output F. 10.7 Index forecast.

2. The method according to claim 1, characterized in that, Step 1a) involves preprocessing the solar activity image data and time series data as follows: Remove distorted or missing image data, as well as data with anomalies F. 10.7 The date data corresponding to the index; Convert the SDO / HMI solar activity image to a single-channel grayscale image and crop it to 224×224 pixels. Use the MinMaxScaler function to scale the original F 10.7 Exponential time series data are normalized to the [0,1] interval using the following formula: In the formula X ′ X represents the normalized data value, while X represents the original data value; min X is the minimum value in the original data. max This represents the maximum value in the original data.

3. The method according to claim 1, characterized in that, The three-dimensional convolutional neural network established in step 2a) has the following structure, function, and parameters for each layer: The multi-layer convolutional layer has a kernel size of 3×3×3, a stride of 1, and an activation function of ReLU to enhance the nonlinear modeling capability of the network. The image feature extraction process is carried out step by step through four convolutional layers. The input and output channels of each convolutional layer are {1,64}, {64,128}, {128,256}, and {256,512}, respectively. The pooling layer is used to compress the extracted image features; The fully connected layer is used to map the compressed image features into a 128-dimensional feature vector, providing effective input for subsequent feature fusion.

4. The method according to claim 1, characterized in that, The temporal convolutional network established in step 2b) has the following structure, function, and parameters for each layer: The multi-layer dilated convolutional residual module consists of three layers of dilated convolutional residual modules. Each layer of dilated convolutional residual module consists of two cascaded dilated convolutional layers and a residual connection module connected in parallel. Each dilated convolutional layer consists of a dilated convolutional module, weight normalization, ReLU activation function, and Dropout layer. The Dropout ratio is set to 0.2 to alleviate the overfitting problem during model training. The residual connection module performs dimensionality transformation through a 1×1 convolutional kernel when the number of input and output channels is inconsistent, so that the input is added to the output of the main branch network to achieve residual connection. The dilation factor of each dilated convolutional residual module is different. The dilation factor of the first layer is set to 1, the dilation factor of the second layer is 2, and the dilation factor of the third layer is 4. These are used to capture long-range dependencies at different time scales, thereby better extracting time series features. The fully connected layer is used to map the time-series features extracted by the dilated convolution residual module into a 128-dimensional vector, which facilitates the input for subsequent feature fusion.

5. The method according to claim 1, characterized in that, In step 2c), the image features and time series features are fused using a self-attention module, as follows: 2c1) Concatenate the 128-dimensional image feature vector with the 128-dimensional time series feature vector to form a 256-dimensional fused feature vector; 2c2) Perform a linear transformation on the fused feature vector to generate a query vector, a key vector, and a value vector, which serve as the input to the self-attention module; 2c3) Calculate the dot product of the query vector Query and the key vector Key to obtain the similarity score between features, and use the Softmax function to normalize the score to generate attention weights to reflect the degree of influence of each feature on the final prediction result. 2c4) The value vector Value is weighted and summed using attention weights to dynamically adjust the interaction between image features and time series features, fully explore the complementary information of multimodal data, generate a fusion vector after passing through the attention module, and improve the prediction performance of the model.

6. The method according to claim 1, characterized in that, Step 2d) is specifically implemented as follows: The 256-dimensional fused features are compressed to 128 dimensions through a first-layer fully connected network, and the ReLU activation function is used for non-linear processing to enhance the expressive power of the model. The 128-dimensional features after nonlinear processing are regressed using a second fully connected network, which maps the feature vectors to a single scalar and outputs F. 10.7 Index forecast.

7. The method according to claim 1, characterized in that, In step 3a), the existing mean squared error (MSE) is used as the loss function of the multimodal prediction model, and its formula is as follows: Where n is the number of samples, y i This represents the true value of the i-th sample. This represents the predicted value of the i-th sample.

8. The method according to claim 1, characterized in that, Step 3b) involves inputting the training set into the multimodal model and training it using the Adam optimizer. The steps include the following: 3b1) Set the learning rate to 0.001, the training batch size to 5, and the maximum number of iterations to 1000 rounds; 3b2) Use a sliding window approach to input data, that is, input historical data from 5 consecutive days into the multimodal model to obtain F. 10.7 Preliminary forecast results for the index; 3b3) Based on the prediction results and the true values ​​in the training set, calculate the loss value of the multimodal prediction model using the loss function; 3b4) The gradient descent method is used to backpropagate the loss value and update the network parameters of the multimodal model based on image features and time series. 3b5) Repeat steps 3b2 to 3b4 until the maximum number of iterations is reached to obtain the trained multimodal model.

Images