Multi-modal F10.7 index prediction method based on image features and time sequence

Abstract

The invention provides a multi-modal F10.7 index prediction method based on image features and time sequence analysis, and mainly solves the problems of low prediction precision and insufficient efficiency in the prior art. According to the implementation scheme, an SDO / HMI solar activity image and F10.7 index time sequence data are sequentially subjected to preprocessing, time alignment and storage to obtain multi-modal data, and the multi-modal data are divided into a training set and a test set; constructing a multi-modal prediction model comprising a three-dimensional convolutional neural network, a time convolutional network, a self-attention module and a full-connection network, and adopting a mean square error as a loss function of the model; inputting the training set into a multi-modal model, and training the multi-modal model through an Adam optimizer; and inputting the test set into the trained multi-modal prediction model, and outputting an F10.7 index prediction value. According to the method, the prediction precision of the F10.7 index is improved, the calculation cost is reduced, the calculation efficiency is improved, and the method can be used for aerospace, communication and space weather early warning.

Description

Technical Field

[0001] The present invention belongs to the intersection of artificial intelligence and space science, and further relates to a multi-modal F 10.7 The index prediction method can be widely used in aerospace, communication and space weather warning. Background Art

[0002] As human beings continue to study solar activity and its impact on the Earth's space environment, 10.7 As an important indicator for measuring the intensity of solar activity, the F index has been widely used in aerospace, satellite communications, navigation systems, and space weather warning. 10.7 The index prediction method still has significant deficiencies in accuracy and adaptability.

[0003] The earliest F 10.7 Index forecasts mainly rely on physical models, which simulate changes in solar activity based on solar physics theory. However, the reliance of these physical models on high-precision observational data limits their practicality. To overcome this problem, statistical methods based on time series analysis have emerged, which make predictions through statistical analysis of a large amount of historical data. However, statistical methods usually assume that the data relationship is linear, which makes it difficult to capture the nonlinear and time-varying characteristics of solar activity, thus affecting the prediction accuracy.

[0004] The introduction of machine learning technology has partially alleviated the above problems. Compared with statistical methods, machine learning performs better in processing nonlinear features. However, traditional machine learning methods rely heavily on manually designed features, especially when processing high-dimensional data, it is difficult to efficiently extract effective features, and the prediction accuracy is still limited.

[0005] In recent years, the rapid development of deep learning technology has 10.7 Exponential prediction provides new ideas. Deep learning methods such as long short-term memory networks (LSTM) have shown great potential in time series modeling, but most existing deep learning models focus on a single mode of time series or image data, ignoring the correlation between solar activity images and time series, and failing to fully explore multi-modal driving characteristics. In addition, deep learning models have high computational overhead during training and are prone to gradient explosion or gradient vanishing problems, especially in years with high solar activity. There is still much room for improvement in prediction accuracy.

[0006] Patent document with application number CN202310775667.8 proposes a F-based solar image data sequence based on multiple bands of solar observations. 10.7 The index prediction method considers the observation images of different solar bands and F 10.7The correlation between the indexes overcomes the problem of insufficient short-term prediction accuracy. However, this method has a high computational cost during the training process, and there is still room for improvement in the fusion of image features and time series features.

[0007] The patent document with application number CN202310134273.4 proposes a F-based method combining SCINet network and empirical mode decomposition (EMD) technology. 10.7 The exponential prediction model decomposes the signal according to the time scale characteristics of the data itself, reduces the time cost of the existing model in the training process, and improves the prediction efficiency. However, this method does not take into account the relationship between solar activity images and F 10.7 Due to the correlation between the indexes, there is still much room for improvement in prediction accuracy. Summary of the invention

[0008] The purpose of the present invention is to address the deficiencies of the above-mentioned prior art and propose a deep learning prediction method based on multimodal data fusion to improve F 10.7 Exponentially improves forecasting accuracy and efficiency and reduces computational costs.

[0009] The technical idea to achieve the purpose of the present invention is to combine solar activity images and time series features and introduce a self-attention mechanism to fully explore the potential correlation between multimodal data to improve F 10.7 The prediction accuracy of the exponential is improved; by introducing the expansion factor in the convolution kernel, that is, inserting intervals between adjacent elements, the receptive field of the convolution kernel is effectively expanded, so that the convolution kernel can cover a wider input area, so as to improve the computing efficiency and reduce the computing cost while keeping the model parameters and calculation amount basically unchanged.

[0010] According to the above technical ideas, the technical solutions of the present invention include the following:

[0011] (1) Constructing multimodal data:

[0012] 1a) Download SDO / HMI solar activity image data of different years from the SDO observation website, and obtain the corresponding year's F from NASA's official website. 10.7 Exponential time series data, and preprocess the two types of data separately;

[0013] 1b) Temporally align the preprocessed image data and time series data to obtain multimodal data, store them in HDF5 format, and divide them into training set and test set by year;

[0014] (2) Building a multimodal prediction model:

[0015] 2a) Establishing a three-dimensional convolutional neural network consisting of a cascade of multiple convolutional layers, pooling layers, and fully connected layers, using the network to extract local and global spatial features of solar activity images and generate highly integrated image feature vectors;

[0016] 2b) Construct a temporal convolutional network consisting of a multi-layer dilated convolution residual module and a cascade of fully connected layers, and use this network to extract F 10.7 The time series features in the exponential time series data generate feature vectors containing important time series information;

[0017] 2c) The image features and time series features are integrated through the self-attention module, the weights of different features are dynamically adjusted, the key features are highlighted, the correlation between the image and time series data is established, and a feature vector with stronger explanatory and predictive capabilities is generated;

[0018] 2d) Input the fused feature vector into a two-layer fully connected network for nonlinear processing and regression analysis.

[0019] Map the feature vector to F 10.7 Index value;

[0020] 2e) After the three-dimensional convolutional neural network is connected in parallel with the temporal convolutional network, it is then connected in series with the feature fusion module and the fully connected network to form a multimodal prediction model;

[0021] (3) Training the multimodal prediction model:

[0022] 3a) Use mean square error (MSE) as the loss function of the multimodal prediction model;

[0023] 3b) Input the training set into the multimodal model and train it using the Adam optimizer until the set number of iterations is reached to obtain a trained multimodal model;

[0024] (4) Input the test set data into the trained multimodal model and output F 10.7 Index forecast value.

[0025] Compared with the prior art, the present invention has the following advantages:

[0026] First, the present invention combines the solar activity image features with the time series features to perform F 10.7 Index prediction, and introduce a self-attention module, which can dynamically adjust the importance of image and time series features according to the characteristics of the input data, improving F 10.7 The forecast accuracy of the index.

[0027] Secondly, the present invention introduces dilated convolution technology into the temporal convolutional network to expand the receptive field of the convolution kernel so that it can cover a wider input area. Under the premise of keeping the model parameters and calculation amount basically unchanged, the calculation efficiency is improved and the calculation cost is reduced. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 It is a flow chart of the implementation of the present invention;

[0029] Figure 2 It is the multimodal F based on image features and time series in this invention. 10.7 Index prediction model;

[0030] Figure 3 It is a block diagram of the dilated convolution residual module in the temporal convolutional network of the present invention;

[0031] Figure 4 It is a schematic diagram of the structure of a multi-layer dilated convolutional residual network based on different dilation factors in the present invention;

[0032] Figure 5 It is a block diagram of the self-attention module in the present invention;

[0033] Figure 6 The 2020 F is simulated by the present invention. 10.7 Index forecast chart;

[0034] Figure 7 The first half of 2024 F is simulated by the present invention 10.7 Index forecast chart. DETAILED DESCRIPTION

[0035] The embodiments and effects of the present invention are further described below in conjunction with the accompanying drawings.

[0036] refer to Figure 1 , the implementation steps of this embodiment are as follows:

[0037] Step 1: Build multimodal data.

[0038] 1.1) Download SDO / HMI solar activity image data of different years from the SDO observation website, and obtain the corresponding year's F from NASA's official website. 10.7 Exponential time series data;

[0039] 1.2) Preprocess the two types of data obtained respectively;

[0040] 1.2.1) Distorted SDO / HMI solar activity images and abnormal F 10.7 The date data corresponding to the index is removed, and the SDO / HMI solar activity image after removal is converted into a single-channel grayscale image, and then cropped to a size of 224×224 pixels;

[0041] 1.2.2) Use the MinMaxScaler function to scale the original F 10.7 The exponential time series data is normalized to the interval [0,1] to obtain the normalized data value X ′ :

[0042]

[0043] In the formula, X is the original data, X min is the minimum value in the original data, X max is the maximum value in the original data;

[0044] 1.3) The preprocessed image data and time series data are time-aligned to obtain multimodal data, which are stored in HDF5 format and divided into training and test sets by year.

[0045] In this example, the data from 2013 to 2019 is divided into the training set, and the data from 2020 to the first half of 2024 is divided into the test set, and it is ensured that both the training set and the test set cover the years of high and low solar activity.

[0046] Step 2: Build a multimodal prediction model.

[0047] Reference Figure 2 , the specific implementation of this step is as follows:

[0048] 2.1) Establish a three-dimensional convolutional neural network consisting of a cascade of multiple convolutional layers, pooling layers, and fully connected layers;

[0049] The multi-layer convolution layer includes four convolution layers, the convolution kernel size of each layer is 3×3×3, the step size is set to 1, and the activation function is ReLU to enhance the nonlinear modeling ability of the network. The image feature extraction process is performed step by step through the four convolution layers, and the input and output channels of each convolution layer are {1,64}, {64,128}, {128,256}, and {256,512} respectively;

[0050] The pooling layer is used to compress the extracted image features;

[0051] The fully connected layer is used to map the compressed image features into a 128-dimensional feature vector to provide effective input for subsequent feature fusion.

[0052] 2.2) Construct a temporal convolutional network consisting of a multi-layer dilated convolution residual module and a cascade of fully connected layers;

[0053] The multi-layer dilated convolution residual module is composed of three layers of dilated convolution residual modules, wherein each layer of dilated convolution residual module is composed of two dilated convolution layers connected in series and a residual connection module connected in parallel, such as Figure 3 As shown in the figure, each dilated convolution layer consists of a dilated convolution module, weight normalization, ReLU activation function and Dropout layer. The ratio of the Dropout layer is set to 0.2 to alleviate the overfitting problem in the model training process. The residual connection module uses a 1×1 convolution kernel to perform dimensional transformation when the number of input and output channels is inconsistent, so that the input is added to the output of the main branch network to achieve residual connection;

[0054] like Figure 4 As shown in the figure, the dilation factor of the dilated convolution layer of each dilated convolution residual module is different. The dilation factor of the first layer is set to 1, the dilation factor of the second layer is 2, and the dilation factor of the third layer is 4, which are used to capture the long-range dependencies on different time scales, so as to better extract time series features.

[0055] The fully connected layer is used to map the time series features extracted by the dilated convolution residual module into a 128-dimensional vector to facilitate input for subsequent feature fusion.

[0056] 2.3) Establish a self-attention module for fusing image features and time series features. The implementation of the self-attention module for fusing image features and time series features is as follows:

[0057] The 128-dimensional image feature vector and the 128-dimensional time series feature vector are concatenated into a 256-dimensional fused feature vector, and the fused feature vector is linearly transformed to generate a query vector Query, a key vector Key, and a value vector Value as the input of the self-attention module. In this example, the query vector Query, the key vector Key, and the value vector Value are all tensor data of [5,5,256]. Figure 5 Only the tensor data example of [3,2,2] is given;

[0058] Calculate the dot product of the query vector Query and the key vector Key to obtain the similarity score between the features, and use the Softmax function to normalize the similarity score to generate attention weights to reflect the influence of each feature on the final prediction result;

[0059] The attention weight is used to perform weighted summation on the value vector Value, dynamically adjust the interactive relationship between image features and time series features, fully explore the complementary information of multimodal data, generate a fusion vector after the attention module, and improve the prediction performance of the model.

[0060] 2.4) Select a fully connected network with two layers to perform regression analysis on the fusion vector:

[0061] The fusion vector after the attention module is input into a two-layer fully connected network. The 256-dimensional fusion feature is compressed into 128 dimensions through the first layer of the fully connected network, and the ReLU activation function is used for nonlinear processing to enhance the expression ability of the model. The 128-dimensional feature after nonlinear processing is then regressed through the second layer of the fully connected network, that is, the feature vector is mapped into a single scalar and the output is F. 10.7 Index forecast value.

[0062] 2.5) After the three-dimensional convolutional neural network is connected in parallel with the temporal convolutional network, it is then connected in series with the feature fusion module and the fully connected network to form a multimodal prediction model.

[0063] Step 3: Train the multimodal prediction model.

[0064] 3.1) The existing mean square error MSE is used as the loss function of the multimodal prediction model, and its formula is as follows:

[0065]

[0066] Where n is the number of samples, y i represents the true value of the i-th sample, Represents the predicted value of the i-th sample;

[0067] 3.2) Input the training set into the multimodal model and train it using the Adam optimizer;

[0068] 3.2.1) Set the learning rate to 0.001, the training batch size to 5, and the maximum number of iterations to 1000 rounds;

[0069] 3.2.2) Use the sliding window method to input data, that is, use 5 consecutive days of historical data to input into the multimodal model to obtain F 10.7 Preliminary forecast results of the index;

[0070] 3.2.3) Calculate the loss value of the multimodal prediction model through the loss function based on the prediction results and the true values ​​in the training set;

[0071] 3.2.4) Use the gradient descent method to back-propagate the loss value and update the network parameters of the multimodal model;

[0072] 3.2.5) Repeat steps 3.2.2 to 3.2.4) until the set maximum number of iterations is reached to obtain a trained multimodal model.

[0073] Step 4: Input the test set data into the trained multimodal model and output F10.7 The predicted value of the index.

[0074] The effects of the present invention are further described below in conjunction with simulation experiments.

[0075] 1. Experimental conditions

[0076] The hardware used in the experiment includes: the processor is Intel Xeon Gold 6148CPU@2.4GHz, the graphics card is two 24GB NVIDIA RTX 3090GPUs. The operating system is 64-bit Ubuntu 18.04, the simulation program is written in Python, and the deep learning framework is PyTorch 1.9.1.

[0077] The evaluation indicators are mean absolute error MAE, root mean square error RMSE, mean relative error MRE and correlation coefficient R; MAE measures the average deviation between the predicted value and the actual value, RMSE emphasizes the impact of large errors, MRE measures the percentage deviation of the predicted value relative to the actual value, and the correlation coefficient R evaluates the linear correlation between the predicted result and the actual value. The calculation formulas are as follows:

[0078]

[0079] In the formula, n is the number of samples, y is i represents the true value of the i-th sample, represents the predicted value of the i-th sample,

[0080] represents the mean of the true values, Represents the mean of the predicted values.

[0081] The lower the MAE, RMSE, and MRE values ​​among these indicators, the higher the prediction accuracy; the higher the correlation coefficient R value, the stronger the correlation of the model prediction.

[0082] The data used in the simulation are self-built multi-modal data, including 4187 days of data from 2013 to the first half of 2024. The SDO / HMI solar activity image data can be downloaded through the link https: / / sdo.gsfc.nasa.gov / assets / img / browse / . 10.7 The index time series data can be downloaded via the link https: / / cdaweb.gsfc.nasa.gov / .

[0083] 2. Experimental content:

[0084] Simulation 1: Under the above experimental conditions, the present invention is used together with the five existing methods LSTM, N-BEATS, TCN, CNN-LSTM, and LSTM-VMD to perform F 10.7 The index is predicted and the prediction indicators are evaluated. The results are shown in Table 1.

[0085] Table 1 The present invention and the five existing methods in F 10.7 Comparative evaluation of index prediction accuracy

[0086]

[0087] The five existing methods in Table 1 are:

[0088] LSTM: A deep learning-based F 10.7 Index forecasting methods;

[0089] N-BEATS: A novel method proposed by Stevenson's group for F 10.7 Deep learning framework for prediction;

[0090] TCN: A prediction algorithm based on temporal convolutional networks constructed by Wang et al.

[0091] CNN-LSTM: A prediction model developed by Luo et al. that combines convolutional neural networks with long short-term memory networks.

[0092] LSTM-VMD: Hao's team uses an innovative method that combines long short-term memory networks with variational mode decomposition;

[0093] It can be seen from Table 1 that the three indicators of mean absolute error MAE, root mean square error RMSE and mean relative error MRE of the prediction results of the present invention are lower than those of the existing methods, and the correlation coefficient R is also higher than those of the five existing methods, indicating that the present invention has better prediction accuracy.

[0094] Simulation 2: Under the above experimental conditions, the present invention and the existing TCN method are used to simulate the F 10.7 The index is predicted and the prediction results of the two methods are obtained respectively. The two prediction results are compared with the true value. The results are as follows Figure 6 As shown. Figure 6 It can be seen that the prediction result of the present invention is closer to the true value, achieving the 10.7 The high-precision prediction of the index proves its effectiveness.

[0095] Simulation 3: Under the above experimental conditions, the F 10.7The index is predicted and the prediction results of the two methods are obtained respectively. The two prediction results are compared with the true value. The results are as follows Figure 7 As shown. Figure 7 It can be seen that the prediction result of the present invention is closer to the true value, achieving the 10.7 The high-precision prediction of the index further proves the effectiveness and applicability of the present invention.

[0096] The above description is only a specific example of the present invention and does not constitute any limitation to the present invention. It is obvious that for professionals in this field, after understanding the content and principles of the present invention, they may make various modifications and changes in form and details without departing from the principles and structures of the present invention. However, these modifications and changes based on the ideas of the present invention are still within the scope of protection of the claims of the present invention.

[0097] It should be noted that the step numbers in the specification and claims of the present invention are only for a clear description of the implementation scheme of the present invention to facilitate understanding, and the order of the numbers is not limited.

Claims

1. A multimodal F based on image features and time series 10.7 The index prediction method is characterized by: include: (1) Constructing multimodal data: 1a) Download SDO / HMI solar activity image data of different years from the SDO observation website, and obtain the corresponding year's F from NASA's official website. 10.7 Exponential time series data, and preprocess the two types of data separately; 1b) Temporally align the preprocessed image data and time series data to obtain multimodal data, store them in HDF5 format, and divide them into training set and test set by year; (2) Building a multimodal prediction model: 2a) Establishing a three-dimensional convolutional neural network consisting of a cascade of multiple convolutional layers, pooling layers, and fully connected layers, using the network to extract local and global spatial features of solar activity images and generate highly integrated image feature vectors; 2b) Construct a temporal convolutional network consisting of a multi-layer dilated convolution residual module and a cascade of fully connected layers, and use this network to extract F 10.7 The time series features in the exponential time series data generate feature vectors containing important time series information; 2c) The image features and time series features are integrated through the self-attention module, the weights of different features are dynamically adjusted, the key features are highlighted, the correlation between the image and time series data is established, and a feature vector with stronger explanatory and predictive capabilities is generated; 2d) Input the fused feature vector into a two-layer fully connected network for nonlinear processing and regression analysis, and map the feature vector to F 10.7 Index value; 2e) After the three-dimensional convolutional neural network is connected in parallel with the temporal convolutional network, it is then connected in series with the feature fusion module and the fully connected network to form a multimodal prediction model; (3) Training the multimodal prediction model: 3a) Use mean square error (MSE) as the loss function of the multimodal prediction model; 3b) Input the training set into the multimodal model and train it using the Adam optimizer until the set number of iterations is reached to obtain a trained multimodal model; (4) Input the test set data into the trained multimodal model and output F 10.7 Index forecast value.

2. The method according to claim 1, characterized in that: In step 1a), the solar activity image data and the time series data are preprocessed as follows: Eliminate distorted or missing image data, as well as abnormal F 10.7 The date data corresponding to the index; The SDO / HMI solar activity images were converted into single-channel grayscale images and cropped to a size of 224 × 224 pixels; Use the MinMaxScaler function to scale the original F 10.7 The exponential time series data is normalized to the interval [0,1], and the calculation formula is as follows: Where X ′ is the normalized data value, X is the original data value; X min is the minimum value in the original data; X max is the maximum value in the original data.

3. The method according to claim 1, characterized in that Step 2a) establishes a three-dimensional convolutional neural network, and the structure functions and parameters of each layer are as follows: The multi-layer convolution layer has a convolution kernel size of 3×3×3 in each layer, a step size of 1, and an activation function of ReLU to enhance the nonlinear modeling ability of the network. The image feature extraction process is performed step by step through four convolution layers, and the input and output channels of each convolution layer are {1,64}, {64,128}, {128,256}, and {256,512} respectively; The pooling layer is used to compress the extracted image features; The fully connected layer is used to map the compressed image features into a 128-dimensional feature vector to provide effective input for subsequent feature fusion.

4. The method according to claim 1, characterized in that Step 2b) establishes a temporal convolutional network, and the structure functions and parameters of each layer are as follows: The multi-layer dilated convolution residual module is composed of three layers of dilated convolution residual modules, wherein each layer of the dilated convolution residual module is composed of two dilated convolution layers connected in series and a residual connection module in parallel, wherein each dilated convolution layer is composed of a dilated convolution module, weight normalization, a ReLU activation function and a Dropout layer, and the Dropout ratio is set to 0.2 to alleviate the overfitting problem in the model training process. When the number of input and output channels is inconsistent, the residual connection module performs dimensional transformation through a 1×1 convolution kernel to add the input to the output of the main branch network to achieve residual connection; The dilation factor of the dilated convolution layer of each dilated convolution residual module is different. The dilation factor of the first layer is set to 1, the dilation factor of the second layer is 2, and the dilation factor of the third layer is 4, which are used to capture the long-range dependencies on different time scales, so as to better extract time series features. The fully connected layer is used to map the time series features extracted by the dilated convolution residual module into a 128-dimensional vector to facilitate input for subsequent feature fusion.

5. The method according to claim 1, characterized in that In step 2c), the image features and time series features are fused through the self-attention module, which is implemented as follows: 2c1) concatenating the 128-dimensional image feature vector and the 128-dimensional time series feature vector into a 256-dimensional fused feature vector; 2c2) Perform linear transformation on the fused feature vector to generate query vector Query, key vector Key and value vector Value as input of the self-attention module; 2c3) Calculate the dot product of the query vector Query and the key vector Key to obtain the similarity score between the features, and use the Softmax function to normalize the score to generate attention weights to reflect the influence of each feature on the final prediction result; 2c4) Use the attention weight to perform weighted summation on the value vector Value, dynamically adjust the interactive relationship between image features and time series features, fully explore the complementary information of multimodal data, generate a fusion vector after the attention module, and improve the prediction performance of the model.

6. The method according to claim 1, characterized in that The step 2d) is specifically implemented as follows: The 256-dimensional fusion features are compressed into 128 dimensions through the first-layer fully connected network, and the ReLU activation function is used for nonlinear processing to enhance the expressiveness of the model; The 128-dimensional features after nonlinear processing are regressed through the second-layer fully connected network, that is, the feature vector is mapped into a single scalar and the output F 10.7 Index forecast value.

7. The method according to claim 1, characterized in that In step 3a), the existing mean square error MSE is used as the loss function of the multimodal prediction model, and its formula is as follows: Where n is the number of samples, y i represents the true value of the i-th sample, Represents the predicted value of the i-th sample.

8. The method according to claim 1, characterized in that In step 3b), the training set is input into the multimodal model and trained using the Adam optimizer. The implementation steps include the following: 3b1) Set the learning rate to 0.001, the training batch size to 5, and the maximum number of iterations to 1000; 3b2) Use the sliding window method to input data, that is, use the historical data of 5 consecutive days to input into the multimodal model to obtain F 10.7 Preliminary forecast results of the index; 3b3) Calculating the loss value of the multimodal prediction model through a loss function based on the prediction results and the true values ​​in the training set; 3b4) Using the gradient descent method to back-propagate the loss value, update the network parameters of the multimodal model based on image features and time series; 3b5) Repeat steps 3b2 to 3b4) until the set maximum number of iterations is reached to obtain a trained multimodal model.

Images