Global ionosphere TEC data completion method based on deep learning technology
By constructing a deep learning-based mask autoencoder and Vision Transformer network, the problems of large-area missing data and poor adaptability to dynamic scenes in ionospheric TEC data were solved, achieving efficient and accurate TEC map completion, which is suitable for data processing in the field of Earth science.
Patent Information
- Application Number
- CN202511119818.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-08-11
- Publication Date
- 2025-11-18
AI Technical Summary
Existing data completion methods are inefficient and lack accuracy when processing large areas of missing ionospheric TEC data, and they are poorly adaptable to dynamic scenarios and have difficulty capturing nonlinear features.
We employ a deep learning-based masked autoencoder (MAE) combined with a Vision Transformer (ViT) backbone network to construct an asymmetric encoder-decoder architecture. We use root mean square error (RMSE) as the loss function to train the model to reconstruct the TEC map, and we ensure the robustness of the model through multi-scenario validation.
It has achieved efficient and accurate completion of global ionospheric TEC data, especially during geomagnetic disturbances and solar activity, thus improving the completeness and accuracy of the data.
Smart Images
Figure CN120976071A_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to the technical field of deep learning and space science, more particularly, to a global ionospheric TEC data completion method based on deep learning technology. BACKGROUND
[0002] Ionospheric total electron content (TEC) is an important parameter for describing the state of the ionosphere, and has important significance for long-term change research of the ionosphere and radio signal propagation. However, the existing data completion methods, such as interpolation, traditional machine learning, etc., have low efficiency and insufficient accuracy when dealing with large-area missing data. First, due to the limited distribution of ground-based receivers, especially in unmonitored areas such as the ocean, the data hole rate is more than 50%, the interpolation method has significant errors, and it is extremely difficult to obtain complete TEC data, resulting in a large number of missing areas in the global TEC map. Secondly, the existing data completion methods have poor dynamic scene adaptability: during geomagnetic disturbances, the ionosphere changes dramatically, and existing methods are difficult to capture nonlinear features. Although individual studies have attempted to use CNN, its receptive field is limited and cannot model global spatial correlation. The advantages of MAE in image restoration have not been applied to the field of earth science.
[0003] In recent years, the development of deep learning technology has provided a new way to solve such problems, but existing methods still have limitations in large-area missing data completion, making it difficult to meet the needs of practical applications. Therefore, there is an urgent need for a method that can efficiently and accurately complete global ionospheric TEC data. SUMMARY
[0004] The purpose of the present application is to provide a global ionospheric TEC data completion method based on deep learning technology, which solves the technical problems of low efficiency, insufficient accuracy, and poor dynamic scene adaptability of the existing data completion method in large-area missing data completion. It combines the mask autoencoder (MAE) technology in deep learning to achieve efficient and accurate TEC map completion.
[0005] To achieve the above purpose, the present application adopts the following technical solutions:
[0006] A global ionospheric TEC data completion method based on deep learning technology, comprising:
[0007] S1. Obtain IGS TEC map data, divide the training set, test set and validation set, and preprocess the TEC map data to simulate the data missing situation in actual observation;
[0008] S2. Build a MAE model, use ViT as the backbone network to process the unmasked pixel block in the encoder, design a lightweight structure for the decoder, reconstruct the TEC map, and use the root mean square error (RMSE) as the loss function.
[0009] S3. Training the MAE model and monitoring the RMSE of the training set and the validation set, evaluating the MAE model, calculating the RMSE and SSIM values between the TEC map output by the MAE model and the original map, verifying the completion performance;
[0010] S4. Applying the trained MAE model to actual TEC map completion tasks, verifying in multiple scenarios, and completing the actual TEC map.
[0011] Further, the S1 comprises the following steps:
[0012] S11. Obtain IGS TEC map data from the NASA Crustal Dynamics Data Information System, select data from January 1, 2003 to December 31, 2013 as the training set, data from January 1, 2014 to December 31, 2017 as the test set, and data from 2018 as the validation set;
[0013] S12. Preprocess the TEC map data, including data normalization and random mask processing, to simulate the data missing scenario in actual observation.
[0014] Further, the S2 comprises the following steps:
[0015] S21. Construct an MAE model based on an asymmetric encoder-decoder architecture, use ViT as the backbone network for the encoder, and perform encoding processing on the unmasked pixel blocks, and design the decoder as a lightweight structure for reconstructing the complete TEC map;
[0016] S22. Use the root mean square error RMSE as the loss function, and through training, enable the model to efficiently and accurately complete the large-area missing TEC data.
[0017] Further, the decoder in the S21 comprises a convolution operation, a self-attention mechanism, and an up-sampling module.
[0018] Further, the S3 comprises the following steps:
[0019] S31. Train the MAE model, monitor the RMSE changes of the training set and the validation set, and ensure that the MAE model does not appear overfitting phenomenon;
[0020] S32. Evaluate the MAE model using the test set and the validation set, calculate the RMSE and structural similarity SSIM values between the TEC map output by the model and the original map, and verify the completion performance of the MAE model under different conditions.
[0021] Further, the RMSE change of the training set and the validation set in the S31 is monitored to determine whether the MAE model is overfitting, and when the RMSE of the training set continues to decrease and the RMSE of the validation set no longer decreases or even increases, the training is stopped to avoid overfitting.
[0022] Further, the actual TEC map completion task in the S4 is verified through the introduction of the geomagnetic disturbance index and the solar activity cycle as evaluation variables to ensure the robustness of the MAE model.
[0023] By adopting the technical scheme, the application has the following advantages:
[0024] The application provides a global ionospheric TEC data completion method based on deep learning technology, adopts a mask autoencoder (MAE) combined with a ViT backbone network and a lightweight decoder, and realizes efficient and accurate completion of large-area missing data of a global ionospheric TEC map. By reasonably dividing a training set, a test set and a validation set and performing effective preprocessing and model training evaluation, the performance and generalization ability of the model are ensured. The model is applied to a TEC map completion task in a geomagnetic disturbance period or a solar activity high year period, and good results are obtained, which provides a new possibility for processing large-area missing observation data in the field of earth science. BRIEF DESCRIPTION OF DRAWINGS
[0025] Other features, objects and advantages of the application will become more apparent after reading the detailed description of non-limiting embodiments with reference to the following drawings.
[0026] Figure 1 is a structural framework diagram of the MAE model of the application;
[0027] Figure 2 is a specific structural diagram of the encoder and the decoder of the MAE model of the application;
[0028] Figure 3 is a curve diagram of the average RMSE of the training set and the validation set with the change of the training round;
[0029] Figure 4 is an example diagram of the large-area completion effect of the MAE model of the application;
[0030] Figure 5 is a diagram of the change of the geomagnetic disturbance Dst index during September 9-10, 2015;
[0031] Figure 6 is an example diagram of the completion effect of the MAE model under the condition of geomagnetic disturbance;
[0032] Figure 7are SSIM and RMSE distribution diagrams between the MAE model output map and the mask part of the original map under different geomagnetic activity conditions;
[0033] Figure 8 are SSIM and RMSE distribution diagrams under different mask proportions;
[0034] Figure 9 are SSIM and RMSE distribution diagrams of the mask part of the MAE model output map and the original map in high solar activity years and low solar activity years. DETAILED DESCRIPTION
[0035] The technical solutions of the present application will be specifically described below in combination with the drawings of the specification. In the detailed description, the detailed features and advantages of the present application are described in detail, which are sufficient to enable any person skilled in the art to understand the technical content of the present application and to implement it. According to the specification, claims and drawings disclosed in the specification, those skilled in the art can easily understand the related purposes and advantages of the present application.
[0036] A global ionospheric TEC data completion method based on deep learning technology, comprising:
[0037] S1. Data acquisition and preprocessing: acquiring IGS TEC map data, dividing training set, test set and validation set, and preprocessing TEC map data to simulate data missing in actual observation;
[0038] Wherein, S1 includes the following specific steps:
[0039] S11. Acquire IGS TEC map data from the U.S. National Aeronautics and Space Administration Crustal Dynamics Data Information System, select data from January 1, 2003 to December 31, 2013 as the training set, data from January 1, 2014 to December 31, 2017 as the test set, and data from 2018 as the validation set;
[0040] S12. Preprocess the TEC map data, including data normalization and random mask processing, to simulate data missing scenarios in actual observation. Data normalization can eliminate the influence of different orders of magnitude data, making the model easier to train; random mask processing can simulate various possible missing situations in practice and improve the generalization ability of the model.
[0041] S2. Build a MAE model, the encoder uses ViT as the backbone network to process the unmasked pixel block, the decoder is designed as a lightweight structure, and the TEC map is reconstructed, and the root mean square error RMSE is used as the loss function;
[0042] In the MAE model, the connection relationship and data processing flow of the encoder and the decoder are as followsFigure 1 As shown: Input processing: split the input image into regular and non-overlapping patches, and randomly mask a portion of the patches. Encoder processing: the encoder only processes the patches that are not masked, adopts the Vision Transformer (ViT) structure. It performs linear projection on the visible patches, adds position encoding, and then extracts high-level semantic features through a series of Transformer blocks. Decoder input: the input of the decoder includes two parts: the latent representation of the visible patches output by the encoder and the mask tokens. Mask tokens are shared learnable vectors indicating the positions of the missing patches to be predicted. Decoder processing: the decoder also uses a series of Transformer blocks to generate patches of the reconstructed image, combining the output of the encoder and the position information of the mask tokens. Loss calculation: only the reconstruction results of the masked patches are used for loss calculation, using mean square error (MSE) in the pixel space as the loss function. This asymmetric encoder-decoder architecture enables MAE to operate efficiently when processing a high proportion of masked data, while learning semantic features of images through the reconstruction task.
[0043] Figure 2 The specific structure diagram of the encoder and decoder of the MAE model of the application is shown. In this structure, the encoder adopts Vision Transformer (ViT) as the backbone network to process the pixel blocks that are not masked. The input of the encoder is the visible pixel blocks after linear projection and position encoding, and the features are extracted through multiple Transformer blocks. The decoder is a lightweight structure, and its input includes the latent representation output by the encoder and the mask tokens. The decoder also uses Transformer blocks combined with position encoding, and finally outputs the prediction value of each masked pixel block through a linear layer. At the end of the decoder, the resolution of the feature map is gradually restored to be consistent with the original image through the upsampling module. This design enables the model to maintain high efficiency when processing a high proportion of masked data, while learning semantic features of images through the reconstruction task.
[0044] S2 includes the following specific steps:
[0045] S21. Constructing an MAE model based on an asymmetric encoder-decoder architecture, specifically as shown in Figure 1 As shown, the encoder adopts Vision Transformer (ViT) as the backbone network to encode the pixel blocks that are not masked, and the decoder is designed as a lightweight structure to reconstruct the complete TEC map; the decoder includes convolution operation, self-attention mechanism, and upsampling module, specifically as shown in Figure 2As shown in the figure, the ViT backbone network has strong feature extraction capability and can capture the complex spatial relationship in the TEC data; the lightweight decoder can improve the operation efficiency while ensuring the reconstruction accuracy.
[0046] S22. Use the root mean square error RMSE as the loss function to train the MAE model to efficiently and accurately complete the large-area missing TEC data.
[0047] S3. Train the MAE model and monitor the root mean square error RMSE of the training set and the validation set, evaluate the MAE model, calculate the root mean square error RMSE and structural similarity SSIM value between the TEC map output by the MAE model and the original map, and verify the completion performance;
[0048] S3 includes the following steps:
[0049] S31. Train the MAE model to gradually reduce and stabilize the loss function RMSE, monitor the root mean square error RMSE of the training set and the validation set, and ensure that the MAE model does not overfit; when the root mean square error RMSE of the training set and the validation set gradually decreases and stabilizes, it indicates that the MAE model training effect is good. If the root mean square error RMSE of the training set continues to decrease while the root mean square error RMSE of the validation set increases, stop training to avoid model overfitting.
[0050] In specific embodiments, Figure 3 The average root mean square error (RMSE) of the training set and the validation set is shown as a function of the training round (Epoch). From Figure 3 It can be seen that as the training round increases, the RMSE of the training set and the validation set gradually decreases and stabilizes. This indicates that the MAE model can effectively learn the features of the TEC map during the training process, and the performance of the MAE model gradually improves and eventually stabilizes. In addition, the RMSE of the training set and the validation set has consistent trend, and the RMSE of the validation set does not show significant increase, indicating that the MAE model does not overfit during the training process and has good generalization ability. This further verifies the effectiveness and reliability of the MAE model in completing global ionospheric TEC map data.
[0051] S32. After training, use the test set and the validation set to evaluate the MAE model, calculate the root mean square error RMSE and structural similarity SSIM value between the TEC map output by the model and the original map, and verify the completion performance of the MAE model under different conditions. The root mean square error RMSE is used to measure the numerical difference between the model output and the original data, and the structural similarity SSIM is used to measure the structural similarity between the two, so that the completion effect of the model can be comprehensively evaluated through these two indicators.
[0052] In specific embodiments, Figure 4 The MAE model's completion effect in the case of large-area missing data is demonstrated, including the original IGS TEC map, the IGS TEC map after masking, the MAE model output map, the model residual map, and the RMSE and SSIM statistical values. It can be seen from Figure 4 that the output map of the MAE model is highly consistent with the original TEC map in terms of overall structure and details, especially in the large-area masked region, the MAE model can accurately reconstruct the missing TEC data. The model residual map shows that the difference between the model output and the original data is small, and mainly concentrates within ±2.5TECU. In addition, the RMSE value is low, and the SSIM value is close to 1, indicating that the MAE model can still efficiently and accurately complete the TEC map completion task in the case of large-area missing data, and has good completion performance and robustness.
[0053] S4. Apply the trained MAE model to the actual TEC map completion task, apply the trained MAE model to the actual scene, such as processing TEC map data during geomagnetic disturbance or during high solar activity years, and perform multi-scene verification to complete the missing areas in the model to obtain a complete TEC map. Multi-scene verification is achieved by introducing geomagnetic disturbance index and solar activity period as evaluation variables to ensure the robustness of the MAE model.
[0054] In specific embodiments, Figure 5 is a plot of the Dst index variation during the geomagnetic disturbance from September 9, 2015 to September 10, 2015, used to illustrate the impact of geomagnetic activity on model completion effect. Figure 6 is an example of model completion effect under geomagnetic disturbance conditions, showing the completion performance of the model under complex conditions. Figure 7 is a plot of the SSIM and RMSE distribution between the masked part of the model output map and the masked part of the original map under different geomagnetic activity conditions, quantitatively analyzing the performance of the model. Figure 8 is a plot of the SSIM and RMSE distribution under different masking ratios, evaluating the completion ability and stability of the model under different degrees of missing data. Figure 9 is a plot of the SSIM and RMSE distribution between the model output map and the original map under different geomagnetic activity conditions, analyzing the impact of solar activity on the completion effect of the model. Figures 5-9 The MAE model's completion performance under various complex conditions is demonstrated from different angles. Figure 5 and Figure 6 The completion effect of the model during geomagnetic disturbance is demonstrated through specific cases, Figure 7 and Figure 8Further, the robustness and adaptability of the model under different geomagnetic activity conditions and different mask proportions are verified by quantitative analysis, Figure 9 The influence of solar activity on the model completion effect is analyzed. These results show that the MAE model can efficiently and accurately complete the TEC map completion task in the face of large-area missing data, geomagnetic disturbances, and different solar activity levels, and has a wide application prospect.
[0055] Finally, it should be pointed out that although the present application has been described with reference to the current specific embodiments, those skilled in the art should realize that the above embodiments are only used to illustrate the present application, and are not used as a limitation on the present application, and various equivalent changes or replacements can be made without departing from the concept of the present application, therefore, any changes, modifications of the above embodiments within the scope of the essential spirit of the present application will fall within the scope of the claims of the present application.
Claims
1. A method for completing global ionospheric TEC data based on deep learning technology, characterized in that, The application relates to a method for reconstructing ionospheric TEC (Total Electron Content) map data, comprising the following steps: S1. Obtaining IGS TEC map data, dividing a training set, a test set and a verification set, preprocessing the TEC map data, and simulating data loss in actual observation; S2. Constructing an MAE model, using ViT as a backbone network for the encoder to process pixel blocks without masks, designing a lightweight structure for the decoder to reconstruct the TEC map, and using the root mean square error (RMSE) as a loss function; S3. Training the MAE model and monitoring the RMSE of the training set and the verification set, evaluating the MAE model, calculating the RMSE and the SSIM (Structural Similarity Index) value between the TEC map output by the MAE model and the original map, and verifying the completion performance; S4. Applying the trained MAE model to an actual TEC map completion task and performing multi-scene verification to complete the actual TEC map.
2. The global ionospheric TEC data completion method based on deep learning technology according to claim 1, wherein, The S1 comprises the following steps: S11. Obtaining IGS TEC map data from the Aerospace Bureau Crustal Dynamics Data Information System, selecting data from January 1, 2003 to December 31, 2013 as the training set, data from January 1, 2014 to December 31, 2017 as the test set, and data in 2018 as the verification set; S12. Preprocessing the TEC map data, including data normalization and random mask processing, and simulating the data loss scene in actual observation. 3.The global ionospheric TEC data completion method based on deep learning technology according to claim 1, wherein, The S2 comprises the following steps: S21. Constructing an MAE model based on an asymmetric encoder-decoder architecture, using ViT as a backbone network for the encoder to encode pixel blocks without masks, and designing a lightweight structure for the decoder to reconstruct the complete TEC map; S22. Using the root mean square error (RMSE) as a loss function, and training the MAE model to efficiently and accurately complete the TEC data with large-area loss. 4.The global ionospheric TEC data completion method based on deep learning technology according to claim 3, wherein, The decoder in the S21 comprises a convolution operation, a self-attention mechanism and an up-sampling module. 5.The global ionospheric TEC data completion method based on deep learning technology according to claim 1, wherein, The S3 comprises the following steps: S31. Training the MAE model, monitoring the RMSE changes of the training set and the verification set, and ensuring that the MAE model does not have overfitting phenomenon; S32. Evaluating the MAE model using the test set and the verification set, calculating the RMSE and the structural similarity (SSIM) value between the TEC map output by the model and the original map, and verifying the completion performance of the MAE model under different conditions.
6. The global ionospheric TEC data completion method based on deep learning technology according to claim 5, characterized in that, In the S31, the RMSE changes of the training set and the verification set are monitored to determine whether the MAE model is overfitted, and when the RMSE of the training set continuously decreases while the RMSE of the verification set no longer decreases or even increases, the training is stopped to avoid overfitting.
7. The global ionospheric TEC data completion method based on deep learning technology according to claim 1, wherein, In the S4, the multi-scene verification of the actual TEC map completion task is achieved by introducing the geomagnetic disturbance index and the solar activity cycle as evaluation variables to ensure the robustness of the MAE model.