Meteorological data compression and restoration method based on artificial intelligence algorithm

Abstract

The invention relates to a meteorological data compression and restoration method based on an artificial intelligence algorithm. Constructing a meteorological data compression and restoration model according to the preprocessed data; training the constructed meteorological data compression and reduction model, and forming a compression and reduction model for different meteorological elements through repeated iteration and optimization; and inputting the preprocessed data into the trained meteorological data compression restoration model to obtain restored data. The invention provides a meteorological data compression and restoration model based on an artificial intelligence algorithm by utilizing the characteristics and rules of meteorological data aiming at the problems that the current meteorological information data is large in capacity and difficult to store and transmit, and the storage and transmission efficiency of the meteorological information is effectively improved.

Description

Technical Field

[0001] This invention belongs to the field of meteorological data processing technology, and in particular to a meteorological data compression and restoration method based on artificial intelligence algorithms. Background Technology

[0002] Meteorological data compression and restoration is an essential and crucial step in meteorological data processing and application. Its purpose is to reduce the data volume of meteorological information to facilitate data storage and transmission. Traditional data compression and restoration methods primarily focus on the data storage method, neglecting the data content. Meteorological information, especially forecast information, is voluminous, making data compression even more urgent. Currently, with the continuous enrichment of meteorological observation methods and the increasing refinement of forecast models, meteorological data is in a rapidly developing big data phase, and traditional data compression and restoration methods can no longer meet the demands of data development. Based on artificial intelligence algorithms, modeling from the perspective of meteorological data content to form a compression and restoration model is of great significance for improving the storage, transmission, and utilization efficiency of meteorological information. Summary of the Invention

[0003] The purpose of this invention is to overcome the shortcomings of the prior art and propose a meteorological data compression and restoration method based on artificial intelligence algorithms, so as to reduce the data capacity of meteorological information and improve the transmission and storage efficiency of meteorological information.

[0004] The technical problem solved by this invention is achieved through the following technical solution: A meteorological data compression and restoration method based on artificial intelligence algorithms includes the following steps: Step 1: Data collection and preprocessing; Step 2: Based on the preprocessed data, construct a meteorological data compression and restoration model; Step 3: Train the constructed meteorological data compression and restoration model, and through repeated iterations and optimizations, form compression and restoration models for different meteorological elements; Step 4: Input the preprocessed data into the trained meteorological data compression and restoration model to obtain compressed meteorological data and meteorological data restored by the present invention.

[0005] Moreover, the specific implementation method of step 1 is as follows: collect meteorological data, including 6 meteorological elements: temperature, humidity, meridional wind speed, zonal wind speed, visibility and total cloud cover, convert the wind speed in the meridional direction to obtain relevant information on wind speed and wind direction, and finally perform statistical analysis on the maximum and minimum values ​​of each element, identify and remove or replace outliers, and perform normalization processing to obtain preprocessed data.

[0006] Moreover, the specific implementation method of step 2, which constructs the meteorological data compression and restoration model, is as follows: based on the characteristics of meteorological elements, an Autoencoder is used to construct a training model. The role of the Autoencoder is to learn the features of the input meteorological data, compress it into a low dimension, and be able to reconstruct it when needed.

[0007] Furthermore, the AutoEncoder consists of two parts: an encoder and a decoder. These two parts are responsible for the compression and decompression operations of meteorological data in the model, respectively.

[0008] Where g is the encoder and f is the decoder, the goal is to minimize the reconstruction error; the encoder is used to transform the original input data into a low-dimensional representation. The encoder processes the input data through neural network layers, extracts the important features, and compresses them into a low-dimensional vector. The low-dimensional vector not only contains the key information of the input data, but also removes the redundant parts, reflecting the inherent structure and pattern of the data. The encoder adopts a multi-layer neural network architecture and introduces nonlinear transformation through activation functions, enabling the model to learn complex input feature relationships; The decoder is used to reconstruct the original form of the input data from the low-dimensional representation obtained from the encoder. The decoder uses a multi-layer neural network, which generates an output with the same dimension as the input data by restoring the low-dimensional vector layer by layer. This process is achieved by inversely mapping the hidden layer vectors.

[0009] Moreover, AutoEncoder is implemented using the deep learning framework PyTorch and a training model is built. During model training, the Adam optimizer is used to accelerate convergence and improve training stability.

[0010] The advantages and positive effects of this invention are: This invention involves data collection and preprocessing; constructing a meteorological data compression and restoration model based on the preprocessed data; training the constructed model through repeated iterations and optimizations to form compression and restoration models for different meteorological elements; and inputting the preprocessed data into the trained model to obtain the restored data. Addressing the challenges of large data volume, difficulty in storage and transmission of current meteorological information, this invention proposes a meteorological data compression and restoration model based on artificial intelligence algorithms, utilizing the characteristics and patterns of meteorological data, effectively improving the storage and transmission efficiency of meteorological information. Attached Figure Description

[0011] Figure 1 This is a typical autoencoder network structure diagram; Figure 2 This is a flowchart of the autoencoder network training process of the present invention. Detailed Implementation

[0012] A meteorological data compression and restoration method based on artificial intelligence algorithms includes the following steps: Step 1: Data collection and preprocessing.

[0013] Meteorological data was collected, including six meteorological elements: temperature, humidity, meridional wind speed, zonal wind speed, visibility, and total cloud cover. The wind speed was converted in the meridional direction to obtain relevant information on wind speed and direction. Finally, the maximum and minimum values ​​of each element were statistically analyzed to identify and remove or replace outliers. At the same time, normalization was performed to obtain preprocessed data.

[0014] Step 2: Based on the preprocessed data, construct a meteorological data compression and restoration model.

[0015] Based on the characteristics of meteorological elements, an Autoencoder is used to build a training model. The role of the Autoencoder is to learn the features of the input meteorological data, compress it into a low dimension, and be able to reconstruct it when needed.

[0016] The AutoEncoder consists of two parts: an encoder and a decoder. These two parts are responsible for the compression and decompression of meteorological data, respectively, within the model. The AutoEncoder algorithm is implemented using the deep learning framework PyTorch, and a training model is built upon this framework. During model training, the Adam optimizer is used to improve training stability. Mean squared error is used as the criterion for evaluating model performance during training.

[0017] An autoencoder is a deep learning model whose primary purpose is to learn the features of input data, compress it into a low-dimensional representation, and reconstruct it when needed. The architecture of an autoencoder typically consists of two main components: an encoder and a decoder, which are responsible for data compression and decompression operations, respectively. The basic form of an autoencoder can be represented as:

[0018] Where g is the encoder and f is the decoder, the goal is to minimize the reconstruction error.

[0019] The function of an encoder is to transform raw input data into a low-dimensional representation, usually called a hidden layer or latent space. In this process, the encoder processes the input data through a series of neural network layers, extracting key features and compressing them into a low-dimensional vector. This low-dimensional vector not only contains the key information of the input data but also removes redundant parts, reflecting the inherent structure and patterns of the data. To achieve this goal, encoders typically employ a multi-layer neural network architecture, introducing nonlinear transformations through activation functions, enabling the model to learn complex relationships between input features.

[0020] The function of a decoder is to reconstruct the original form of the input data from the low-dimensional representation obtained from the encoder. A decoder can also employ a multi-layer neural network, generating an output with the same dimensions as the input data by progressively restoring the low-dimensional vectors layer by layer. This process is achieved by inversely mapping the hidden layer vectors.

[0021] During the training of an Autoencoder, the network optimizes the model by minimizing reconstruction errors (such as mean squared error or cross-entropy). Specifically, the model feeds input data into the encoder, generates a low-dimensional vector, and then reconstructs the original data using the decoder. The training goal is to make the reconstructed output as close as possible to the original input, allowing the model to continuously adjust its weights during learning so that the low-dimensional vector obtained in the hidden layers effectively represents the essential features of the input data. In this way, the Autoencoder can capture the latent distribution of high-dimensional data and compress it into more representative low-dimensional features. A typical Autoencoder structure is as follows: Figure 1 As shown.

[0022] The encoder's structure begins with an input single-channel image. The first convolutional layer increases the number of input channels from 1 to 32, and halves the spatial size using a 3x3 convolutional kernel with a stride of 2. Next, a ReLU activation function is used to introduce non-linearity. Then, two residual blocks begin working. The first residual block, ResidualBlock, doubles the number of channels while performing feature extraction through two 3x3 convolutional layers. The second residual block reduces the number of channels, also consisting of two 3x3 convolutional layers. After these two residual blocks, the encoder's final convolutional layer compresses the number of channels back to 1, generating the final encoded representation.

[0023] The decoder's structure begins by increasing the number of channels through a 1x1 convolutional layer, followed by non-linear processing using the ReLU activation function. Next, two residual blocks are used for feature restoration: the first block doubles the number of channels again, while the second block restores the channel count. Then, a deconvolution operation is performed with a 3x3 kernel, a stride of 2, and output padding of 1, aiming to restore the spatial dimension and resize the feature map. Finally, two convolutional layers are used to progressively reduce the number of channels: first, a single convolutional layer compresses the number of channels, and then a 1x1 convolutional layer further reduces the number of channels back to 1. The final layer of the decoder uses a sigmoid or tanh function to generate the restored image.

[0024] This step includes the following steps: like Figure 2 As shown, an Autoencoder is used to thin high-resolution data such as temperature, relative humidity, wind speed, wind direction, visibility, and total cloud cover into low-resolution data. An intelligent restoration model is designed based on the Autoencoder to restore the thinned low-resolution data. The model is iterated and optimized through repeated training to form an intelligent compression and restoration model for temperature, relative humidity, wind speed, wind direction, visibility, and total cloud cover.

[0025] Step 3: Train the constructed meteorological data compression and restoration model, and through repeated iterations and optimizations, form compression and restoration models for different meteorological elements.

[0026] This invention uses the deep learning framework PyTorch to implement the AutoEncoder algorithm and build a training model. PyTorch provides flexible tensor operations and automatic differentiation, greatly facilitating the construction and training of neural networks. During model training, the Adam optimizer is used to accelerate convergence and improve training stability. The working principle of Adam can be summarized as follows: First, it calculates the momentum of the current gradient, i.e., the exponential moving average of the first moment of the gradient, which helps alleviate gradient oscillation problems; second, it calculates the exponential moving average of the squared gradient, i.e., the second moment of the gradient, used to adaptively adjust the learning rate of each parameter. Specifically, Adam ensures better estimation in the early stages of training by correcting for the deviation of the first and second moments, making parameter updates more robust. Each parameter update in Adam considers not only the current gradient but also information from previous gradients, giving it both the acceleration characteristics of a momentum optimizer and the ability to adaptively adjust the learning rate. For the loss function, mean squared error is used as the standard for evaluating model performance during training. Mean squared error is a commonly used regression loss function that can effectively measure the difference between the reconstructed image and the real image, allowing the model to focus on reducing reconstruction error and improving the quality of the generated image during training.

[0027] Step 4: Input the preprocessed data into the trained meteorological data compression and restoration model to obtain the restored data.

[0028] Based on the above-mentioned meteorological data compression and restoration method based on artificial intelligence algorithms, the effectiveness of the present invention is verified by calculating data from the European Centre for Medium-Range Weather Forecasts between 2019 and 2021.

[0029] Step 1: Data collection and preprocessing.

[0030] This embodiment collects 1 / 8 degree weather forecast data from the European Centre for Medium-Range Weather Forecasts (ECMWF) between 2019 and 2021, covering six main meteorological elements: temperature, humidity, 10-meter meridional wind speed, 10-meter zonal wind speed, visibility, and total cloud cover. To better analyze the wind speed data, a zonal-meridian wind speed conversion was performed to obtain relevant information on wind speed and direction. During the data preprocessing stage, statistical analysis of the maximum and minimum values ​​for each element was conducted to identify and remove or replace outliers. The handling of outliers is crucial because they can interfere with model training and affect the final prediction results.

[0031] Next, to facilitate model training, the data was normalized to ensure that all elements had similar numerical ranges. This accelerates model convergence and improves model performance. After data collection, the 2019-2020 data was divided into a training set and the 2021 data into a test set, and a CSV index was created for subsequent data access and management. This systematic data management approach helps improve data processing efficiency and ensures that the model can quickly access the required data during training and testing.

[0032] Step 2, Model Building.

[0033] An intelligent compression and restoration model for relative humidity, wind speed, wind direction, visibility, and total cloud cover is constructed using an Autoencoder. This model mainly consists of two parts: an encoder and a decoder. The encoder compresses the original ECMWF forecast data, reducing its length and width to half of the original data, and its data size to one-quarter of the original. After processing by the encoder, the data is converted into a low-dimensional compressed code. The decoder is responsible for decoding the compressed code output by the encoder, restoring the data to a size consistent with the original data. In this way, the Autoencoder can effectively capture the latent features of the data while maintaining the structural similarity of the data.

[0034] Step 3: Model training.

[0035] Data from 2019 to 2020 was used as the training set to train the constructed compressed-reduction model. Through repeated iterations and optimization, compressed-reduction models for six meteorological elements—temperature, humidity, wind speed, wind direction, total cloud cover, and visibility—were developed. This process not only focused on the training accuracy of the model but also imposed strict requirements on its generalization ability to ensure good performance in practical applications.

[0036] The specific training process includes selecting an appropriate loss function and optimization algorithm to better tune model parameters. At each stage of training, the model's performance metrics are closely monitored to ensure it achieves the expected performance on the validation set. An early stopping strategy is also employed to prevent overfitting and ensure the model's generalization ability on unseen data. This invention sets the initial learning rate for Adam to 0.0001. Multiple experiments have verified that this learning rate effectively balances convergence speed and stability during model training, reducing problems such as non-convergence or excessively rapid convergence caused by inappropriate learning rate selection. To ensure the model's effectiveness and reliability, the maximum number of training iterations is set to 2000. In each training round, in addition to monitoring changes in the loss function, this invention also monitors the structural similarity of the test set to ensure the effectiveness of the model's compression and reconstruction.

[0037] Step 4: Inspection and evaluation.

[0038] Using 2021 data as the test set, the evaluation mainly involved calculating the correlation coefficient, structural similarity, and root mean square error between the compressed and restored data and the original data to verify the reliability of the compression and restoration model proposed in this invention. The evaluation elements included at least temperature, air pressure, wind speed, wind direction, relative humidity, visibility, and total cloud cover. The evaluation results are shown in Table 1.

[0039] Table 1. Validation and evaluation results of the restored data

[0040] As shown in Table 1, the structural similarity between the restored data and the original data is over 95%, and the field similarity is over 99%, ensuring the effectiveness of the compressed data. Therefore, using the model of this invention can significantly reduce the data volume of meteorological information and has high application value.

[0041] It should be emphasized that the embodiments described in this invention are illustrative rather than limiting. Therefore, this invention includes, but is not limited to, the embodiments described in the specific implementation. Any other implementations derived by those skilled in the art based on the technical solutions of this invention are also within the scope of protection of this invention.

Claims

1. A method for compressing and restoring meteorological data based on artificial intelligence algorithms, characterized in that: Includes the following steps: Step 1: Data collection and preprocessing; Step 2: Based on the preprocessed data, construct a meteorological data compression and restoration model; Step 3: Train the constructed meteorological data compression and restoration model, and through repeated iterations and optimizations, form compression and restoration models for different meteorological elements; Step 4: Input the preprocessed data into the trained meteorological data compression and restoration model to obtain compressed meteorological data and meteorological data restored by the present invention.

2. The meteorological data compression and restoration method based on artificial intelligence algorithms according to claim 1, characterized in that: The specific implementation method of step 1 is as follows: collect meteorological data, including 6 meteorological elements: temperature, humidity, meridional wind speed, zonal wind speed, visibility and total cloud cover, convert the wind speed in the meridional direction to obtain relevant information on wind speed and wind direction, and finally perform statistical analysis on the maximum and minimum values ​​of each element, identify and remove or replace outliers, and perform normalization processing to obtain preprocessed data.

3. The meteorological data compression and restoration method based on artificial intelligence algorithms according to claim 1, characterized in that: The specific implementation method of step 2, which constructs the meteorological data compression and restoration model, is as follows: based on the characteristics of meteorological elements, an Autoencoder is used to construct a training model. The role of the Autoencoder is to learn the features of the input meteorological data, compress it into a low dimension, and be able to reconstruct it when needed.

4. The meteorological data compression and restoration method based on artificial intelligence algorithms according to claim 3, characterized in that: The AutoEncoder consists of two parts: an encoder and a decoder. These two parts are responsible for the compression and decompression of meteorological data in the model, respectively. ； Where g is the encoder and f is the decoder, the goal is to minimize the reconstruction error; the encoder is used to transform the original input data into a low-dimensional representation. The encoder processes the input data through neural network layers, extracts the important features, and compresses them into a low-dimensional vector. The low-dimensional vector not only contains the key information of the input data, but also removes the redundant parts, reflecting the inherent structure and pattern of the data. The encoder adopts a multi-layer neural network architecture and introduces nonlinear transformation through activation functions, enabling the model to learn complex input feature relationships; The decoder is used to reconstruct the original form of the input data from the low-dimensional representation obtained from the encoder. The decoder uses a multi-layer neural network, which generates an output with the same dimension as the input data by restoring the low-dimensional vector layer by layer. This process is achieved by inversely mapping the hidden layer vectors.

5. The meteorological data compression and restoration method based on artificial intelligence algorithms according to claim 4, characterized in that: The AutoEncoder is implemented using the deep learning framework PyTorch and a training model is built. During model training, the Adam optimizer is used to accelerate convergence and improve training stability.

Images