Method, system, device and medium for meteorological data compression based on variational autoencoder

Abstract

This invention belongs to the field of meteorological data compression technology, and discloses a meteorological data compression method, system, device, and medium based on a variational autoencoder, to solve the problems of low compression ratio, poor reconstruction accuracy at high compression ratios, high computational complexity, and lack of targeted processing for uneven latitude and longitude distribution in existing technologies. The method of this invention includes: acquiring meteorological data to be compressed; performing latitude and longitude location encoding; inputting the encoded data into the encoder of a trained variational autoencoder model, which extracts multi-scale features and outputs low-dimensional latent representations through alternating stacked residual convolutional units and sliding window self-attention units; quantizing the representations to generate a compressed bitstream; and inputting the compressed bitstream into a decoder to reconstruct the meteorological data. This invention achieves extremely high compression ratios while significantly maintaining the numerical accuracy of meteorological data and the structural characteristics of extreme events, and also considers computational efficiency.

Description

Technical Field

[0001] This invention belongs to the field of meteorological data compression technology, specifically relating to a meteorological data compression method, system, equipment, and medium based on a variational autoencoder. Background Technology

[0002] Currently, meteorological data compression mainly employs the linear quantization technique built into the GRIB2 format or common image compression algorithms such as JPEG2000. However, these methods fail to fully utilize the strong correlations in the spatial, temporal, and multivariate dimensions of meteorological data, resulting in limited compression ratios, typically struggling to achieve more than 50 times. More importantly, under high compression ratios, the reconstruction accuracy of these methods significantly decreases, making it difficult to meet the data accuracy requirements of subsequent meteorological analysis and forecasting operations.

[0003] In recent years, neural compression methods based on deep learning have made initial progress, but the following shortcomings still exist. First, most existing methods employ a two-stage training strategy, i.e., training either the encoder or decoder first and then jointly optimizing them, resulting in a complex training process and difficult deployment, limiting their widespread application in business systems. Second, some methods use a pure Transformer structure, which, although capable of capturing long-distance dependencies, has a computational complexity on the order of the square of the input size, resulting in excessively long encoding times and difficulty in efficiently processing high-resolution global meteorological data. Furthermore, existing methods generally lack targeted processing for the unique attributes of meteorological data, such as the uneven distribution of latitude and longitude grid areas leading to overly dense polar grids and overly sparse equatorial grids, as well as the special characteristics of polar regions, resulting in poor physical consistency of the reconstructed data globally.

[0004] Therefore, how to achieve extremely high compression ratios while significantly maintaining the numerical accuracy of meteorological data and the structural characteristics of extreme events, and taking into account computational efficiency, is a technical problem that urgently needs to be solved in this field. Summary of the Invention

[0005] Based on the aforementioned shortcomings and deficiencies in the prior art, one of the objectives of this invention is to at least solve one or more of the aforementioned problems in the prior art. In other words, one of the objectives of this invention is to provide a method, system, device, and medium for meteorological data compression based on a variational autoencoder that meets one or more of the aforementioned requirements, so as to achieve an extremely high compression ratio while significantly maintaining the numerical accuracy of meteorological data and the structural characteristics of extreme events, and taking into account computational efficiency.

[0006] To achieve the above-mentioned objectives, the present invention adopts the following technical solution: In a first aspect, the present invention provides a meteorological data compression method based on a variational autoencoder, comprising the following steps: Acquire meteorological data to be compressed; The meteorological data to be compressed is encoded by latitude and longitude to obtain encoded meteorological data; The encoded meteorological data is input into the encoder of the trained variational autoencoder model. The encoder extracts multi-scale features through alternately stacked residual convolutional units and sliding window self-attention units, and outputs low-dimensional latent representations. The low-dimensional latent representation is quantized to generate a compressed bitstream; The compressed bitstream is input into the decoder of the trained variational autoencoder model to reconstruct the reconstructed meteorological data corresponding to the meteorological data to be compressed.

[0007] As a preferred option: The variational autoencoder model is trained using a variational lower bound loss function; The variational lower bound loss function includes a latitude-weighted L1 loss term and a KL divergence regularization term; The weighting coefficient of the latitude-weighted L1 loss is defined as the ratio of the cosine value of each latitude to the sum of the cosine values ​​of all latitudes, in order to compensate for the differences in grid area at different latitudes.

[0008] As a preferred embodiment, the method further includes a step of training the variational autoencoder model: Meteorological data of a preset duration is used as a training set, and the meteorological data contains multiple meteorological variables; The meteorological data in the training set is encoded with latitude and longitude, and the encoded location information is used as an additional channel and concatenated with the data channel of the meteorological data to obtain the model input data. Construct an initial model of a variational autoencoder, which includes an encoder and a decoder; The model input data is input into the initial model of the variational autoencoder, and the encoder and decoder are jointly trained end-to-end using the variational lower bound loss function to obtain the trained variational autoencoder model.

[0009] As a preferred option: Based on the number of latitude and longitude grid points of the meteorological data to be compressed, each latitude grid point value and each longitude grid point value are encoded into vectors in sine and cosine function forms, respectively, as location encoding vectors; The location encoding vector is used as an additional channel and concatenated with the meteorological data to be compressed along the channel dimension to obtain the encoded meteorological data.

[0010] As a preferred embodiment, before inputting the encoded meteorological data into the encoder, the following steps are also included: The encoded meteorological data is divided into multiple non-overlapping image blocks according to the two-dimensional spatial structure composed of latitude grid points and longitude grid points. Each image block has the same size, and each image block is flattened and projected onto a fixed dimension through a linear embedding layer.

[0011] As a preferred option: The encoder includes a mean branch and a variance branch, which output the mean vector and log-variance vector of the low-dimensional latent representation, respectively. The low-dimensional latent representation is obtained by sampling from the Gaussian distribution determined by the mean vector and the log-variance vector using a reparameterization method.

[0012] As a preferred option: The residual convolutional units employ skip connections; The sliding window self-attention unit adopts a cyclic shift window mechanism and introduces a learnable relative position bias table.

[0013] In a second aspect, the present invention provides a meteorological data compression system based on a variational autoencoder, for implementing the meteorological data compression method described in the first aspect.

[0014] Thirdly, the present invention provides an electronic device, the electronic device including a memory, a processor and a computer program, wherein when the computer program is executed by the processor, it implements the meteorological data compression method as described in the first aspect.

[0015] Fourthly, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the meteorological data compression method as described in the first aspect.

[0016] Compared with the prior art, the present invention has the following beneficial effects: This invention employs a variational autoencoder framework, using variational inference to encode meteorological data into latent representations following a Gaussian distribution, thus giving the latent space good structural regularity and continuity. Compared to traditional compression methods that struggle to fully utilize the multidimensional correlations of meteorological data, this invention, through the probabilistic modeling capabilities of the variational autoencoder, can maintain the numerical accuracy of key meteorological elements even at extremely low bit rates, laying the foundation for achieving extremely high compression ratios.

[0017] This invention employs a hybrid encoder-decoder structure combining residual convolution and sliding window self-attention. The residual convolutional units effectively extract local details of the meteorological field through skip connections, mitigating the gradient vanishing problem in deep networks. The sliding window self-attention units utilize a cyclic shifting window mechanism, reducing the computational complexity of self-attention from O(N²) to O(N), enabling the capture of global dependencies in atmospheric circulation with linear complexity. Compared to existing solutions where pure convolutional structures struggle to capture long-range dependencies and pure Transformer structures suffer from excessively high computational complexity, this invention achieves dual preservation of numerical accuracy for key meteorological elements and structural features of extreme events at extremely low bit rates, while also maintaining computational efficiency.

[0018] This invention introduces a latitude-longitude weighted loss function, where the latitude weight coefficient is defined as the ratio of the cosine value of each latitude to the sum of all latitude cosine values. This design effectively compensates for the uneven sampling caused by the small polar grid area and the large equatorial grid area, enabling the model to differentiate errors across different latitude regions during training and avoiding the "dilution" of reconstruction accuracy in polar regions. Compared to existing deep learning methods that generally lack targeted processing of meteorological data-specific attributes, this invention maintains physical consistency of the reconstructed data globally, which is particularly beneficial for the accurate reconstruction of polar climate characteristics.

[0019] This invention employs a single-stage end-to-end joint training strategy, simultaneously optimizing both the encoder and decoder through a variational lower bound loss function, eliminating the need for staged pre-training or complex two-stage tuning. Compared to the two-stage training strategies commonly used in existing neural compression methods, this invention significantly simplifies the training process, lowers the deployment threshold, and facilitates rapid iteration and migration in real-world business systems.

[0020] Further or more detailed beneficial effects will be described in conjunction with specific embodiments in the detailed implementation. Attached Figure Description

[0021] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0022] Figure 1 This is a flowchart illustrating the meteorological data compression method described in Embodiment 1 of the present invention.

[0023] Figure 2 This is a schematic diagram of the meteorological data compression system described in Embodiment 2 of the present invention.

[0024] Figure 3 This is a structural diagram of the electronic device described in Embodiment 3 of the present invention.

[0025] Icon labels: 300. Electronic devices; 301. Processor; 302. Communication bus; 303. User interface; 304. Network interface; 305. Memory. Detailed Implementation

[0026] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention.

[0027] In the following description, several embodiments of the present invention are provided. Different embodiments can be substituted or combined. Therefore, the present invention can also be considered to include all possible combinations of the same and / or different embodiments described. Thus, if one embodiment includes features A, B, and C, and another embodiment includes features B and D, then the present invention should also be considered to include embodiments containing one or more other possible combinations of A, B, C, and D, even if such embodiments are not explicitly described in the following text.

[0028] The following description provides examples and does not limit the scope, applicability, or examples set forth in the claims. Changes may be made to the function and arrangement of the described elements without departing from the scope of the invention. Various processes or components may be appropriately omitted, substituted, or added to the various examples. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Furthermore, features described with respect to some examples may be combined into other examples.

[0029] To facilitate a better understanding of the embodiments of the present invention, its application scenarios will be explained before providing a detailed explanation of the specific implementation methods.

[0030] The meteorological data compression method described in the embodiments of this specification is applied to scenarios of meteorological data storage, transmission, and sharing. In these scenarios, the application of the meteorological data compression method aims to compress high-resolution, multivariate massive meteorological data into low-dimensional latent representations and generate compressed bitstreams by using a deep learning model based on variational autoencoders, thereby significantly reducing the data storage space and transmission bandwidth requirements. At the same time, when needed, the original meteorological data can be reconstructed from the compressed bitstream with high fidelity, maintaining the numerical accuracy of key meteorological elements and the structural characteristics of extreme events, solving the problems of limited compression ratio, poor reconstruction accuracy at high compression ratios, and high computational complexity in existing technologies.

[0031] The following is a brief explanation of the variational autoencoder, residual convolutional unit, sliding window self-attention unit, low-dimensional latent representation, compressed bitstream, variational lower bound loss function, dimension-weighted L1 loss term, KL divergence regularization term, end-to-end joint training, and reparameterization method involved in several embodiments of this specification: A variational autoencoder (VAE) is a generative deep learning model consisting of an encoder and a decoder. The encoder maps the input data to a probability distribution (usually Gaussian) in the latent space, and the decoder samples from this distribution to reconstruct the original data. Unlike traditional autoencoders, the variational autoencoder imposes regularization constraints on the latent space, making the latent representation continuous and structured, which facilitates subsequent compression and generation tasks. In this specification, the variational autoencoder is used to compress high-dimensional meteorological data into low-dimensional latent representations.

[0032] A residual convolutional unit (RCU) is a module based on a convolutional neural network. By introducing skip connections, it directly adds the input to the output, thus mitigating the vanishing gradient problem in deep networks and enabling the network to learn deeper features more effectively. The RCU in this specification consists of two convolutional layers, two batch normalization layers, and two LeakyReLU activation functions, used to extract local spatial features from meteorological data, such as the eye structure of a typhoon and the temperature gradient of a front.

[0033] The sliding window self-attention unit is a module based on the self-attention mechanism. It uses a cyclic shifting window mechanism to divide the feature map into multiple local windows, calculates self-attention within each window, and achieves cross-window information interaction through window shifting. Compared to global self-attention, sliding window self-attention reduces the computational complexity from O(N²) to O(N), significantly improving computational efficiency. In this specification, the sliding window self-attention unit is used to capture long-range dependencies in atmospheric circulation, such as teleconnections and monsoon systems.

[0034] Low-dimensional latent representation refers to the low-dimensional vector representation obtained by the encoder after mapping high-dimensional input meteorological data to a latent space. Raw meteorological data typically has dozens of variable channels, hundreds of latitude grid points, and thousands of longitude grid points, resulting in a massive data volume. Through multi-level compression by the encoder, the low-dimensional latent representation condenses the high-dimensional information of the original data into a vector space of hundreds of dimensions, which is key to achieving a high compression ratio. In this specification, the latent space dimension is preferably 256.

[0035] A compressed bitstream refers to a sequence of discrete integers obtained by quantizing a low-dimensional latent representation. Since the low-dimensional latent representation is a continuous floating-point number, it cannot be directly used for storage and transmission. By uniformly quantizing it into discrete integers, a compressed bitstream is generated, thereby achieving the actual compressed storage and transmission of meteorological data.

[0036] The variational lower bound loss function, also known as ELBO (Evidence Lower Bound), is the loss function used during the training of a variational autoencoder. This loss function consists of two parts: a reconstruction error term and a KL divergence regularization term. The reconstruction error term measures the difference between the decoder-reconstructed data and the original input data, while the KL divergence regularization term constrains the distribution of the latent representation to approximate a standard Gaussian distribution. By minimizing the variational lower bound loss function, the model can ensure reconstruction quality while maintaining a good structural regularity in the latent space.

[0037] The latitude-weighted L1 loss term is an improved reconstruction error term that introduces latitude weights into the traditional L1 loss. Since the Earth is a sphere, the grid area varies significantly across different latitudes: the grid area is large at the equator and small at the poles. The latitude-weighted L1 loss assigns a weight proportional to the cosine of each latitude grid point, causing the model to focus more on the larger equatorial region during training while appropriately reducing the weight of the smaller polar regions. This compensates for the impact of the varying grid areas at different latitudes, ensuring the reconstructed data maintains physical consistency globally.

[0038] The Kullback-Leibler divergence regularization term measures the difference between the latent distribution of the encoder output and the standard Gaussian distribution. By constraining the distribution of the latent representation to conform to the standard Gaussian distribution, the latent space gains continuity and integrity, facilitating subsequent sampling, quantization, and compression operations. The Kullback-Leibler divergence regularization term is one of the core features that distinguishes variational autoencoders from traditional autoencoders.

[0039] End-to-end joint training refers to treating the encoder and decoder as a whole, simultaneously optimizing all parameters of both the encoder and decoder using the same loss function, rather than training them separately in stages or modules. This specification employs a single-stage end-to-end joint training strategy, simultaneously optimizing both the encoder and decoder using a variational lower bound loss function, simplifying the training process and improving the overall model performance and convergence efficiency.

[0040] Reparameterization (also known as reparameterization trick) is a key technique in variational autoencoder training, used to address the problem that gradients from sampling operations within a probability distribution cannot be directly backpropagated. Specifically, for a latent representation z ~ N(μ,σ²) following a Gaussian distribution, reparameterization re-represents the random sampling process as... ,in Standard Gaussian noise , This represents element-wise multiplication. Through this transformation, the randomness of sampling is transferred to a noise variable independent of the model parameters. This allows the gradient to be effectively backpropagated through μ and σ, thus enabling end-to-end joint training of the encoder and decoder. In this specification, a reparameterization method is used to sample from a Gaussian distribution determined by the mean vector and log-variance vector to obtain a low-dimensional latent representation.

[0041] Example 1: like Figure 1 As shown, this embodiment provides a meteorological data compression method based on a variational autoencoder, including the following steps: Step S1: Obtain the meteorological data to be compressed: Specifically, the meteorological data to be compressed is acquired. The meteorological data includes reanalysis data, numerical weather prediction model output data, or satellite observation data, and the data has at least three dimensions: the number of meteorological variable channels C, the number of latitude grid points H, and the number of longitude grid points W.

[0042] Preferably, this embodiment uses a publicly available weather dataset as the compression target. Seventy key meteorological variables are selected, including: geopotential height (z), temperature (t), zonal wind (u), meridional wind (v), and specific humidity (q) at 13 pressure levels (1000hPa, 925hPa, 850hPa, 700hPa, 600hPa, 500hPa, 400hPa, 300hPa, 250hPa, 200hPa, 150hPa, 100hPa, 50hPa), as well as surface variables: 2-meter temperature (t). 2m ), 10-meter zonal wind (u 10 ), 10-meter meridional wind (v 10 The data includes sea level pressure (msl). The spatial resolution is 0.25°, corresponding to 721×1440 grid points (latitude×longitude), the temporal resolution is 6 hours, and the time span is from 1979 to 2023. The summary of input variables is shown in Table 1. Table 1:

[0043] Step S2: Preprocess the meteorological data to be compressed: Specifically, preprocessing includes data standardization, latitude and longitude location encoding, and block processing.

[0044] The data standardization includes: For each meteorological variable, calculate the mean and standard deviation over time, and standardize the data to a distribution with a mean of 0 and a standard deviation of 1. The standardization formula is as follows: , In the formula, and These are the global mean and standard deviation of the variable on the training set, respectively. Standardization unifies the values ​​of each meteorological variable to a distribution range with a mean of 0 and a standard deviation of 1, thus improving the stability and convergence speed of model training.

[0045] The latitude and longitude location code includes: Based on the number of latitude and longitude grid points in the meteorological data to be compressed, each latitude grid point value and each longitude grid point value are encoded as vectors in the form of sine and cosine functions, respectively, as location encoding vectors; the location encoding vectors are used as additional channels and concatenated with the standardized meteorological data in the channel dimension to obtain coded meteorological data with fused location information.

[0046] The specific method of position encoding is as follows: , , In the formula, and The first i The first latitudinal grid point and the second j The coordinates of a grid point of longitude. For dimensional indexing, d For encoding dimensions, in this embodiment d The preferred value is 64.

[0047] The block processing includes: Before inputting the encoded meteorological data into the encoder, a block processing step is also included: the encoded meteorological data is divided into multiple non-overlapping image blocks according to the two-dimensional spatial structure composed of latitude grid points and longitude grid points. Each image block has the same size, and each image block is flattened and projected onto a fixed dimension through a linear embedding layer.

[0048] Preferably, in this embodiment, the 721×1440 meteorological field is divided into 45×90 non-overlapping image blocks, each block being 16×16 in size, and each block is flattened and projected onto 256 dimensions through a linear embedding layer.

[0049] Step S3: Input the preprocessed meteorological data into the encoder of the trained variational autoencoder model. The encoder extracts multi-scale features through alternately stacked residual convolutional units and sliding window self-attention units, and outputs low-dimensional latent representations.

[0050] The encoder structure includes: Input layer: Receives meteorological data with dimensions of 70×721×1440, which is then divided into blocks and linearly embedded and converted into feature maps with dimensions of 256×45×90.

[0051] The first hybrid compression module contains a residual convolutional unit and a sliding window self-attention unit. The residual convolutional unit consists of two convolutional layers (3×3 kernel, stride 2, padding 1), halving the feature map size to 23×45 and doubling the number of channels to 512. The sliding window self-attention unit uses an 8×8 window and stacks two attention blocks.

[0052] The second hybrid compression module has a similar structure to the first module, but halves the feature map size to 12×23 and doubles the number of channels to 1024.

[0053] The third hybrid compression module: halves the feature map size to 6×12 and doubles the number of channels to 2048.

[0054] The fourth hybrid compression module: halves the feature map size to 3×6 and doubles the number of channels to 4096.

[0055] The latent parameter output layer consists of two parallel fully connected layers that output the mean vector μ and the log-variance vector logσ², respectively. The latent space dimension is L=256.

[0056] The specific structure of each residual convolutional unit is as follows: a first convolutional layer (3×3 kernel, stride 2, padding 1), a first batch of normalization layers, a first LeakyReLU activation function, a second convolutional layer (3×3 kernel, stride 1, padding 1), a second batch of normalization layers, a second LeakyReLU activation function, and a skip connection is used to adjust the dimensions of the input by a 1×1 convolution (stride 2) before adding it to the output.

[0057] The sliding window self-attention unit adopts a cyclic shifting window mechanism, with a fixed window size of 8×8, and introduces a learnable relative position bias table. The attention calculation method is as follows: , In the formula, Q , K , V These are query, key, and value matrices, respectively. Let be the dimension of the key vector. B This is a learnable relative position offset.

[0058] The low-dimensional latent representation is obtained by sampling from the Gaussian distribution determined by the mean vector and the log-variance vector using a reparameterization technique: , In the formula, This represents element-wise multiplication. It is standard Gaussian noise.

[0059] The decoder structure is symmetrical to the encoder, including: Latent Input Layer: Maps the latent representation z back to a 3×6×4096 feature map through a fully connected layer.

[0060] The first hybrid decompression module contains transposed convolutional units and sliding window self-attention units, upsampling the feature map size to 6×12 and halving the number of channels to 2048.

[0061] Second hybrid decompression module: Upsampled to 12×23, 1024 channels.

[0062] The third hybrid decompression module: upsampled to 23×45, with 512 channels.

[0063] Fourth hybrid decompression module: Upsampled to 45×90, 256 channels.

[0064] Output convolutional layer: The number of channels is projected to 69 through 1×1 convolution, and the resolution is restored to 721×1440 through pixel rearrangement, outputting the reconstructed meteorological data.

[0065] Step S4: Quantize the low-dimensional latent representation to generate a compressed bitstream.

[0066] Specifically, the low-dimensional latent representation z obtained from sampling is uniformly quantized: , In the formula, Use the scaling factor to generate a compressed bitstream.

[0067] Step S5: Input the compressed bitstream into the decoder of the trained variational autoencoder model to reconstruct the reconstructed meteorological data corresponding to the meteorological data to be compressed.

[0068] Specifically, the compressed bitstream is dequantized and input into the decoder. The decoder is then upsampled layer by layer according to the decoder structure described above, and finally the reconstructed meteorological data with the same resolution as the original is output.

[0069] This embodiment also includes a step of training the variational autoencoder model. Specifically: Data from 1979 to 2019 was used as the training set, data from 2020 to 2021 was used as the validation set, and data from 2022 to 2023 was used as the test set.

[0070] The meteorological data in the training set is preprocessed (including standardization, latitude and longitude location encoding, and block processing). The encoded location information is then concatenated with the data channels of the meteorological data as an additional channel to obtain the model input data. A variational autoencoder model is constructed, which includes an encoder and a decoder. The model input data is input into the variational autoencoder model, and the encoder and decoder are jointly trained end-to-end using a variational lower bound loss function to obtain the trained variational autoencoder model.

[0071] The loss function uses the variational lower bound loss function, specifically expressed as: , in, For the first i The original meteorological data of each sample, To reconstruct the data, For the sample size, The KL divergence weighting coefficient (in this embodiment) =0.001), 256 is the potential spatial dimension (i.e., L). and The first j The mean and variance of the dimension.

[0072] Training employs a latitude-weighted L1 loss to compensate for the impact of differences in grid area at different latitudes. The weighting coefficients are defined as follows: , In the formula, For the first i Latitude values ​​of each latitude grid point H This represents the total number of grid points along the latitudinal direction. The denominator of this weighting coefficient is the sum of the cosine values ​​of all latitudes, and the numerator is the cosine value of the current latitude grid point, such that the sum of the weights of all latitude grid points is 1.

[0073] The latitude-weighted L1 loss is: , In the formula, N The number of training samples. H The number of grid points in the latitudinal direction. W This represents the number of grid points along the longitude direction. This represents the weighting coefficient corresponding to the h-th latitude grid point; For the first i The sample is located at latitude h and longitude h. w The original meteorological data value at the location; This corresponds to the reconstructed meteorological data value.

[0074] By introducing latitudinal weights This invention assigns different weights to reconstruction errors at different latitudes: lower weights to high-latitude regions (smaller cosΦ values) and higher weights to equatorial regions (larger cosΦ values), thereby compensating for the uneven sampling caused by the small polar grid area and the large equatorial grid area, and ensuring that the reconstructed data maintains physical consistency globally.

[0075] The model training employs a single-stage end-to-end training strategy, using the AdamW optimizer, with an initial learning rate set to 2×10⁻⁶. -4 The weight decay is set to 10. -5 We use a cosine annealing learning rate scheduling strategy and set the batch size to 16.

[0076] To further verify the model's effectiveness, this embodiment uses global latitude-weighted RMSE (root mean square error) and regional RMSE (root mean square error) as performance evaluation metrics. Global latitude-weighted RMSE is defined as: , In the formula, H and W These represent the number of grid points for latitude and longitude, respectively. For channel c In latitude i ,longitude j Reconstruction values ​​on For channel c In latitude i ,longitude j The truth value of , latitude Weight at each location.

[0077] Through the above steps, this embodiment achieves efficient compression and high-fidelity reconstruction of meteorological data.

[0078] Example 2: like Figure 2 As shown, this embodiment provides a meteorological data compression system based on a variational autoencoder, including: The data acquisition module is used to acquire the meteorological data to be compressed; The location encoding module is used to encode the meteorological data to be compressed using latitude and longitude to obtain the encoded meteorological data. A variational autoencoder model includes an encoder and a decoder; the encoder is used to compress the encoded meteorological data into a low-dimensional latent representation, which extracts multi-scale features through alternately stacked residual convolutional units and sliding window self-attention units, and outputs the low-dimensional latent representation. The quantization module is used to quantize the low-dimensional latent representation and generate a compressed bitstream; The decoder is used to reconstruct the compressed bitstream to obtain reconstructed meteorological data corresponding to the meteorological data to be compressed.

[0079] Example 3: like Figure 3 As shown, this embodiment provides an electronic device, which may include: at least one processor, at least one network interface, a user interface, a memory, and at least one communication bus.

[0080] The communication bus can be used to enable communication between the various components mentioned above.

[0081] The user interface may include buttons, and optional user interfaces may also include standard wired interfaces and wireless interfaces.

[0082] The network interface may include, but is not limited to, Bluetooth modules, NFC modules, Wi-Fi modules, etc.

[0083] The processor may include one or more processing cores. It connects various parts of the electronic device via various interfaces and lines, executing instructions, programs, code sets, or instruction sets stored in memory, and accessing data stored in memory to perform various functions and process data. Optionally, the processor can be implemented using at least one hardware form of DSP, FPGA, or PLA. The processor may integrate one or more of the following: CPU, GPU, and modem. The CPU primarily handles the operating system, user interface, and applications; the GPU is responsible for rendering and drawing the content required for display; and the modem handles wireless communication. It is understood that the modem may also be implemented as a separate chip without being integrated into the processor.

[0084] The memory may include RAM or ROM. Optionally, the memory may include a non-transitory computer-readable medium. The memory can be used to store instructions, programs, code, code sets, or instruction sets. The memory may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for at least one function (such as touch function, sound playback function, image playback function, etc.), instructions for implementing the above-described method embodiments, etc.; the data storage area may store data involved in the above-described method embodiments, etc. Optionally, the memory may also be at least one storage device located remotely from the aforementioned processor. The memory, as a computer storage medium, may include an operating system, a network communication module, a user interface module, and a compression application program. The processor can be used to call the compression application program stored in the memory and execute the steps of the meteorological data compression method mentioned in the foregoing embodiments.

[0085] Example 4: This embodiment provides a computer-readable storage medium storing instructions that, when executed on a computer or processor, cause the computer or processor to perform the above-described instructions. Figure 1 One or more steps in the illustrated embodiment. If the constituent modules of the above-described electronic device are implemented as software functional units and sold or used as independent products, they can be stored in the computer-readable storage medium.

[0086] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this specification are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in or transmitted through a computer-readable storage medium. The computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, Digital Subscriber Line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium accessible to a computer or a data storage device such as a server or data center that integrates one or more available media. The available media may be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., Digital Versatile Discs (DVDs)), or semiconductor media (e.g., Solid State Disks (SSDs)).

[0087] Those skilled in the art will understand that all or part of the processes in the method of Embodiment 1 described above can be implemented by a computer program instructing related hardware. This program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the methods described above. The aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks. Unless otherwise specified, the technical features of this embodiment and the implementation scheme can be combined arbitrarily.

[0088] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that the present invention is not limited to the described order of actions, as some steps can be performed in other orders or simultaneously according to the present invention. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to the present invention.

[0089] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0090] The foregoing description is merely an exemplary embodiment of the present invention and should not be construed as limiting the scope of the invention. Any equivalent changes and modifications made in accordance with the teachings of this invention are still within the scope of this invention. Those skilled in the art will readily conceive of embodiments of the invention upon considering the specification and practicing the disclosure herein. This invention is intended to cover any variations, uses, or adaptations of the invention that follow the general principles of the invention and include knowledge or means in the art not described herein. The specification and embodiments are to be considered exemplary only, and the scope and spirit of the invention are defined by the claims.

Claims

1. A meteorological data compression method based on a variational autoencoder, characterized in that, Including the following steps: Acquire meteorological data to be compressed; The meteorological data to be compressed is encoded by latitude and longitude to obtain encoded meteorological data; The encoded meteorological data is input into the encoder of the trained variational autoencoder model. The encoder extracts multi-scale features through alternately stacked residual convolutional units and sliding window self-attention units, and outputs low-dimensional latent representations. The low-dimensional latent representation is quantized to generate a compressed bitstream; The compressed bitstream is input into the decoder of the trained variational autoencoder model to reconstruct the reconstructed meteorological data corresponding to the meteorological data to be compressed.

2. The meteorological data compression method according to claim 1, characterized in that: The variational autoencoder model is trained using a variational lower bound loss function; The variational lower bound loss function includes a latitude-weighted L1 loss term and a KL divergence regularization term; The weighting coefficient of the latitude-weighted L1 loss is defined as the ratio of the cosine value of each latitude to the sum of the cosine values ​​of all latitudes, in order to compensate for the differences in grid area at different latitudes.

3. The meteorological data compression method according to claim 2, characterized in that, It also includes the step of training the variational autoencoder model: Meteorological data of a preset duration is used as a training set, and the meteorological data contains multiple meteorological variables; The meteorological data in the training set is encoded with latitude and longitude, and the encoded location information is used as an additional channel and concatenated with the data channel of the meteorological data to obtain the model input data. Construct an initial model of a variational autoencoder, which includes an encoder and a decoder; The model input data is input into the initial model of the variational autoencoder, and the encoder and decoder are jointly trained end-to-end using the variational lower bound loss function to obtain the trained variational autoencoder model.

4. The meteorological data compression method according to claim 1, characterized in that: Based on the number of latitude and longitude grid points of the meteorological data to be compressed, each latitude grid point value and each longitude grid point value are encoded into vectors in sine and cosine function forms, respectively, as location encoding vectors; The location encoding vector is used as an additional channel and concatenated with the meteorological data to be compressed along the channel dimension to obtain the encoded meteorological data.

5. The meteorological data compression method according to claim 4, characterized in that, Before inputting the encoded meteorological data into the encoder, the following steps are also included: The encoded meteorological data is divided into multiple non-overlapping image blocks according to the two-dimensional spatial structure composed of latitude grid points and longitude grid points. Each image block has the same size, and each image block is flattened and projected onto a fixed dimension through a linear embedding layer.

6. The meteorological data compression method according to claim 1, characterized in that: The encoder includes a mean branch and a variance branch, which output the mean vector and log-variance vector of the low-dimensional latent representation, respectively. The low-dimensional latent representation is obtained by sampling from the Gaussian distribution determined by the mean vector and the log-variance vector using a reparameterization method.

7. The meteorological data compression method according to claim 1, characterized in that: The residual convolutional units employ skip connections; The sliding window self-attention unit adopts a cyclic shift window mechanism and introduces a learnable relative position bias table.

8. A meteorological data compression system based on a variational autoencoder, characterized in that, Used to implement the meteorological data compression method as described in any one of claims 1 to 7.

9. An electronic device, the electronic device comprising a memory, a processor, and a computer program, characterized in that, When the computer program is executed by the processor, it implements the meteorological data compression method as described in any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the meteorological data compression method as described in any one of claims 1 to 7.

Images