A meteorological prediction data compression storage optimization method and device

Abstract

The application discloses a meteorological prediction data compression storage optimization method, and belongs to the technical field of meteorological data processing. The method comprises the following steps: obtaining preprocessed meteorological prediction data; performing three-dimensional wavelet decomposition on the preprocessed meteorological prediction data by using an orthogonal wavelet basis function to obtain a tensor; performing high-order tensor decomposition on the tensor to obtain a core tensor and a plurality of modal feature matrices; performing regularized constraint dynamic adjustment and sparse processing on the rank of the core tensor, and performing dimension reduction processing on the plurality of modal feature matrices to obtain a core tensor with reduced rank and a plurality of modal feature matrices with reduced storage size, and then performing entropy coding and quantization processing on the core tensor and the plurality of modal feature matrices to obtain compressed representations of the core tensor and the plurality of modal feature matrices after the entropy coding and quantization processing; and performing dynamic calibration on the storage precision and information loss of the compressed representations of the core tensor and the plurality of modal feature matrices after the entropy coding and quantization processing by using variational inference to complete compression storage optimization of the meteorological prediction data.

Description

Technical Field

[0001] This invention relates to the field of meteorological data processing technology, and in particular to a method and apparatus for optimizing the compression and storage of meteorological forecast data. Background Technology

[0002] Meteorological forecasting data is a crucial foundation for weather forecasting, climate research, and natural disaster early warning, characterized by its high dimensionality, high resolution, and large scale. With the continuous development of numerical weather prediction models and observation techniques, the generation speed and scale of meteorological data are growing exponentially, placing enormous pressure on storage and transmission. How to efficiently compress and store meteorological data while ensuring its accuracy and physical consistency has become a pressing technical challenge.

[0003] Currently, technologies for compressing and storing meteorological data mainly focus on dimensionality reduction methods based on feature extraction and compression algorithms based on information entropy. However, existing methods have certain limitations when dealing with the spatiotemporal and multi-scale characteristics of meteorological data. Specifically, traditional dimensionality reduction methods are prone to losing the physical characteristics of the data during compression, making it difficult for the restored data to meet actual forecasting needs. Meanwhile, common compression algorithms, such as entropy coding-based processing techniques, while reducing data storage, still have room for improvement in compression efficiency and restoration accuracy when dealing with the high dimensionality and complex spatiotemporal correlations of meteorological data. Furthermore, for data with significant local characteristics, such as extreme weather events, these methods often struggle to achieve a dynamic balance between compression and restoration accuracy.

[0004] In practical applications, meteorological data needs to possess the capabilities of large-scale storage, rapid transmission, and high-precision restoration, especially in real-time weather forecasting and disaster monitoring scenarios where the timeliness and accuracy of the data are crucial. However, existing technologies have failed to fully leverage the multi-scale characteristics and high-dimensional tensor structure of meteorological data, and have failed to achieve global optimization of compressed storage and efficient restoration. Summary of the Invention

[0005] The purpose of this invention is to provide a method and apparatus for optimizing the compression and storage of meteorological forecast data. This method uses an entropy coding optimization algorithm to compress the numerical values ​​of frequently occurring core tensors and multiple modal feature matrices, and then stores the compressed core tensors and multiple modal feature matrices in a designated storage medium, thus completing the compression and storage. This invention achieves efficient compression and storage of large-scale meteorological forecast data while ensuring the accuracy of meteorological data restoration and physical consistency. This invention is achieved through the following technical solutions.

[0006] In a first aspect, the present invention provides a method for optimizing the compression and storage of meteorological forecast data, comprising: Acquire meteorological forecast data, construct the meteorological forecast data into a three-dimensional tensor form, and preprocess the meteorological forecast data transformed into a three-dimensional tensor form to obtain preprocessed meteorological forecast data. Three-dimensional wavelet decomposition of preprocessed meteorological forecast data was performed using orthogonal wavelet basis functions to obtain tensors; The tensor is decomposed into a higher-order tensor to obtain the core tensor and multiple modal feature matrices; The rank of the core tensor is dynamically adjusted by regularization constraints and sparsified. The dimensionality of multiple modal feature matrices is reduced to obtain a core tensor with reduced rank and multiple modal feature matrices with reduced storage size. Entropy encoding and quantization are performed on the core tensor with reduced rank and the multiple modal feature matrices with reduced storage size to obtain a compressed representation of the core tensor and multiple modal feature matrices after entropy encoding and quantization. Variational inference is used to dynamically calibrate the storage accuracy and information loss of the compressed representation of the core tensor and multiple modal feature matrices after entropy encoding and quantization, thereby optimizing the compressed storage of meteorological forecast data.

[0007] In practical applications, traditional dimensionality reduction methods often lose the physical properties of data during compression, making it difficult to restore data that meets actual forecasting needs. The entropy optimization algorithm proposed in this invention, combined with a variational inference-based dynamic calibration method, maintains high-precision restoration while compressing data. This effectively reduces the storage space required for meteorological data, significantly decreasing storage and transmission costs. Particularly in distributed storage and long-distance transmission scenarios, this method significantly improves the economy and practicality of data processing. The method is applicable to both large-scale gridded data generated by numerical weather prediction models and spatiotemporal series data from actual observation stations. Its multimodal and multi-scale processing capabilities make it widely applicable in weather analysis, climate modeling, and disaster early warning.

[0008] Optionally, the orthogonal wavelet basis functions include Haar wavelet basis functions or Daubechies wavelet basis functions; The preprocessed meteorological forecast data is decomposed into three-dimensional wavelet components using orthogonal wavelet basis functions, which also yields low-frequency and high-frequency components. The low-frequency components are used to describe the global features of the meteorological forecast data, while the high-frequency components are used to describe the local features of the meteorological forecast data.

[0009] The results of three-dimensional wavelet decomposition preserve the multi-scale characteristics of the data.

[0010] Optionally, thresholding the high-frequency components includes setting high-frequency component coefficients less than a pre-set absolute value threshold to zero, and retaining high-frequency component coefficients greater than or equal to the threshold.

[0011] By performing saliency screening on high-frequency components, coefficients that make significant contributions to restoration and feature extraction are retained, while noise and redundant information are removed, thereby further reducing data storage and optimizing compression efficiency.

[0012] Optionally, the step of performing high-order tensor decomposition on the tensor to obtain the core tensor and multiple modal feature matrices includes: Flatten the tensor along each modal direction to obtain the flattened matrix of each mode; Singular value decomposition is performed on the flattened matrix of each mode to obtain multiple modal feature matrices; The core tensor is obtained by performing tensor-matrix multiplication on the tensor and multiple modal feature matrices.

[0013] By using high-order tensor decomposition, meteorological forecast data is mapped from multidimensional space to a combined representation of low-rank core tensors and modal feature matrices, thereby significantly reducing data storage requirements while preserving the main features of the data.

[0014] Optionally, the entropy encoding and quantization processing of the reduced-rank core tensor and the multiple modal feature matrices with reduced storage size includes: calculating the entropy value of the data value distribution of the reduced-rank core tensor and the multiple modal feature matrices with reduced storage size, and using the entropy value as a parameter of the entropy encoding optimization algorithm to guide data compression. This further reduces the storage space of meteorological forecast data.

[0015] Optionally, the entropy coding optimization algorithm includes the following steps: The values ​​of each element in the reduced-rank core tensor and the multiple modal feature matrices with reduced storage size are statistically analyzed, the frequency of each element value is calculated, and the probability distribution of the reduced-rank core tensor and the multiple modal feature matrices with reduced storage size is generated. Based on the probability distribution of the reduced-rank core tensor and the multiple modal feature matrices with reduced storage size, an entropy coding algorithm is used to compress the reduced-rank core tensor and the multiple modal feature matrices with reduced storage size. The parameters of the entropy coding algorithm are dynamically adjusted to prioritize the compression of frequently occurring element values ​​in the reduced-rank core tensor and the multiple modal feature matrices with reduced storage size, thereby maximizing data compression efficiency. The compressed reduced-rank core tensor and the multiple modal feature matrices with reduced storage size are then stored in a specified storage medium to complete the compressed storage.

[0016] Optionally, the dynamic calibration of the storage accuracy and information loss of the compressed representation of the core tensor and multiple modal feature matrices after entropy encoding and quantization using variational inference includes: constructing a posterior distribution and an approximate distribution model of the core tensor after entropy encoding and quantization, and minimizing the Kullback-Leibler divergence between the posterior distribution and the approximate distribution of the core tensor after entropy encoding and quantization, thereby dynamically adjusting the storage accuracy and information loss of the core tensor after entropy encoding and quantization to ensure the reliability of data restoration.

[0017] By analyzing and optimizing the statistical characteristics of data using entropy coding technology, redundant information can be further removed, thereby significantly improving data storage efficiency. By constructing posterior and approximate distribution models of the core tensor, information theory optimization techniques are used to achieve a dynamic trade-off between storage efficiency and restoration accuracy, thus ensuring the high efficiency and reliability of compressed data.

[0018] Optionally, the difference in the posterior distribution is calibrated by optimizing the parameters of the approximate distribution to minimize the Kullback-Leibler divergence between the approximate and posterior distributions, ensuring that the accuracy of data compression is within a preset threshold range.

[0019] In a second aspect, the present invention provides a device for optimizing the compression and storage of meteorological forecast data, comprising: The data preprocessing module is used to acquire meteorological forecast data, convert the meteorological forecast data into a three-dimensional tensor form, and preprocess the meteorological forecast data converted into a three-dimensional tensor form to obtain preprocessed meteorological forecast data. The wavelet decomposition module is used to perform three-dimensional wavelet decomposition on the preprocessed meteorological forecast data using orthogonal wavelet basis functions to obtain tensors. The tensor decomposition module is used to perform high-order tensor decomposition on tensors to obtain the core tensor and multiple modal feature matrices. The structured low-rank compression module is used to dynamically adjust the rank of the core tensor by regularization constraints and perform sparsity processing, and to reduce the dimensionality of multiple modal feature matrices to obtain a core tensor with reduced rank and multiple modal feature matrices with reduced storage size. The entropy coding module is used to perform entropy coding and quantization on the core tensor with reduced rank and multiple modal feature matrices with reduced storage size, so as to obtain a compressed representation of the core tensor and multiple modal feature matrices after entropy coding and quantization. The dynamic calibration and storage module is used to dynamically calibrate the storage accuracy and information loss of the compressed representation of the core tensor and multiple modal feature matrices after entropy encoding and quantization using variational inference, thereby optimizing the compressed storage of meteorological forecast data. Beneficial effects

[0020] (1) This invention utilizes a combination of three-dimensional wavelet decomposition and higher-order tensor decomposition to perform multi-level compression optimization of meteorological forecast data. Three-dimensional wavelet decomposition extracts the low-frequency and high-frequency characteristics of the data, while higher-order tensor decomposition further performs rank constraint and dimensionality reduction on the high-dimensional data, effectively reducing redundant storage requirements and significantly improving the data compression rate. By leveraging the multi-scale processing mechanism of three-dimensional wavelet decomposition and the parallelism of tensor decomposition, rapid compression and restoration of large-scale meteorological data are achieved.

[0021] (2) This invention achieves high-precision restoration while compressing data by optimizing the entropy of the core tensor and multiple modal feature matrices, combined with a variational inference-based dynamic calibration method. Even under high compression ratios, the physical consistency and spatiotemporal characteristics of the restored data are fully guaranteed, meeting the practical application requirements of meteorological forecasting. Entropy optimization further reduces the storage and processing overhead of the core tensor.

[0022] (3) This invention introduces variational inference technology, which dynamically calibrates different data scenarios, such as high-precision prediction or fast storage, by constructing a posterior distribution model and information loss constraints. This adaptability ensures that the compression scheme can be flexibly applied to various demand scenarios, such as extreme weather prediction or long-term meteorological archiving. Attached Figure Description

[0023] Figure 1 The diagram shown is a flowchart of the meteorological forecast data compression and storage optimization method of the present invention. Figure 2 The diagram shown is a schematic of the structure of the device for optimizing the compression and storage of meteorological forecast data according to the present invention. In the diagram: 10 - Data preprocessing module, 20 - Wavelet decomposition module, 30 - Tensor decomposition module, 40 - Structured low-rank compression module, 50 - Entropy coding module, 60 - Dynamic calibration and storage module. Detailed Implementation

[0024] The following description, in conjunction with the accompanying drawings and specific embodiments, provides further details. In this description, it should be understood that the terms "first," "second," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Therefore, a feature defined with "first," "second," etc., may explicitly or implicitly include one or more of that feature.

[0025] Example 1 This embodiment introduces a method for optimizing the compression and storage of meteorological forecast data, such as... Figure 1 As shown, it includes the following: Acquire meteorological forecast data, construct the meteorological forecast data into a three-dimensional tensor form, and preprocess the meteorological forecast data transformed into a three-dimensional tensor form to obtain preprocessed meteorological forecast data. Three-dimensional wavelet decomposition of preprocessed meteorological forecast data was performed using orthogonal wavelet basis functions to obtain tensors; The tensor is decomposed into a higher-order tensor to obtain the core tensor and multiple modal feature matrices; The rank of the core tensor is dynamically adjusted and sparsified by regularization constraints, and the dimensionality of multiple modal feature matrices is reduced to obtain a core tensor with reduced rank and multiple modal feature matrices with reduced storage size. Entropy encoding and quantization are performed on the core tensor with reduced rank and the multiple modal feature matrices with reduced storage size to obtain a compressed representation of the core tensor and multiple modal feature matrices after entropy encoding and quantization. Variational inference is used to dynamically calibrate the storage accuracy and information loss of the compressed representation of the core tensor and multiple modal feature matrices after entropy encoding and quantization, thereby optimizing the compressed storage of meteorological forecast data.

[0026] In practical applications, traditional dimensionality reduction methods often lose the physical properties of data during compression, making it difficult to restore data that meets actual forecasting needs. The entropy optimization algorithm proposed in this invention, combined with a variational inference-based dynamic calibration method, maintains high-precision restoration while compressing data. This effectively reduces the storage space required for meteorological data, significantly decreasing storage and transmission costs. Particularly in distributed storage and long-distance transmission scenarios, this method significantly improves the economy and practicality of data processing. The method is applicable to both large-scale gridded data generated by numerical weather prediction models and spatiotemporal series data from actual observation stations. Its multimodal and multi-scale processing capabilities make it widely applicable in weather analysis, climate modeling, and disaster early warning.

[0027] Example 2 Based on Example 1, this example describes the specific implementation process of a method for optimizing the compression and storage of meteorological forecast data, such as... Figure 1 As shown, it specifically includes the following: I. Data Preprocessing First, meteorological forecast data is acquired, and then the meteorological forecast data is constructed into a three-dimensional tensor form. The meteorological forecast data converted into a three-dimensional tensor form is preprocessed to obtain preprocessed meteorological forecast data. Meteorological forecast data typically originates from numerical weather prediction models or observation stations. Its data structure includes a time dimension (e.g., hourly or daily data), a spatial dimension (e.g., latitude and longitude grids), and physical variable dimensions (e.g., temperature, humidity, and wind speed). Due to different sources or measurement errors, these data may exhibit differences in magnitude, irregularities, missing data, or inconsistent formats. Therefore, after being constructed into a three-dimensional tensor form, preprocessing is required.

[0028] As an alternative, meteorological forecast data can first be reorganized according to time, space, and physical variables; specifically, the input data can be rearranged into a three-dimensional matrix form.

[0029] For example, in some embodiments, hourly recorded temperature, humidity, and wind speed data can each correspond to different layers of physical variables in a three-dimensional tensor. In this case, the input data can be represented as... Where x is the input data, I is the time step, such as 24 hours, J is the number of spatial grid points, such as a 10×10 latitude and longitude grid, and K is the number of physical variables, such as temperature, humidity, and wind speed, a total of 3 variables.

[0030] To eliminate the difference in magnitude between different physical variables, each physical variable is standardized in this embodiment.

[0031] Through standardization, the data distribution of each variable is adjusted to a form with a mean of zero and a standard deviation of one, which facilitates the equal processing of each physical variable by the subsequent decomposition algorithm.

[0032] As one implementation method, if missing values ​​exist in the data, this embodiment uses interpolation methods to fill them in. Specifically, algorithms such as linear interpolation, spline interpolation, or nearest neighbor interpolation can be used to fill in the missing points. Taking linear interpolation as an example, when data at a certain time step is missing, the value of the missing point can be linearly calculated using the data values ​​at adjacent time points.

[0033] It should be noted that the choice of interpolation method can be adjusted according to the specific characteristics of the data. For example, for extreme weather data with strong abrupt changes, spline interpolation can be chosen to better capture the changing trend.

[0034] To ensure data compatibility with subsequent wavelet decomposition and tensor decomposition operations, the acquired meteorological forecast data in this embodiment is uniformly organized into a three-dimensional tensor format. This formatting operation includes adjusting the data storage order, filling in missing data dimensions, and assigning uniform identifiers to physical variables. For example, temperature, humidity, and wind speed can correspond to the first, second, and third layers of tensor data, respectively.

[0035] In summary, the standardized preprocessing scheme for meteorological forecast data provided in this embodiment includes data standardization, interpolation, and formatting operations, and optionally noise filtering and smoothing. These operations provide high-quality input data for subsequent three-dimensional wavelet decomposition, higher-order tensor decomposition, and compression optimization, ensuring the processing effectiveness and algorithm performance of subsequent steps.

[0036] II. Three-dimensional wavelet decomposition and thresholding The orthogonal wavelet basis functions include Haar wavelet basis functions or Daubechies wavelet basis functions; The preprocessed meteorological forecast data is decomposed into three-dimensional wavelet components using orthogonal wavelet basis functions, which also yields low-frequency and high-frequency components. The low-frequency components are used to describe the global features of the meteorological forecast data, while the high-frequency components are used to describe the local features of the meteorological forecast data.

[0037] Thresholding the high-frequency components includes setting high-frequency component coefficients below a pre-set absolute value threshold to zero, and retaining high-frequency component coefficients that are greater than or equal to the threshold.

[0038] In this embodiment, the high-frequency components generated by three-dimensional wavelet decomposition contain a large number of detailed features, which may be mixed with measurement noise, random fluctuations, and redundant information that contributes little to data reconstruction. To reduce data storage redundancy, a threshold-based filtering method is designed to retain only the significant coefficients in the high-frequency components. The threshold can be determined as follows: based on the energy distribution of the high-frequency components, the top 90% or 95% of the total energy is selected as the threshold retention range. High-frequency component coefficients below the threshold are set to zero, while high-frequency component coefficients greater than or equal to the threshold are retained.

[0039] The embodiment also incorporates a multi-level dynamic adjustment mechanism. For example, after the initial screening, the threshold can be adjusted based on the total number of remaining coefficients: if the total number of filtered coefficients is less than a preset lower limit, such as 10% of the total number of coefficients, the threshold is appropriately lowered; if the total number of filtered coefficients is higher than a preset upper limit, such as 50% of the total number of coefficients, the threshold is appropriately increased. This dynamic adjustment mechanism ensures that the filtered high-frequency components have sufficient information, meeting compression requirements without affecting the accuracy of data restoration.

[0040] The screening method in this embodiment is not only applicable to single-layer high-frequency components, but can also be extended to the overall screening of all high-frequency components. In this extended implementation, the coefficient distributions of all high-frequency components can be jointly analyzed to determine a global unified threshold. This approach is suitable for situations where the energy distribution differences between high-frequency components are small, further simplifying the complexity of threshold setting.

[0041] It's important to understand that the high-frequency component filtering process is a further optimization of the three-dimensional wavelet decomposition results. Its core objective is to minimize the storage requirements for redundant information while retaining sufficient significance coefficients to support data reconstruction and analysis. Through this process, the storage size of meteorological forecast data can be effectively reduced, providing high-quality input data for subsequent higher-order tensor decomposition and entropy optimization.

[0042] III. Higher-order tensor decomposition Three-dimensional wavelet decomposition is performed on preprocessed meteorological forecast data using orthogonal wavelet basis functions to obtain tensors. Higher-order tensor decomposition is then performed on these tensors to obtain a core tensor and multiple modal feature matrices. Higher-order tensor decomposition is a dimensionality reduction technique suitable for high-dimensional data. Its goal is to represent the original tensor as a product of a core tensor and several modal feature matrices. This representation can capture the structural characteristics of data across different modalities, such as time, space, and physical variables, and achieve data compression through rank constraints. The higher-order tensor decomposition of the tensor to obtain the core tensor and multiple modal feature matrices specifically includes the following: Flatten the tensor along each modal direction to obtain the flattened matrix of each mode; Singular value decomposition is performed on the flattened matrix of each mode to obtain multiple modal feature matrices; The core tensor is obtained by performing tensor-matrix multiplication on the tensor and multiple modal feature matrices.

[0043] Higher-order tensor decomposition can further compress the data after three-dimensional wavelet decomposition while preserving its main structural characteristics, providing a compact and efficient input for subsequent entropy optimization and dynamic calibration.

[0044] IV. Entropy Coding The entropy encoding and quantization processing of the reduced-rank core tensor and the multiple modal feature matrices with reduced storage size includes: calculating the entropy value of the data value distribution of the reduced-rank core tensor and the multiple modal feature matrices with reduced storage size, and using the entropy value as a parameter of the entropy encoding optimization algorithm to guide data compression.

[0045] The entropy coding optimization algorithm includes the following steps: The values ​​of each element in the reduced-rank core tensor and the multiple modal feature matrices with reduced storage size are statistically analyzed, the frequency of each element value is calculated, and the probability distribution of the reduced-rank core tensor and the multiple modal feature matrices with reduced storage size is generated. Based on the probability distribution of the reduced-rank core tensor and the multiple modal feature matrices with reduced storage size, an entropy coding algorithm is used to compress the reduced-rank core tensor and the multiple modal feature matrices with reduced storage size. The parameters of the entropy coding algorithm are dynamically adjusted to prioritize the compression of the element values ​​in the high-frequency reduced-rank core tensor and the multiple modal feature matrices with reduced storage size. The compressed reduced-rank core tensor and the multiple modal feature matrices with reduced storage size are then stored in the specified storage medium to complete the compressed storage.

[0046] By following the steps above, the storage requirements for meteorological forecast data can be significantly reduced, providing a compact and efficient compressed representation for subsequent dynamic calibration and data restoration.

[0047] V. Dynamic Calibration and Data Storage The method of dynamically calibrating the storage accuracy and information loss of the compressed representation of the core tensor and multiple modal feature matrices after entropy encoding and quantization using variational inference includes: constructing posterior and approximate distribution models of the core tensor after entropy encoding and quantization, and minimizing the Kullback-Leibler divergence between the posterior and approximate distributions of the core tensor after entropy encoding and quantization, thereby dynamically adjusting the storage accuracy and information loss of the core tensor after entropy encoding and quantization to ensure the reliability of data restoration.

[0048] The difference in the posterior distribution is calibrated by optimizing the parameters of the approximate distribution to minimize the Kullback-Leibler divergence between the approximate and posterior distributions, ensuring that the data compression accuracy is within a preset threshold range. This completes the optimization of meteorological forecast data compression and storage.

[0049] The core advantage of variational inference lies in its dynamic adaptability. By calibrating the stored representation of the core tensor in real time, optimization strategies can be flexibly adjusted according to the storage requirements and restoration accuracy requirements of different scenarios. For example, for critical meteorological data requiring high-precision restoration, such as extreme weather-related information, a lower information loss threshold can be set to ensure that the restoration error is controlled within a smaller range.

[0050] The above methods can effectively optimize the storage accuracy of core tensors, providing technical support for the efficient compression and reliable restoration of meteorological forecast data.

[0051] This invention extracts multi-scale features from meteorological data through three-dimensional wavelet decomposition, achieves efficient dimensionality reduction by combining it with high-order tensor decomposition, further reduces storage redundancy through entropy optimization, and utilizes variational inference to dynamically calibrate information loss during compression. This invention not only significantly improves the compression efficiency of meteorological data but also maintains its physical consistency and prediction accuracy during data restoration, providing an efficient and reliable solution for the storage and transmission of large-scale meteorological data.

[0052] Example 3 This embodiment provides a device for optimizing the compression and storage of meteorological forecast data, such as... Figure 2 As shown, it includes the following: The data preprocessing module 10 is used to acquire meteorological forecast data, convert the meteorological forecast data into a three-dimensional tensor form, and preprocess the meteorological forecast data converted into a three-dimensional tensor form to obtain preprocessed meteorological forecast data. Wavelet decomposition module 20 is used to perform three-dimensional wavelet decomposition on the preprocessed meteorological forecast data using orthogonal wavelet basis functions to obtain tensors; Tensor decomposition module 30 is used to perform high-order tensor decomposition on tensors to obtain core tensors and multiple modal feature matrices. The structured low-rank compression module 40 is used to dynamically adjust the rank of the core tensor by regularization constraints and perform sparsity processing, and to reduce the dimensionality of multiple modal feature matrices to obtain a core tensor with reduced rank and multiple modal feature matrices with reduced storage size. The entropy coding module 50 is used to perform entropy coding and quantization on the core tensor with reduced rank and multiple modal feature matrices with reduced storage size, so as to obtain a compressed representation of the core tensor and multiple modal feature matrices after entropy coding and quantization. The dynamic calibration and storage module 60 is used to dynamically calibrate the storage accuracy and information loss of the compressed representation of the core tensor and multiple modal feature matrices after entropy encoding and quantization using variational inference, thereby optimizing the compressed storage of meteorological forecast data. The embodiments of the present invention have been described above with reference to the accompanying drawings. However, the present invention is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of the present invention without departing from the spirit and scope of the claims. All of these forms are within the protection scope of the present invention.

Claims

1. A method for optimizing the compression and storage of meteorological forecast data, characterized in that, include: Meteorological forecast data is acquired, constructed into a three-dimensional tensor, and preprocessed to obtain preprocessed meteorological forecast data. The meteorological forecast data is reorganized according to time, space, and physical variables. The input data is then rearranged into a three-dimensional matrix. Three-dimensional wavelet decomposition of preprocessed meteorological forecast data was performed using orthogonal wavelet basis functions to obtain tensors; The tensor is decomposed into a higher-order tensor to obtain the core tensor and multiple modal feature matrices; The rank of the core tensor is dynamically adjusted by regularization constraints and sparsified. The dimensionality of multiple modal feature matrices is reduced to obtain a core tensor with reduced rank and multiple modal feature matrices with reduced storage size. Entropy encoding and quantization are performed on the core tensor with reduced rank and the multiple modal feature matrices with reduced storage size to obtain a compressed representation of the core tensor and multiple modal feature matrices after entropy encoding and quantization. Variational inference is used to dynamically calibrate the storage accuracy and information loss of the compressed representation of the core tensor and multiple modal feature matrices after entropy encoding and quantization. By constructing posterior and approximate distribution models of the core tensor after entropy encoding and quantization, and minimizing the Kullback-Leibler divergence between the posterior and approximate distributions of the core tensor after entropy encoding and quantization, the storage accuracy and information loss of the core tensor after entropy encoding and quantization are dynamically adjusted to ensure the reliability of data restoration and complete the optimization of compressed storage of meteorological forecast data.

2. The method for optimizing the compression and storage of meteorological forecast data according to claim 1, characterized in that, The orthogonal wavelet basis functions include Haar wavelet basis functions or Daubechies wavelet basis functions; The preprocessed meteorological forecast data is decomposed into three-dimensional wavelet components using orthogonal wavelet basis functions, which also yields low-frequency and high-frequency components. The low-frequency components are used to describe the global features of the meteorological forecast data, while the high-frequency components are used to describe the local features of the meteorological forecast data.

3. The method for optimizing the compression and storage of meteorological forecast data according to claim 2, characterized in that, Thresholding the high-frequency components includes setting high-frequency component coefficients below a pre-set absolute value threshold to zero, and retaining high-frequency component coefficients that are greater than or equal to the threshold.

4. The method for optimizing the compression and storage of meteorological forecast data according to claim 1, characterized in that, The higher-order tensor decomposition of the tensor to obtain the core tensor and multiple modal feature matrices includes: Flatten the tensor along each modal direction to obtain the flattened matrix of each mode; Singular value decomposition is performed on the flattened matrix of each mode to obtain multiple modal feature matrices; The core tensor is obtained by performing tensor-matrix multiplication on the tensor and multiple modal feature matrices.

5. The method for optimizing the compression and storage of meteorological forecast data according to claim 1, characterized in that, The entropy encoding and quantization processing of the reduced-rank core tensor and the multiple modal feature matrices with reduced storage size includes: calculating the entropy value of the data value distribution of the reduced-rank core tensor and the multiple modal feature matrices with reduced storage size, and using the entropy value as a parameter of the entropy encoding optimization algorithm to guide data compression.

6. The method for optimizing the compression and storage of meteorological forecast data according to claim 5, characterized in that, The entropy coding optimization algorithm includes the following steps: The values ​​of each element in the reduced-rank core tensor and the multiple modal feature matrices with reduced storage size are statistically analyzed, the frequency of each element value is calculated, and the probability distribution of the reduced-rank core tensor and the multiple modal feature matrices with reduced storage size is generated. Based on the probability distribution of the reduced-rank core tensor and the multiple modal feature matrices with reduced storage size, an entropy coding algorithm is used to compress the reduced-rank core tensor and the multiple modal feature matrices with reduced storage size. The parameters of the entropy coding algorithm are dynamically adjusted to prioritize the compression of the element values ​​in the high-frequency reduced-rank core tensor and the multiple modal feature matrices with reduced storage size. The compressed reduced-rank core tensor and the multiple modal feature matrices with reduced storage size are then stored in the specified storage medium to complete the compressed storage.

7. The method for optimizing the compression and storage of meteorological forecast data according to claim 1, characterized in that, The difference in the posterior distribution is calibrated by optimizing the parameters of the approximate distribution to minimize the Kullback-Leibler divergence between the approximate and posterior distributions, ensuring that the accuracy of data compression is within a preset threshold range.

8. A device for optimizing the compression and storage of meteorological forecast data, characterized in that, include: The data preprocessing module is used to acquire meteorological forecast data, convert the meteorological forecast data into a three-dimensional tensor form, and preprocess the meteorological forecast data in the three-dimensional tensor form to obtain preprocessed meteorological forecast data; wherein, the meteorological forecast data is reorganized according to time, space and physical variables; and the input data is rearranged into a three-dimensional matrix form. The wavelet decomposition module is used to perform three-dimensional wavelet decomposition on the preprocessed meteorological forecast data using orthogonal wavelet basis functions to obtain tensors. The tensor decomposition module is used to perform high-order tensor decomposition on tensors to obtain the core tensor and multiple modal feature matrices. The structured low-rank compression module is used to dynamically adjust the rank of the core tensor by regularization constraints and perform sparsity processing, and to reduce the dimensionality of multiple modal feature matrices to obtain a core tensor with reduced rank and multiple modal feature matrices with reduced storage size. The entropy coding module is used to perform entropy coding and quantization on the core tensor with reduced rank and multiple modal feature matrices with reduced storage size, so as to obtain a compressed representation of the core tensor and multiple modal feature matrices after entropy coding and quantization. The dynamic calibration and storage module is used to dynamically calibrate the storage accuracy and information loss of the compressed representation of the core tensor and multiple modal feature matrices after entropy encoding and quantization using variational inference. By constructing the posterior distribution and approximate distribution models of the core tensor after entropy encoding and quantization, and minimizing the Kullback-Leibler divergence between the posterior distribution and the approximate distribution of the core tensor after entropy encoding and quantization, the module dynamically adjusts the storage accuracy and information loss of the core tensor after entropy encoding and quantization, ensuring the reliability of data restoration and completing the optimization of compressed storage of meteorological forecast data.

Images