Meteorological prediction data compression storage optimization method and device

Abstract

The invention discloses a meteorological prediction data compression storage optimization method, which belongs to the technical field of meteorological data processing, and comprises the following steps: obtaining preprocessed meteorological prediction data; performing three-dimensional wavelet decomposition on the preprocessed meteorological prediction data by using an orthogonal wavelet basis function to obtain a tensor; performing high-order tensor decomposition on the tensor to obtain a core tensor and a plurality of modal feature matrixes; regularization constraint dynamic adjustment and sparse processing are carried out on the rank of the core tensor, dimension reduction processing is carried out on the multiple modal feature matrixes, and the core tensor with the reduced rank and the multiple modal feature matrixes with the reduced storage scale are obtained and entropy coding and quantization processing are carried out on the core tensor and the multiple modal feature matrixes; obtaining the core tensor subjected to entropy coding and quantization processing and the compressed representation of a plurality of modal feature matrixes; and dynamically calibrating the storage precision and the information loss of the compression representation of the core tensor and the plurality of modal feature matrixes after entropy coding and quantization processing by using variational inference to complete the compression storage optimization of the meteorological prediction data.

Description

Technical Field

[0001] The present invention relates to the technical field of meteorological data processing, and particularly to a method and device for optimizing the compression and storage of meteorological prediction data. Background Art

[0002] Meteorological prediction data is an important basic data that supports weather forecasting, climate research, and natural disaster warning, and has the characteristics of high dimension, high resolution, and large scale. With the continuous development of numerical weather prediction models and observation technologies, the generation speed and scale of meteorological data have increased exponentially, bringing huge pressure to storage and transmission. How to efficiently compress and store meteorological data while ensuring its accuracy and physical consistency has become a technical problem to be solved urgently.

[0003] Currently, the technologies for compressing and storing meteorological data mainly focus on dimensionality reduction methods based on feature extraction and compression algorithms based on information entropy. However, existing methods have certain limitations in dealing with the spatio-temporal characteristics and multi-scale characteristics of meteorological data. Specifically, traditional dimensionality reduction methods are prone to losing the physical characteristics of data during the compression process, resulting in the restored data being difficult to meet the actual prediction requirements. At the same time, common compression algorithms such as entropy coding-based processing technologies can reduce the data storage volume, but their compression efficiency and restoration accuracy still need to be improved when dealing with the high dimensionality and complex spatio-temporal correlation of meteorological data. In addition, for data with significant local characteristics such as extreme weather events, it is often difficult for such methods to achieve a dynamic balance between compression and restoration accuracy.

[0004] In practical applications, meteorological data needs to have the capabilities of large-scale storage, fast transmission, and high-precision restoration. Especially in real-time weather forecasting and disaster monitoring scenarios, the timeliness and accuracy of data are crucial. However, existing technologies have not fully combined the multi-scale characteristics and high-dimensional tensor structure of meteorological data, and have not achieved global optimization of compression storage and efficient restoration. Summary of the Invention

[0005] The purpose of the present invention is to provide a method and device for optimizing the compression and storage of meteorological prediction data. By using an entropy coding optimization algorithm to compress the element values in the core tensor and multiple modal feature matrices that appear frequently, and storing the compressed core tensor and multiple modal feature matrices in a specified storage medium, the compression storage is completed. The present invention realizes the efficient compression storage of large-scale meteorological prediction data while ensuring the restoration accuracy and physical consistency of meteorological data. The present invention is realized through the following technical solutions.

[0006] In the first aspect, the present invention provides a method for optimizing the compression and storage of meteorological prediction data, including: Obtain meteorological prediction data, construct the meteorological prediction data in the form of a three-dimensional tensor, and preprocess the meteorological prediction data converted into the three-dimensional tensor form to obtain the preprocessed meteorological prediction data; Perform three-dimensional wavelet decomposition on the preprocessed meteorological prediction data using orthogonal wavelet basis functions to obtain a tensor; Perform high-order tensor decomposition on the tensor to obtain a core tensor and multiple modal feature matrices; Perform regularization constraint dynamic adjustment and sparse processing on the rank of the core tensor, and perform dimensionality reduction processing on the multiple modal feature matrices to obtain a core tensor with reduced rank and multiple modal feature matrices with reduced storage scale; Perform entropy coding and quantization processing on the core tensor with reduced rank and the multiple modal feature matrices with reduced storage scale to obtain compressed representations of the core tensor and the multiple modal feature matrices after entropy coding and quantization processing; Use variational inference to dynamically calibrate the storage accuracy and information loss of the compressed representations of the core tensor and the multiple modal feature matrices after entropy coding and quantization processing to complete the compression storage optimization of the meteorological prediction data.

[0007] In practical applications, since traditional dimensionality reduction methods are prone to losing the physical characteristics of data during the compression process, resulting in the restored data being difficult to meet the actual prediction requirements, the entropy optimization algorithm and the dynamic calibration method combining variational inference proposed in the present invention can maintain a high-precision restoration effect while compressing the data. It can effectively reduce the storage space occupied by meteorological data and significantly reduce the storage and transmission costs of data. Especially in distributed storage and remote transmission scenarios, this method can significantly improve the economy and practicality of data processing. The method of the present invention is applicable to both large-scale grid data generated by numerical weather prediction models and spatio-temporal sequence data of actual observation stations. Its multi-modal and multi-scale processing capabilities make this method widely applicable in fields such as weather analysis, climate modeling, and disaster warning.

[0008] Optionally, the orthogonal wavelet basis functions include Haar wavelet basis functions or Daubechies wavelet basis functions; When performing three-dimensional wavelet decomposition on the preprocessed meteorological prediction data using orthogonal wavelet basis functions, low-frequency components and high-frequency components are also obtained, where the low-frequency components are used to describe the global characteristics of the meteorological prediction data, and the high-frequency components are used to describe the local characteristics of the meteorological prediction data.

[0009] The result of three-dimensional wavelet decomposition retains the multi-scale characteristics of the data.

[0010] Optionally, performing threshold processing on the high-frequency components includes setting to zero the high-frequency component coefficients less than a threshold based on an absolute value threshold of a preset high-frequency component coefficient, and retaining the high-frequency component coefficients greater than or equal to the threshold.

[0011] By performing significance screening on the high-frequency components, coefficients that make important contributions to restoration and feature extraction are retained, while noise and redundant information are removed, thereby further reducing the data storage amount and optimizing the compression efficiency.

[0012] Optionally, the performing high-order tensor decomposition on the tensor to obtain a core tensor and multiple modal feature matrices includes: Flattening the tensor along each modal direction to obtain flattened matrices for each modality; Performing singular value decomposition on the flattened matrices for each modality to obtain multiple modal feature matrices; Performing a tensor-matrix product operation on the tensor and the multiple modal feature matrices to obtain a core tensor.

[0013] Through high-order tensor decomposition, meteorological prediction data is mapped from a multi-dimensional space to a combined representation of a low-rank core tensor and modal feature matrices, thereby significantly reducing the data storage requirement while retaining the main features of the data.

[0014] Optionally, performing entropy encoding and quantization processing on the rank-reduced core tensor and multiple modal feature matrices with reduced storage scale includes: calculating the entropy value of the data value distribution of the rank-reduced core tensor and multiple modal feature matrices with reduced storage scale, and using the entropy value as a parameter of an entropy encoding optimization algorithm to guide data compression. Thereby further reducing the storage space of meteorological prediction data.

[0015] Optionally, the entropy encoding optimization algorithm includes the following steps: Statistically analyzing the numerical values of each element in the rank-reduced core tensor and multiple modal feature matrices with reduced storage scale, calculating the frequency of occurrence of each numerical value, and generating the probability distribution of the rank-reduced core tensor and multiple modal feature matrices with reduced storage scale; Based on the probability distribution of the rank-reduced core tensor and multiple modal feature matrices with reduced storage scale, using an entropy encoding algorithm to compress the rank-reduced core tensor and multiple modal feature matrices with reduced storage scale, dynamically adjusting the parameters of the entropy encoding algorithm, preferentially compressing the numerical values of the elements in the rank-reduced core tensor and multiple modal feature matrices with reduced storage scale that appear frequently, maximizing the data compression efficiency; storing the compressed rank-reduced core tensor and multiple modal feature matrices with reduced storage scale in a specified storage medium to complete compressed storage.

[0016] Optionally, the dynamic calibration of the storage precision and information loss of the compressed representations of the core tensor and multiple modal feature matrices after entropy encoding and quantization processing includes: by constructing a posterior distribution and an approximate distribution model of the core tensor after entropy encoding and quantization processing, and minimizing the Kullback-Leibler divergence between the posterior distribution of the core tensor after entropy encoding and quantization processing and the approximate distribution of the core tensor after entropy encoding and quantization processing, dynamically adjusting the storage precision and information loss of the core tensor after entropy encoding and quantization processing to ensure the reliability of data restoration.

[0017] By analyzing and optimizing the statistical characteristics of data through entropy encoding technology, further removal of redundant information can be achieved, thus significantly improving the storage efficiency of data. By constructing a posterior distribution and an approximate distribution model of the core tensor and using information theory optimization technology to achieve dynamic trade-off between storage efficiency and restoration precision, the efficiency and reliability of compressed data are ensured.

[0018] Optionally, the difference in the posterior distribution is calibrated by optimizing the parameters of the approximate distribution to minimize the Kullback-Leibler divergence between the approximate distribution and the posterior distribution, ensuring that the precision of data compression is within a preset threshold range.

[0019] In a second aspect, the present invention provides an apparatus for optimizing the compressed storage of meteorological prediction data, including: A data preprocessing module, configured to obtain meteorological prediction data, convert the meteorological prediction data into a three-dimensional tensor form, and preprocess the meteorological prediction data in the three-dimensional tensor form to obtain preprocessed meteorological prediction data; A wavelet decomposition module, configured to perform three-dimensional wavelet decomposition on the preprocessed meteorological prediction data using orthogonal wavelet basis functions to obtain a tensor; A tensor decomposition module, configured to perform high-order tensor decomposition on the tensor to obtain a core tensor and multiple modal feature matrices; A structured low-rank compression module, configured to perform regularized constraint dynamic adjustment and sparse processing on the rank of the core tensor, and perform dimensionality reduction processing on multiple modal feature matrices to obtain a core tensor with reduced rank and multiple modal feature matrices with reduced storage scale; An entropy encoding module, configured to perform entropy encoding and quantization processing on the core tensor with reduced rank and multiple modal feature matrices with reduced storage scale to obtain compressed representations of the core tensor and multiple modal feature matrices after entropy encoding and quantization processing; A dynamic calibration and storage module, configured to dynamically calibrate the storage precision and information loss of the compressed representations of the core tensor and multiple modal feature matrices after entropy encoding and quantization processing, and complete the optimization of the compressed storage of meteorological prediction data Beneficial effects

[0020] (1) By combining three-dimensional wavelet decomposition and high-order tensor decomposition, the present invention performs multi-level compression optimization on meteorological prediction data. Three-dimensional wavelet decomposition extracts the low-frequency and high-frequency characteristics of the data, and high-order tensor decomposition further performs rank constraint and dimensionality reduction representation on high-dimensional data, effectively reducing the redundant storage requirements and significantly improving the data compression ratio. By utilizing the multi-scale processing mechanism of three-dimensional wavelet decomposition and the parallelism of tensor decomposition, fast compression and restoration of large-scale meteorological data are achieved.

[0021] (2) Through entropy optimization of the core tensor and multiple modal feature matrices, combined with the dynamic calibration method of variational inference, the present invention can maintain a high-precision restoration effect while compressing the data. Even under high compression ratio conditions, the physical consistency and spatio-temporal characteristics of the restored data are fully guaranteed, meeting the actual application requirements of meteorological prediction. Entropy optimization further reduces the storage and processing overhead of the core tensor.

[0022] (3) The present invention introduces variational inference technology, and through constructing a posterior distribution model and information loss constraint, dynamic calibration is performed for different data scenarios, such as high-precision prediction or fast storage. This adaptability ensures that the compression scheme can be flexibly applied to various demand scenarios, such as extreme weather prediction or long-term meteorological archiving. Brief description of the drawings

[0023] Figure 1 The figure shows the flowchart of the method for optimizing the compression and storage of meteorological prediction data of the present invention; Figure 2 The figure shows the schematic structural diagram of the device for optimizing the compression and storage of meteorological prediction data of the present invention; In the figure: 10 - data preprocessing module, 20 - wavelet decomposition module, 30 - tensor decomposition module, 40 - structured low-rank compression module, 50 - entropy coding module, 60 - dynamic calibration and storage module. Detailed implementation manners

[0024] The following is further described in conjunction with the drawings and specific embodiments. In the description of the present invention, it should be understood that terms such as "first", "second", etc. are only used for descriptive purposes and cannot be understood as indicating or implying relative importance or implicitly specifying the quantity of the indicated technical features. Thus, features defined with "first", "second", etc. may explicitly or implicitly include one or more of such features.

[0025] Embodiment 1 This embodiment introduces a method for optimizing the compression and storage of meteorological prediction data, as Figure 1 shown, including the following content: Obtain meteorological prediction data, construct the meteorological prediction data in the form of a three-dimensional tensor, and preprocess the meteorological prediction data converted into the three-dimensional tensor form to obtain the preprocessed meteorological prediction data; Perform three-dimensional wavelet decomposition on the preprocessed meteorological prediction data using orthogonal wavelet basis functions to obtain a tensor; Perform high-order tensor decomposition on the tensor to obtain a core tensor and multiple modal feature matrices; Perform regularization constraint dynamic adjustment and sparse processing on the rank of the core tensor, and perform dimensionality reduction processing on multiple modal feature matrices to obtain a core tensor with reduced rank and multiple modal feature matrices with reduced storage scale; Perform entropy coding and quantization processing on the core tensor with reduced rank and multiple modal feature matrices with reduced storage scale to obtain compressed representations of the core tensor and multiple modal feature matrices after entropy coding and quantization processing; Use variational inference to dynamically calibrate the storage accuracy and information loss of the compressed representations of the core tensor and multiple modal feature matrices after entropy coding and quantization processing to complete the compression storage optimization of meteorological prediction data.

[0026] In practical applications, since traditional dimensionality reduction methods are prone to losing the physical characteristics of data during the compression process, resulting in the restored data being difficult to meet the actual prediction requirements, the entropy optimization algorithm and the dynamic calibration method combining variational inference proposed in the present invention can maintain a high-precision restoration effect while compressing data. It can effectively reduce the storage space occupied by meteorological data and significantly reduce the storage and transmission costs of data. Especially in distributed storage and remote transmission scenarios, this method can significantly improve the economy and practicality of data processing. The method of the present invention is applicable to both large-scale grid data generated by numerical weather prediction models and spatio-temporal sequence data of actual observation stations. Its multi-modal and multi-scale processing capabilities make this method widely applicable in the fields of weather analysis, climate modeling, disaster warning, etc.

[0027] Embodiment 2 Based on Embodiment 1, this embodiment introduces a specific implementation process of a method for optimizing the compression storage of meteorological prediction data, as Figure 1 shown, which specifically includes the following contents: I. Data preprocessing First, obtain meteorological prediction data, construct the meteorological prediction data in the form of a three-dimensional tensor, and preprocess the meteorological prediction data converted into the three-dimensional tensor form to obtain the preprocessed meteorological prediction data; Meteorological prediction data usually comes from numerical weather prediction models or observation stations. Its data structure includes a time dimension, such as hourly or daily data, a spatial dimension such as longitude and latitude grids, and a physical variable dimension such as temperature, humidity, and wind speed, etc. Due to different data sources or measurement errors, these data may have magnitude differences, irregular missing values, or inconsistent formats. Therefore, after being constructed into a three-dimensional tensor form, preprocessing operations are required.

[0028] As an option, meteorological prediction data can first be reorganized according to time, space, and physical variables; specifically, the input data can be rearranged into a three-dimensional matrix form.

[0029] For example, in some embodiments, the hourly recorded temperature, humidity, and wind speed data can respectively correspond to different physical variable layers of a three-dimensional tensor. In this case, the input data can be expressed as , where x is the input data, I is the time step, such as 24 hours, J is the number of spatial grid points, such as a 10×10 longitude and latitude grid, and K is the number of physical variables, such as 3 variables including temperature, humidity, and wind speed.

[0030] To eliminate the magnitude differences between different physical variables, in this embodiment, each physical variable is standardized.

[0031] Through the standardization process, the data distribution of each variable is adjusted to a form with a mean of zero and a standard deviation of one, which is convenient for subsequent decomposition algorithms to equally process each physical variable.

[0032] As an implementation method, if there are missing values in the data, in this embodiment, interpolation methods are used to fill them. Specifically, algorithms such as linear interpolation, spline interpolation, or nearest neighbor interpolation can be used to fill the missing points. Taking linear interpolation as an example, when the data at a certain time step is missing, the value of the missing point can be linearly calculated through the data values of adjacent time points.

[0033] It should be noted that the choice of interpolation method can be adjusted according to the specific characteristics of the data. For example, for extreme weather data with strong mutations, spline interpolation can be selected to better capture the change trend.

[0034] To ensure that the data is compatible with subsequent wavelet decomposition and tensor decomposition operations, in this embodiment, the obtained meteorological prediction data is uniformly organized into a three-dimensional tensor format. This formatting operation includes adjusting the storage order of the data, filling the missing dimensions of the data, and assigning a unified identifier to the physical variables. For example, temperature, humidity, and wind speed can respectively correspond to the data of the first layer, the second layer, and the third layer of the tensor.

[0035] In summary, the standardized preprocessing scheme for meteorological prediction data provided by this embodiment includes data standardization, interpolation processing, and formatting operations, and optionally noise filtering and smoothing processing. These operations provide high-quality input data for subsequent three-dimensional wavelet decomposition, high-order tensor decomposition, and compression optimization, ensuring the processing effect and algorithm performance of subsequent steps.

[0036] II. Three-dimensional wavelet decomposition and threshold processing The orthogonal wavelet basis function includes Haar wavelet basis function or Daubechies wavelet basis function; The three-dimensional wavelet decomposition of the preprocessed meteorological prediction data using the orthogonal wavelet basis function also obtains a low-frequency component and a high-frequency component, where the low-frequency component is used to describe the global characteristics of the meteorological prediction data, and the high-frequency component is used to describe the local characteristics of the meteorological prediction data.

[0037] The threshold processing of the high-frequency component includes setting the high-frequency component coefficients less than the threshold to zero and retaining the high-frequency component coefficients greater than or equal to the threshold based on the absolute value threshold of the pre-set high-frequency component coefficients.

[0038] In this embodiment, the high-frequency components generated by the three-dimensional wavelet decomposition contain a large number of detailed features, and these features may be mixed with measurement noise, random fluctuations, and redundant information that contributes less to data restoration. In order to reduce the redundancy of data storage, a threshold-based screening method is designed to retain only the significant coefficients in the high-frequency components. The threshold can be determined in the following way: according to the energy distribution of the high-frequency components, select the first 90% or 95% of the total energy as the threshold retention range. Set the high-frequency component coefficients less than the threshold to zero and retain the high-frequency component coefficients greater than or equal to the threshold.

[0039] The embodiment also designs a multi-level dynamic adjustment mechanism. For example, after the initial screening, the threshold can be adjusted according to the total amount of the remaining coefficients: if the total amount of the screened coefficients is less than the preset lower limit, such as 10% of the total number of coefficients, the threshold is appropriately reduced; if the total amount of the screened coefficients is higher than the preset upper limit, such as 50% of the total number of coefficients, the threshold is appropriately increased. This dynamic adjustment mechanism can ensure that the screened high-frequency components have sufficient information content, meeting both the compression requirements and not affecting the accuracy of data restoration.

[0040] The screening method in this embodiment is not only applicable to the high-frequency components of a single layer, but also can be extended to the overall screening of all high-frequency components. In this extended implementation method, the coefficient distributions of all high-frequency components can be jointly analyzed to determine a global unified threshold. This method is applicable to the case where the energy distribution differences between high-frequency components are small, and can further simplify the complexity of threshold setting.

[0041] It should be understood that the process of screening high-frequency components is a further optimization of the three-dimensional wavelet decomposition result. Its core objective is to minimize the storage requirements for redundant information while retaining sufficient significant coefficients to support data reconstruction and analysis. Through the above processing, the storage scale of meteorological prediction data can be effectively reduced, providing high-quality input data for subsequent higher-order tensor decomposition and entropy optimization.

[0042] III. Higher-order Tensor Decomposition Perform three-dimensional wavelet decomposition on the preprocessed meteorological prediction data using orthogonal wavelet basis functions to obtain a tensor; perform higher-order tensor decomposition on the tensor to obtain a core tensor and multiple modal feature matrices. Higher-order tensor decomposition is a dimensionality reduction technique applicable to high-dimensional data, and its goal is to represent the original tensor as the product of a core tensor and several modal feature matrices. This representation can capture the structural characteristics of the data in different modalities, such as time, space, and physical variables, and achieve data compression through rank constraints. The performing higher-order tensor decomposition on the tensor to obtain a core tensor and multiple modal feature matrices specifically includes the following: Flatten the tensor along each modal direction to obtain flattened matrices for each modality; Perform singular value decomposition on the flattened matrices for each modality to obtain multiple modal feature matrices; Perform a tensor-matrix multiplication operation on the tensor and the multiple modal feature matrices to obtain the core tensor.

[0043] Through higher-order tensor decomposition, the data after three-dimensional wavelet decomposition can be further compressed while retaining its main structural characteristics, providing a compact and efficient input for subsequent entropy optimization and dynamic calibration.

[0044] IV. Entropy Encoding The entropy encoding and quantization processing of the core tensor with reduced rank and the multiple modal feature matrices with reduced storage scale include: calculating the entropy value of the data value distribution of the core tensor with reduced rank and the multiple modal feature matrices with reduced storage scale, and using the entropy value as a parameter of the entropy encoding optimization algorithm to guide data compression.

[0045] The entropy encoding optimization algorithm includes the following steps: Statistically analyze the numerical values of each element in the core tensor with reduced rank and the multiple modal feature matrices with reduced storage scale, calculate the frequency of occurrence of each numerical value, and generate the probability distribution of the core tensor with reduced rank and the multiple modal feature matrices with reduced storage scale; Based on the probability distributions of the low-rank core tensor and multiple modality feature matrices with reduced storage scale, use the entropy coding algorithm to compress the low-rank core tensor and multiple modality feature matrices with reduced storage scale, dynamically adjust the parameters of the entropy coding algorithm, preferentially compress the element values in the frequently occurring low-rank core tensor and multiple modality feature matrices with reduced storage scale, and store the compressed low-rank core tensor and multiple modality feature matrices with reduced storage scale in the specified storage medium to complete the compressed storage.

[0046] Through the above steps, the storage requirements of meteorological prediction data can be significantly reduced, providing a compact and efficient compressed representation for subsequent dynamic calibration and data restoration.

[0047] V. Dynamic Calibration and Data Storage The dynamic calibration of the storage precision and information loss of the compressed representations of the core tensor and multiple modality feature matrices after entropy coding and quantization using variational inference includes: by constructing the posterior distribution and approximate distribution model of the core tensor after entropy coding and quantization, and minimizing the Kullback-Leibler divergence between the posterior distribution of the core tensor after entropy coding and quantization and the approximate distribution of the core tensor after entropy coding and quantization, dynamically adjust the storage precision and information loss of the core tensor after entropy coding and quantization to ensure the reliability of data restoration.

[0048] The difference in the posterior distribution is calibrated by optimizing the parameters of the approximate distribution to minimize the Kullback-Leibler divergence between the approximate distribution and the posterior distribution, ensuring that the data compression accuracy is within the preset threshold range. The optimization of the compressed storage of meteorological prediction data is completed.

[0049] The core advantage of variational inference lies in its dynamic adaptability. By performing real-time calibration on the storage representation form of the core tensor, the optimization strategy can be flexibly adjusted according to the storage requirements and restoration accuracy requirements of different scenarios. For example, for key meteorological data that requires high-precision restoration, such as extreme weather-related information, a lower information loss threshold can be set to ensure that the restoration error is controlled within a smaller range.

[0050] Through the above method, the storage precision of the core tensor can be effectively optimized, providing technical support for the efficient compression and reliable restoration of meteorological prediction data.

[0051] The present invention extracts multi-scale features of meteorological data through three-dimensional wavelet decomposition, combines high-order tensor decomposition to achieve efficient dimensionality reduction, further reduces storage redundancy through entropy optimization, and dynamically calibrates information loss during the compression process using variational inference. The present invention can not only significantly improve the compression efficiency of meteorological data, but also maintain its physical consistency and prediction accuracy during data restoration, providing an efficient and reliable solution for the storage and transmission of large-scale meteorological data.

[0052] Embodiment 3 This embodiment provides an apparatus for optimizing the compression and storage of meteorological prediction data, as Figure 2 shown, including the following: A data preprocessing module 10, configured to obtain meteorological prediction data, convert the meteorological prediction data into a three-dimensional tensor form, and preprocess the meteorological prediction data in the three-dimensional tensor form to obtain preprocessed meteorological prediction data; A wavelet decomposition module 20, configured to perform three-dimensional wavelet decomposition on the preprocessed meteorological prediction data using orthogonal wavelet basis functions to obtain a tensor; A tensor decomposition module 30, configured to perform high-order tensor decomposition on the tensor to obtain a core tensor and multiple modal feature matrices; A structured low-rank compression module 40, configured to perform regularized constraint dynamic adjustment and sparse processing on the rank of the core tensor, perform dimensionality reduction processing on multiple modal feature matrices, and obtain a core tensor with reduced rank and multiple modal feature matrices with reduced storage scale; An entropy coding module 50, configured to perform entropy coding and quantization processing on the core tensor with reduced rank and multiple modal feature matrices with reduced storage scale to obtain compressed representations of the core tensor and multiple modal feature matrices after entropy coding and quantization processing; A dynamic calibration and storage module 60, configured to dynamically calibrate the storage accuracy and information loss of the compressed representations of the core tensor and multiple modal feature matrices after entropy coding and quantization processing using variational inference to complete the optimization of the compression and storage of meteorological prediction data The embodiments of the present invention have been described above in conjunction with the accompanying drawings. However, the present invention is not limited to the above specific embodiments. The above specific embodiments are merely illustrative and not restrictive. Under the inspiration of the present invention, those of ordinary skill in the art can also make many forms without departing from the spirit and scope protected by the present invention and the claims. These all fall within the protection scope of the present invention.

Claims

1. A method for optimizing the compression and storage of meteorological forecast data, characterized in that: include: Acquire weather forecast data, construct the weather forecast data into a three-dimensional tensor form, and preprocess the weather forecast data converted into the three-dimensional tensor form to obtain preprocessed weather forecast data; The preprocessed meteorological forecast data is decomposed into three-dimensional wavelets using orthogonal wavelet basis functions to obtain tensors; Perform high-order tensor decomposition on the tensor to obtain the core tensor and multiple modal feature matrices; The rank of the core tensor is dynamically adjusted and sparsely processed by regularization constraints, and multiple modal feature matrices are reduced in dimension to obtain a core tensor with reduced rank and multiple modal feature matrices with reduced storage size; Performing entropy coding and quantization processing on the core tensor with reduced rank and multiple modal feature matrices with reduced storage size, and obtaining compressed representations of the core tensor and multiple modal feature matrices after entropy coding and quantization processing; Variational inference is used to dynamically calibrate the storage accuracy and information loss of the compressed representation of the core tensor and multiple modal feature matrices after entropy coding and quantization, thereby completing the compression storage optimization of meteorological forecast data.

2. The method for optimizing the compression and storage of weather forecast data according to claim 1, characterized in that: The orthogonal wavelet basis function includes a Haar wavelet basis function or a Daubechies wavelet basis function; The three-dimensional wavelet decomposition of the preprocessed meteorological forecast data using orthogonal wavelet basis functions also obtains low-frequency components and high-frequency components, wherein the low-frequency components are used to describe the global characteristics of the meteorological forecast data, and the high-frequency components are used to describe the local characteristics of the meteorological forecast data.

3. The method for optimizing the compression and storage of weather forecast data according to claim 2, characterized in that: The threshold processing of the high frequency component includes setting the high frequency component coefficients smaller than the threshold to zero and retaining the high frequency component coefficients greater than or equal to the threshold based on a preset absolute value threshold of the high frequency component coefficients.

4. The method for optimizing the compression and storage of weather forecast data according to claim 1, characterized in that: The high-order tensor decomposition of the tensor to obtain the core tensor and multiple modal feature matrices includes: Flatten the tensor along each mode direction to obtain the flattened matrix of each mode; Perform singular value decomposition on the flattened matrix of each mode to obtain multiple modal characteristic matrices; Perform tensor-matrix product operations on the tensor and multiple modal feature matrices to obtain the core tensor.

5. The method for optimizing the compression and storage of weather forecast data according to claim 1, characterized in that: The entropy coding and quantization processing of the reduced-rank core tensor and the multiple modal feature matrices with reduced storage scale includes: calculating the entropy value of the data value distribution of the reduced-rank core tensor and the multiple modal feature matrices with reduced storage scale, and using the entropy value as a parameter of the entropy coding optimization algorithm to guide data compression.

6. The method for optimizing the compression and storage of weather forecast data according to claim 5, characterized in that: The entropy coding optimization algorithm comprises the following steps: The numerical values ​​of each element in the core tensor with reduced rank and the multiple modal feature matrices with reduced storage scale are counted, the frequency of occurrence of each element numerical value is calculated, and the probability distribution of the core tensor with reduced rank and the multiple modal feature matrices with reduced storage scale is generated; Based on the probability distribution of the reduced-rank core tensor and the multiple modal feature matrices with reduced storage scale, an entropy coding algorithm is used to compress the reduced-rank core tensor and the multiple modal feature matrices with reduced storage scale, and the parameters of the entropy coding algorithm are dynamically adjusted to give priority to compressing the values ​​of elements in the reduced-rank core tensor and the multiple modal feature matrices with reduced storage scale that appear frequently, and the compressed reduced-rank core tensor and the multiple modal feature matrices with reduced storage scale are stored in a designated storage medium to complete the compressed storage.

7. The method for optimizing the compression and storage of weather forecast data according to claim 1, characterized in that: The method uses variational inference to dynamically calibrate the storage accuracy and information loss of the compressed representation of the core tensor and multiple modal feature matrices after entropy coding and quantization, including: constructing a posterior distribution and an approximate distribution model of the core tensor after entropy coding and quantization, and minimizing the Kullback-Leibler divergence between the posterior distribution of the core tensor after entropy coding and quantization and the approximate distribution of the core tensor after entropy coding and quantization, dynamically adjusting the storage accuracy and information loss of the core tensor after entropy coding and quantization, and ensuring the reliability of data recovery.

8. The method for optimizing the compression and storage of weather forecast data according to claim 7, characterized in that: The difference in the posterior distribution is calibrated by optimizing the parameters of the approximate distribution to minimize the Kullback-Leibler divergence between the approximate distribution and the posterior distribution, ensuring that the accuracy of data compression is within a preset threshold range.

9. A device for compressing and storing weather forecast data, characterized in that: include: A data preprocessing module is used to obtain weather forecast data, convert the weather forecast data into a three-dimensional tensor form, and preprocess the weather forecast data converted into the three-dimensional tensor form to obtain the preprocessed weather forecast data; The wavelet decomposition module is used to perform three-dimensional wavelet decomposition on the pre-processed meteorological forecast data using orthogonal wavelet basis functions to obtain a tensor; Tensor decomposition module, used to perform high-order tensor decomposition on tensors to obtain core tensors and multiple modal feature matrices; The structured low-rank compression module is used to dynamically adjust the rank of the core tensor with regularization constraints and perform sparse processing, and to reduce the dimensionality of multiple modal feature matrices to obtain a core tensor with reduced rank and multiple modal feature matrices with reduced storage size; An entropy coding module is used to perform entropy coding and quantization processing on the core tensor with reduced rank and multiple modal feature matrices with reduced storage size, so as to obtain compressed representations of the core tensor and multiple modal feature matrices after entropy coding and quantization processing; The dynamic calibration and storage module is used to dynamically calibrate the storage accuracy and information loss of the compressed representation of the core tensor and multiple modal feature matrices after entropy coding and quantization using variational inference, so as to complete the compression storage optimization of meteorological forecast data.

Images