A method for compressing and reconstructing meteorological data

Abstract

The application discloses a meteorological data compression reconstruction method, and belongs to the technical field of data compression reconstruction. The method comprises the following steps: obtaining tensor data by preprocessing meteorological monitoring data based on a normalized mapping function and geographical mask information; obtaining basic meteorological reconstruction data by processing the tensor data based on a cascaded enhanced hybrid coding network; if the physical error between the meteorological monitoring data and the basic meteorological reconstruction data is less than or equal to a preset threshold, then taking the basic binary code stream as a compression result; otherwise, constructing a sparse binary mask according to the basic meteorological reconstruction data whose physical error is greater than the preset threshold; extracting mask region residual values and performing quantization to generate an incremental patch code stream, and taking the basic binary code stream and the incremental patch code stream as the compression result; and obtaining reconstruction data by joint decoding and de-normalization based on the compression result. The application solves the problems in the prior art that high compression rate and high fidelity are difficult to achieve, fluid dynamics characteristics are easy to lose, and extreme abnormal value error is uncontrollable.

Description

Technical Field

[0001] This invention relates to a method for compressing and reconstructing meteorological data, belonging to the field of data compression and reconstruction technology. Background Technology

[0002] Meteorological data compression and reconstruction are key technical aspects of meteorological monitoring, forecasting, and climate research. Traditional meteorological data compression methods mainly employ general-purpose lossless compression algorithms or transform-based lossy compression techniques. While general-purpose lossless compression algorithms, such as arithmetic coding, can ensure complete data reconstruction, their compression ratios are low, making it difficult to meet the demands of efficient storage and transmission of large-scale meteorological data. Transform-based lossy compression techniques, by converting data to the frequency domain using discrete wavelet transform or discrete cosine transform before quantization and entropy coding, improve compression efficiency to some extent. In recent years, with the development of deep learning technology, autoencoder architectures based on convolutional neural networks have been gradually introduced into the field of meteorological data compression. These architectures achieve feature extraction and reconstruction through end-to-end training, demonstrating potential in compression performance. Furthermore, some studies attempt to incorporate the physical characteristics of meteorological data into the compression framework, such as using geographic masks to remove invalid regions and utilizing normalized mapping functions to adapt to the dimensional differences of different physical quantities, aiming to preserve key meteorological features during the compression process.

[0003] However, existing meteorological data compression methods still face many limitations in practical applications. First, it is difficult to achieve both high compression ratio and high fidelity: while general lossless compression algorithms can guarantee reconstruction accuracy, their compression ratio is limited; and lossy compression methods, in pursuing high compression ratios, often lead to significant loss of detail and blurring effects in the reconstructed data, making it difficult to achieve an ideal balance between compression efficiency and data quality. Second, fluid dynamic features are easily lost: meteorological data have typical fluid dynamic characteristics, such as mesoscale and small-scale structures like vortices, fronts, and jet streams. These structures are crucial for meteorological analysis and forecasting, but existing compression methods are usually based on general image or signal processing frameworks, lacking targeted modeling of physical priors such as the spatial continuity of the meteorological field, gradient distribution, and vortex structure, resulting in over-smoothing or distortion of key dynamic features during compression. Third, the error of extreme outliers is uncontrollable: There are a large number of extreme outliers in meteorological data, such as the central pressure of typhoons, extreme rainfall, and extreme temperatures. These outliers often contain important extreme weather information. However, existing compression methods adopt a global optimization strategy in error allocation, which cannot differentiate the extreme outlier regions. This results in the error of extreme values ​​in the reconstructed data being uncontrollable, which seriously affects the ability to accurately identify and warn of extreme weather events. Summary of the Invention

[0004] The purpose of this invention is to provide a meteorological data compression and reconstruction method. The method uses a cascaded enhanced hybrid coding network to perform basic compression to obtain a basic binary code stream. Based on whether the physical error between the basic meteorological reconstructed data and the original data exceeds a preset threshold, an incremental patch code stream is selectively generated to form a joint compression result with the basic binary code stream. After joint decoding and inverse normalization, the data is reconstructed. This method solves the problems in the prior art, such as the difficulty in achieving both high compression ratio and high fidelity, the easy loss of fluid dynamic characteristics, and the uncontrollable errors of extreme outliers.

[0005] To solve the above-mentioned technical problems, the present invention is implemented using the following technical solution.

[0006] This invention provides a method for compressing and reconstructing meteorological data, comprising:

[0007] Obtain meteorological monitoring data;

[0008] The meteorological monitoring data is preprocessed based on the normalized mapping function and geographic mask information to obtain tensor data;

[0009] Based on the tensor data, feature extraction and probability distribution prediction are performed using a pre-trained cascaded enhanced hybrid coding network to obtain prediction results. Based on the prediction results, compression is performed to obtain a basic binary code stream, and the basic binary code stream is reconstructed locally at the encoding end to obtain basic meteorological reconstruction data.

[0010] If the physical error between the meteorological monitoring data and the basic meteorological reconstruction data is less than or equal to a preset threshold, then the basic binary code stream will be used as the compression result.

[0011] If the physical error between meteorological monitoring data and basic meteorological reconstruction data is greater than a preset threshold, a sparse binary mask is constructed based on the basic meteorological reconstruction data with the physical error greater than the preset threshold and the preset physical tolerance threshold.

[0012] Extract the residual values ​​of the mask region of the sparse binary mask and quantize them to generate an incremental patch bitstream;

[0013] The incremental patch bitstream and the basic binary bitstream are used as the compression result;

[0014] The reconstructed data is obtained by joint decoding and inverse normalization based on the compression results.

[0015] Furthermore, the meteorological monitoring data is preprocessed based on the normalized mapping function and geographic masking information to obtain tensor data, including:

[0016] The physical quantity data in the meteorological monitoring data are mapped using a normalized mapping function;

[0017] Based on the geographic mask information, "0" is filled into the invalid areas of the mapped physical quantity data to obtain tensor data.

[0018] Furthermore, the network structure of the cascaded enhanced hybrid coding network includes:

[0019] The front-end feature enhancement module includes at least two cascaded residual dense groups, which are used to extract features from the tensor data and increase the number of channels before outputting them to the core compression module.

[0020] The core compression module is used to compress tensor data after increasing the number of channels to obtain a compressed tensor; perform probability distribution prediction on the compressed tensor to obtain a prediction result, and perform entropy encoding on the prediction result to obtain a basic binary bitstream; perform entropy decoding on the basic binary bitstream to obtain a reconstruction result; and decompress the reconstruction result to obtain a decompressed tensor.

[0021] The backend feature reconstruction module includes at least two cascaded residual dense groups, which are used to reduce the number of channels by decompressing the tensor and then reconstruct the basic meteorological reconstruction data.

[0022] Furthermore, the core compression module includes a downsampling layer, a super-prior network, and an upsampling layer connected in sequence;

[0023] The downsampling layer includes at least two sequentially connected residual blocks, which are used to compress tensor data after increasing the number of channels by folding spatial dimensions to obtain a compressed tensor.

[0024] The super-prior network is used to predict the probability distribution of the compressed tensor through statistical modeling to obtain the prediction result, and to perform entropy encoding based on the prediction result to obtain the basic binary code stream, and to perform entropy decoding on the basic binary code stream to obtain the reconstruction result.

[0025] The upsampling layer includes at least two sequentially connected residual blocks, which are used to decompress the reconstruction results by restoring spatial dimensions to obtain a decompressed tensor and output it to the backend feature reconstruction module.

[0026] Further, a sparse binary mask is constructed based on the basic meteorological reconstruction data where the physical error exceeds a preset threshold and a preset physical tolerance threshold, including:

[0027] The physical residual matrix is ​​obtained by calculating the location-by-location residuals of the basic meteorological reconstruction data with physical errors exceeding a preset threshold and the corresponding geographic grid locations of the meteorological monitoring data. The absolute value of the physical residual matrix is ​​then processed to obtain the absolute error distribution. Finally, the location error values ​​in the absolute error distribution are compared with a preset physical tolerance threshold.

[0028] When the absolute error value corresponding to the geographic grid location is greater than the preset physical tolerance threshold, the sparse binary mask corresponding to the geographic grid location is marked as "1".

[0029] When the absolute error value corresponding to the geographic grid location is less than or equal to the preset physical tolerance threshold, the sparse binary mask corresponding to the geographic grid location is marked as "0".

[0030] Further, the residual values ​​of the mask region of the sparse binary mask are extracted and quantized to generate an incremental patch bitstream, including:

[0031] The residual values ​​corresponding to the geographic grid locations are extracted based on the sparse binary mask; the residual values ​​corresponding to the geographic grid locations are quantized to obtain quantized residual values; the quantized residual values ​​are entropy encoded based on the geographic grid locations corresponding to the sparse binary mask to obtain incremental patch bitstreams.

[0032] Furthermore, based on the compression results, joint decoding and inverse normalization are performed to obtain the reconstructed data, including:

[0033] When the compression result includes only the basic binary bitstream:

[0034] The basic binary code stream is parsed and decoded to recover the basic meteorological reconstruction data; the basic meteorological reconstruction data is then denormalized to obtain the reconstructed data.

[0035] When the compression result includes both the incremental patch bitstream and the base binary bitstream:

[0036] The basic binary code stream is parsed and decoded to recover the basic meteorological reconstruction data; the incremental patch code stream is parsed and decoded to recover the residual values ​​of the mask region of the sparse binary mask and the corresponding geographic grid positions.

[0037] The residual values ​​of the mask region of the sparse binary mask are fused into the basic meteorological reconstruction data according to the corresponding geographic grid location to obtain the corrected basic meteorological reconstruction data; the corrected basic meteorological reconstruction data is then subjected to inverse normalization to obtain the reconstruction data.

[0038] Furthermore, the reconstructed data is represented as follows:

[0039] ;

[0040] In the formula, Indicates the first Reconstruction data for each channel, Indicates the first Normalized reconstructed data for each channel after residual correction Indicates the first The maximum value parameter used by each channel during normalization mapping. Indicates the first The minimum parameter used by each channel during normalization mapping. This represents element-wise multiplication. This represents the geographic mask matrix, specifically the land mask matrix. This indicates that the reconstruction results for non-land areas are preserved while land areas are masked.

[0041] Furthermore, the method for training the cascaded enhanced hybrid coding network includes:

[0042] Historical meteorological monitoring data is acquired, and a sample set is constructed after preprocessing the historical meteorological monitoring data based on the normalized mapping function and geographic mask information. The sample set is then divided into training samples and validation samples.

[0043] Construct a joint loss function, including bit rate loss and mean squared error loss;

[0044] The training samples are input into the cascaded enhanced hybrid coding network, the joint loss function is minimized through the backpropagation algorithm, the network parameters are iteratively updated until convergence, and the optimal cascaded enhanced hybrid coding network parameters are selected based on the validation samples to obtain the trained cascaded enhanced hybrid coding network.

[0045] Furthermore, the joint loss function is expressed as:

[0046] ;

[0047] In the formula, Denotes the joint loss function. This represents the estimated bitrate value of the main latent feature portion corresponding to the basic binary bitstream. This represents the quantified principal latent features. This represents the estimated bitrate value of the extra-prior latent features. This represents the quantified prior latent features. Represents meteorological monitoring data With reconstruction data The mean square error between them These are Lagrange multipliers used to balance compressibility and physical accuracy.

[0048] Compared with the prior art, the beneficial effects achieved by the present invention are as follows:

[0049] 1. This invention constructs a two-layer coding architecture of basic compression plus incremental compensation. In the basic compression stage, a cascaded enhanced hybrid coding network is used to extract features, predict probability distributions, and entropy encode preprocessed tensor data, generating a basic binary code stream and performing local reconstruction. When the physical error between the basic meteorological reconstruction data and the original meteorological monitoring data exceeds a preset threshold, a sparse binary mask is constructed based on a preset physical tolerance threshold. The residual values ​​of the mask region are extracted and quantized to generate an incremental patch code stream. The incremental patch code stream and the basic binary code stream are used together as the compression result for joint decoding and inverse normalization. The basic compression layer achieves efficient feature extraction and probabilistic modeling through a cascaded enhanced hybrid coding network, preserving the main structure and key features of the meteorological data while ensuring a high compression ratio. The incremental compensation layer, through a physical error feedback mechanism and a preset physical tolerance threshold, only performs targeted residual compensation for key areas where the error exceeds the limit, avoiding the loss of details and the spread of extreme outlier errors caused by global quantization. Therefore, this invention achieves efficient storage and transmission under high compression ratio, effectively preserves fluid dynamic characteristics such as vortices and fronts through differentiated error control strategies, and ensures that the reconstruction error of extreme anomalies such as typhoon center pressure and extreme rainfall values ​​is controllable. This solves the problems of difficulty in achieving both high compression ratio and high fidelity, easy loss of fluid dynamic characteristics, and uncontrollable errors of extreme anomalies in the prior art.

[0050] 2. In the basic compression stage, this invention utilizes a cascaded enhanced hybrid coding network comprising a front-end feature enhancement module, a core compression module, and a back-end feature reconstruction module to process the preprocessed tensor data. Specifically, the front-end feature enhancement module extracts features and increases the number of channels through at least two cascaded residual dense sets; the core compression module compresses the data through spatial size folding of the downsampling layer, performs entropy coding after statistical modeling and probability distribution prediction via a priori network; and the back-end feature reconstruction module reduces the number of channels before reconstruction using at least two cascaded residual dense sets. This network structure enhances feature representation capabilities through cascaded residual dense sets, combined with probabilistic modeling and entropy coding optimization via a priori network, achieving high-quality local reconstruction while maintaining a high compression ratio. This overcomes the technical bottleneck of the traditional method where compression ratio and fidelity are mutually constrained.

[0051] 3. When the physical error between the basic meteorological reconstruction data and the original meteorological monitoring data exceeds a preset threshold, this invention performs position-by-position residual calculation on the geographic grid locations corresponding to the basic meteorological reconstruction data and meteorological monitoring data with physical errors exceeding the preset threshold. This yields a physical residual matrix, which is then processed to obtain the absolute error distribution. The error values ​​at each location are compared with a preset physical tolerance threshold to construct a sparse binary mask. Based on this, the residual values ​​corresponding to the geographic grid locations are extracted from the sparse binary mask and quantized and entropy-encoded to generate incremental patch bitstreams. This mechanism accurately locates areas with substandard reconstruction quality through physical error feedback and uses a preset physical tolerance threshold to filter out characteristic regions with significant physical meaning, such as vortices and fronts, for targeted compensation, avoiding excessive smoothing or distortion of key fluid dynamic structures due to global uniform quantization.

[0052] 4. This invention achieves differentiated error control for extreme outliers by constructing a two-layer coding architecture of basic compression plus incremental compensation, combined with geographic mask information preprocessing and joint decoding and denormalization processes. Specifically, in the preprocessing stage, a normalization mapping function is used to adapt to the dimensional differences of different physical quantities, and zeros are filled into invalid areas based on geographic mask information to ensure the data integrity of valid areas. In the joint decoding stage, when the compression result only includes the basic binary code stream, the reconstructed data is obtained by direct decoding and denormalization; when the compression result includes the incremental patch code stream and the basic binary code stream, the incremental patch code stream is parsed and decoded to recover the residual values ​​of the sparse binary mask area and the corresponding geographic grid positions, which are then fused into the basic meteorological reconstruction data and corrected before denormalization. In this process, extreme outliers such as typhoon center pressure and extreme rainfall values ​​are accurately marked by the sparse binary mask due to large physical errors, and targeted residual compensation is obtained through the incremental patch code stream. Their reconstruction error is controlled within a preset physical tolerance threshold range, thereby solving the problem of uncontrollable errors of extreme outliers in the prior art. Attached Figure Description

[0053] Figure 1 This is a schematic flowchart of a meteorological data compression and reconstruction method provided in an embodiment of the present invention;

[0054] Figure 2 This is a schematic diagram of the structure of the cascaded enhanced hybrid coding network provided in an embodiment of the present invention;

[0055] Figure 3 This is a schematic diagram of the structure of the residual dense group provided in an embodiment of the present invention. Detailed Implementation

[0056] The technical solution of the present invention will be described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the embodiments of the present invention and the specific features in the embodiments are detailed descriptions of the technical solution of the present invention, rather than limitations thereof. In the absence of conflict, the embodiments of the present invention and the technical features in the embodiments can be combined with each other.

[0057] The term "and / or" simply describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. Additionally, the character " / " generally indicates that the preceding and following related objects have an "or" relationship.

[0058] Example 1

[0059] like Figure 1 As shown in the figure, this embodiment introduces a meteorological data compression and reconstruction method, including:

[0060] Step 1: Obtain meteorological monitoring data.

[0061] This embodiment provides the original input data source for the present invention by acquiring meteorological monitoring data. The meteorological monitoring data covers extreme anomalies such as typhoon center pressure, extreme rainfall, and extreme temperature, as well as fluid dynamic characteristics such as vortices and fronts. This lays a complete data foundation for subsequent compression and reconstruction processing, ensuring that key meteorological information is not lost due to the data acquisition process.

[0062] Step 2: Preprocess the meteorological monitoring data based on the normalized mapping function and geographic mask information to obtain tensor data.

[0063] This embodiment uses a normalized mapping function to uniformly map different physical quantities in meteorological monitoring data, eliminating the dimensional differences between the physical quantities and improving the numerical stability of subsequent network training and compression processing. At the same time, zeros are filled into invalid regions of the mapped physical quantity data based on geographic mask information, which not only eliminates the invalid use of compression resources by geographic invalid regions, but also provides a standardized input structure for the cascaded enhanced hybrid coding network through a unified tensor data format, laying the foundation for efficient feature extraction.

[0064] Step 3: Based on the tensor data, feature extraction and probability distribution prediction are performed using a pre-trained cascaded enhanced hybrid coding network to obtain prediction results. Based on the prediction results, the data is compressed to obtain a basic binary code stream. At the encoding end, the basic binary code stream is locally reconstructed to obtain basic meteorological reconstruction data.

[0065] This embodiment utilizes a pre-trained cascaded enhanced hybrid coding network to process tensor data. The cascaded enhanced hybrid coding network extracts features and increases the number of channels through at least two cascaded residual dense groups in the front-end feature enhancement module. It then compresses the data through spatial size folding in the downsampling layer of the core compression module, performs entropy coding after statistical modeling and probability distribution prediction by the prior network, decompresses it through spatial size restoration in the upsampling layer, and finally reconstructs the data by reducing the number of channels through at least two cascaded residual dense groups in the back-end feature reconstruction module. This process enhances feature representation capabilities through cascaded residual dense groups and achieves efficient entropy coding through probabilistic modeling by the prior network. It completes local reconstruction while maintaining a high compression ratio, and the resulting basic meteorological reconstruction data provides quality assurance for the preservation of the main meteorological features.

[0066] Step 4: Compare the physical errors between meteorological monitoring data and basic meteorological reconstruction data:

[0067] If the physical error between the meteorological monitoring data and the basic meteorological reconstruction data is less than or equal to a preset threshold, then the basic binary code stream will be used as the compression result.

[0068] If the physical error between meteorological monitoring data and basic meteorological reconstruction data is greater than a preset threshold, a sparse binary mask is constructed based on the basic meteorological reconstruction data with the physical error greater than the preset threshold and the preset physical tolerance threshold.

[0069] This embodiment achieves adaptive judgment of compression quality by comparing the physical error between meteorological monitoring data and basic meteorological reconstruction data. When the physical error between the meteorological monitoring data and the basic meteorological reconstruction data is less than or equal to a preset threshold, the basic binary bitstream is directly used as the compression result, avoiding unnecessary incremental processing overhead and ensuring compression efficiency. When the physical error between the meteorological monitoring data and the basic meteorological reconstruction data is greater than the preset threshold, a sparse binary mask is constructed based on the basic meteorological reconstruction data with a physical error greater than the preset threshold and a preset physical tolerance threshold. Specifically, the physical residual matrix is ​​obtained by calculating the positional residual of the basic meteorological reconstruction data with a physical error greater than the preset threshold and the corresponding geographic grid positions of the meteorological monitoring data. After absolute value processing, it is compared with the preset physical tolerance threshold. Geographic grid positions with an absolute error value greater than the preset physical tolerance threshold are marked as 1, and positions with an absolute error value less than or equal to the preset physical tolerance threshold are marked as 0, thereby accurately identifying areas with substandard reconstruction quality and providing accurate spatial positioning information for subsequent directional compensation.

[0070] Step 5: Extract the residual values ​​of the mask region of the sparse binary mask and quantize them to generate incremental patch bitstream.

[0071] This embodiment extracts residual values ​​corresponding to geographic grid locations based on a sparse binary mask. After quantizing the extracted residual values, entropy encoding is performed on the quantized residual values ​​according to the geographic grid locations corresponding to the sparse binary mask, generating an incremental patch bitstream. This process only extracts and encodes residuals in key areas marked by the sparse binary mask, avoiding the bitrate overhead of global residual encoding. At the same time, quantization and entropy encoding further compress the incremental data volume, achieving accurate compensation for substandard reconstruction quality areas with minimal bitrate cost.

[0072] Step 6: Compile the incremental patch bitstream and the base binary bitstream as the compression result.

[0073] This embodiment outputs both the incremental patch bitstream and the basic binary bitstream as the compression result, forming a two-layer bitstream architecture of basic compression plus incremental compensation. The basic binary bitstream carries the main structure and key features of the meteorological data, ensuring the overall compression ratio; the incremental patch bitstream carries residual compensation information for sparse binary mask-marked regions, ensuring the reconstruction accuracy of hydrodynamic features such as vortices and fronts, as well as extreme anomalies such as typhoon center pressure and extreme rainfall values. The two bitstreams work together to achieve a balance between high compression ratio and high fidelity.

[0074] Step 7: Perform joint decoding and inverse normalization based on the compression results to obtain the reconstructed data.

[0075] This embodiment performs joint decoding and inverse normalization processing based on different compression results: when the compression result only includes the basic binary bitstream, the basic binary bitstream is parsed and decoded to recover the basic meteorological reconstruction data, and then inverse normalization is performed to obtain the reconstructed data; when the compression result includes the incremental patch bitstream and the basic binary bitstream, the basic binary bitstream is parsed and decoded to recover the basic meteorological reconstruction data, and the incremental patch bitstream is parsed and decoded to recover the residual values ​​of the sparse binary mask region and the corresponding geographic grid positions. The residual values ​​are then fused into the basic meteorological reconstruction data according to the corresponding geographic grid positions to obtain the corrected basic meteorological reconstruction data, and finally inverse normalization is performed to obtain the reconstructed data. This joint decoding process accurately integrates incremental compensation information into the basic reconstruction data, ensuring that the final reconstructed data maintains high fidelity of the main structure while ensuring that the reconstruction errors of key areas and extreme outliers are controllable. This comprehensively solves the problems of difficulty in achieving both high compression ratio and high fidelity, easy loss of fluid dynamic features, and uncontrollable errors of extreme outliers in existing technologies.

[0076] Example 2

[0077] Based on the same inventive concept as Embodiment 1, this embodiment describes the implementation steps of a meteorological data compression and reconstruction method, including:

[0078] Step 1: Obtain meteorological monitoring data.

[0079] Step 2: Preprocess the meteorological monitoring data based on the normalized mapping function and geographic mask information to obtain tensor data.

[0080] Step 2.1: Map the physical quantity data in the meteorological monitoring data using a normalized mapping function.

[0081] Step 2.2: Based on the geographic mask information, fill the invalid areas of the mapped physical quantity data with "0" to obtain tensor data.

[0082] Step 3: Based on the tensor data, feature extraction and probability distribution prediction are performed using a pre-trained cascaded enhanced hybrid coding network to obtain prediction results. Based on the prediction results, the data is compressed to obtain a basic binary code stream. At the encoding end, the basic binary code stream is locally reconstructed to obtain basic meteorological reconstruction data.

[0083] In this embodiment, the network structure of the cascaded enhanced hybrid coding network is as follows: Figure 2 As shown, it includes a front-end feature enhancement module, a core compression module, and a back-end feature reconstruction module; the front-end feature enhancement module includes at least two cascaded residual dense groups, wherein the structure of the residual dense group is as follows: Figure 3 As shown, the tensor data is used to extract features and increase the number of channels before outputting to the core compression module; the core compression module is used to compress the tensor data after increasing the number of channels to obtain a compressed tensor; to perform probability distribution prediction on the compressed tensor to obtain a prediction result, and to perform entropy encoding on the prediction result to obtain a basic binary code stream, and to perform entropy decoding on the basic binary code stream to obtain a reconstruction result; the reconstruction result is decompressed to obtain a decompressed tensor; the back-end feature reconstruction module includes at least two cascaded residual dense groups, used to reduce the number of channels of the decompressed tensor before reconstruction to obtain basic meteorological reconstruction data.

[0084] In this embodiment, the core compression module includes a downsampling layer, a priori network, and an upsampling layer connected in sequence. The downsampling layer includes at least two residual blocks connected in sequence, used to compress the tensor data after increasing the number of channels by folding the spatial dimensions to obtain a compressed tensor. The priori network is used to predict the probability distribution of the compressed tensor through statistical modeling to obtain a prediction result, and to perform entropy encoding based on the prediction result to obtain a basic binary bitstream, and to perform entropy decoding on the basic binary bitstream to obtain a reconstruction result. The upsampling layer includes at least two residual blocks connected in sequence, used to decompress the reconstruction result by restoring the spatial dimensions to obtain a decompressed tensor and output it to the backend feature reconstruction module.

[0085] In this embodiment, the method for training the cascaded enhanced hybrid coding network includes:

[0086] Historical meteorological monitoring data is acquired, and a sample set is constructed after preprocessing the historical meteorological monitoring data based on the normalized mapping function and geographic mask information. The sample set is divided into training samples and validation samples. A joint loss function is constructed, including bit rate loss and mean squared error loss. The training samples are input into the cascaded enhanced hybrid coding network, and the joint loss function is minimized through the backpropagation algorithm. The network parameters are iteratively updated until convergence, and the optimal cascaded enhanced hybrid coding network parameters are selected based on the validation samples to obtain the trained cascaded enhanced hybrid coding network.

[0087] In this embodiment, the joint loss function is expressed as:

[0088] ;

[0089] In the formula, Denotes the joint loss function. This represents the estimated bitrate value of the main latent feature portion corresponding to the basic binary bitstream. This represents the quantified principal latent features. This represents the estimated bitrate value of the extra-prior latent features. This represents the quantified prior latent features. Represents meteorological monitoring data With reconstruction data The mean square error between them These are Lagrange multipliers used to balance compressibility and physical accuracy.

[0090] Step 4: Compare the physical errors between meteorological monitoring data and basic meteorological reconstruction data:

[0091] Step 4.1: If the physical error between the meteorological monitoring data and the basic meteorological reconstruction data is less than or equal to a preset threshold, then the basic binary code stream is used as the compression result.

[0092] Step 4.2: If the physical error between the meteorological monitoring data and the basic meteorological reconstruction data is greater than a preset threshold, then a sparse binary mask is constructed based on the basic meteorological reconstruction data with the physical error greater than the preset threshold and the preset physical tolerance threshold.

[0093] Step 4.2.1: Perform position-by-position residual calculation on the basic meteorological reconstruction data with physical errors greater than a preset threshold and the geographic grid locations corresponding to the meteorological monitoring data to obtain the physical residual matrix.

[0094] Step 4.2.2: Perform absolute value processing on the physical residual matrix to obtain the absolute error distribution.

[0095] Step 4.2.3: Compare the error values ​​at each location in the absolute error distribution with the preset physical tolerance threshold:

[0096] When the absolute error value corresponding to the geographic grid location is greater than the preset physical tolerance threshold, the sparse binary mask corresponding to the geographic grid location is marked as "1".

[0097] When the absolute error value corresponding to the geographic grid location is less than or equal to the preset physical tolerance threshold, the sparse binary mask corresponding to the geographic grid location is marked as "0".

[0098] Step 5: Extract the residual values ​​of the mask region of the sparse binary mask and quantize them to generate incremental patch bitstream.

[0099] Step 5.1: Extract the residual values ​​corresponding to the geographic grid locations based on the sparse binary mask.

[0100] Step 5.2: Quantize the residual values ​​corresponding to the geographic grid locations to obtain quantized residual values.

[0101] Step 5.3: Perform entropy encoding on the quantized residual value according to the geographic grid position corresponding to the sparse binary mask to obtain the incremental patch bitstream.

[0102] Step 6: Compile the incremental patch bitstream and the base binary bitstream as the compression result.

[0103] Step 7: Perform joint decoding and inverse normalization based on the compression results to obtain the reconstructed data.

[0104] Step 7.1: When the compression result only includes the basic binary bitstream:

[0105] Step 7.1.1: Perform parsing and decoding on the basic binary code stream to restore the basic meteorological reconstruction data.

[0106] Step 7.1.2: Perform inverse normalization on the basic meteorological reconstruction data to obtain the reconstructed data.

[0107] Step 7.2: When the compression result includes both the incremental patch bitstream and the basic binary bitstream:

[0108] Step 7.2.1: Perform parsing and decoding on the basic binary code stream to restore the basic meteorological reconstruction data.

[0109] Step 7.2.2: Perform parsing and decoding on the incremental patch bitstream to recover the mask region residual values ​​and corresponding geographic grid positions of the sparse binary mask.

[0110] Step 7.2.3: The residual values ​​of the mask region of the sparse binary mask are fused into the basic meteorological reconstruction data according to the corresponding geographic grid location to obtain the corrected basic meteorological reconstruction data.

[0111] Step 7.2.4: Perform inverse normalization on the corrected basic meteorological reconstruction data to obtain the reconstruction data.

[0112] In this embodiment, the reconstructed data is represented as:

[0113] ;

[0114] In the formula, Indicates the first Reconstruction data for each channel, Indicates the first Normalized reconstructed data for each channel after residual correction Indicates the first The maximum value parameter used by each channel during normalization mapping. Indicates the first The minimum parameter used by each channel during normalization mapping. This represents element-wise multiplication. This represents a geographic mask matrix. In this embodiment, the geographic mask matrix is ​​a land mask matrix, where elements corresponding to land areas have a value of "1", and elements corresponding to non-land areas have a value of "0". This indicates that the reconstruction results for non-land areas are preserved while land areas are masked.

[0115] Example 3

[0116] Based on the same inventive concept as other embodiments, this embodiment describes a computer-readable storage medium having computer instructions stored thereon, which, when executed by a processor, implement the steps of the methods of Embodiment 1 or 2 described above.

[0117] Example 4

[0118] Based on the same inventive concept as other embodiments, this embodiment introduces a computer program product, including computer instructions that, when executed by a processor, implement the steps of the methods described in Embodiment 1 or 2 above.

[0119] In summary, this invention constructs a two-layer coding architecture of basic compression plus incremental compensation. In the basic compression stage, a cascaded enhanced hybrid coding network is used to extract features, predict probability distributions, and entropy encode preprocessed tensor data, generating a basic binary code stream and performing local reconstruction. When the physical error between the basic meteorological reconstruction data and the original meteorological monitoring data exceeds a preset threshold, a sparse binary mask is constructed based on a preset physical tolerance threshold. The residual values ​​of the mask region are extracted and quantized to generate an incremental patch code stream. The incremental patch code stream and the basic binary code stream are used together as the compression result for joint decoding and inverse normalization. The basic compression layer achieves efficient feature extraction and probabilistic modeling through a cascaded enhanced hybrid coding network, preserving the main structure and key features of the meteorological data while ensuring a high compression ratio. The incremental compensation layer, through a physical error feedback mechanism and a preset physical tolerance threshold, only performs targeted residual compensation for key areas where the error exceeds the limit, avoiding the loss of details and the spread of extreme outlier errors caused by global quantization. Therefore, this invention achieves efficient storage and transmission under high compression ratio, effectively preserves fluid dynamic characteristics such as vortices and fronts through differentiated error control strategies, and ensures that the reconstruction error of extreme anomalies such as typhoon center pressure and extreme rainfall values ​​is controllable. This solves the problems of difficulty in achieving both high compression ratio and high fidelity, easy loss of fluid dynamic characteristics, and uncontrollable errors of extreme anomalies in the prior art.

[0120] In the basic compression stage, this invention utilizes a cascaded enhanced hybrid coding network comprising a front-end feature enhancement module, a core compression module, and a back-end feature reconstruction module to process preprocessed tensor data. Specifically, the front-end feature enhancement module extracts features and increases the number of channels through at least two cascaded residual dense groups; the core compression module compresses data through spatial size folding of downsampling layers, performs entropy coding after statistical modeling and probability distribution prediction via a priori network; and the back-end feature reconstruction module reduces the number of channels before reconstruction using at least two cascaded residual dense groups. This network structure enhances feature representation capabilities through cascaded residual dense groups and combines probabilistic modeling and entropy coding optimization via a priori network, achieving high-quality local reconstruction while maintaining a high compression ratio. This overcomes the technical bottleneck of the trade-off between compression ratio and fidelity in traditional methods.

[0121] This invention addresses situations where the physical error between basic meteorological reconstruction data and original meteorological monitoring data exceeds a preset threshold. It performs location-by-location residual calculations on the geographic grid locations corresponding to the basic meteorological reconstruction data and meteorological monitoring data with physical errors exceeding the preset threshold. This yields a physical residual matrix, which is then processed to obtain the absolute error distribution. The error values ​​at each location are compared with a preset physical tolerance threshold to construct a sparse binary mask. Based on this, the residual values ​​corresponding to the geographic grid locations are extracted from the sparse binary mask and quantized and entropy-encoded to generate incremental patch bitstreams. This mechanism accurately locates areas with substandard reconstruction quality through physical error feedback and uses the preset physical tolerance threshold to filter out physically significant feature regions such as vortices and fronts for targeted compensation, avoiding excessive smoothing or distortion of key fluid dynamic structures due to global uniform quantization.

[0122] This invention achieves differentiated error control for extreme outliers by constructing a two-layer coding architecture of basic compression plus incremental compensation, combined with geographic mask information preprocessing and joint decoding and denormalization. Specifically, in the preprocessing stage, a normalization mapping function is used to adapt to the dimensional differences of different physical quantities, and zeros are filled into invalid areas based on geographic mask information to ensure the data integrity of valid areas. In the joint decoding stage, when the compression result only includes the basic binary bitstream, the reconstructed data is obtained by direct decoding and denormalization; when the compression result includes both incremental patch bitstream and basic binary bitstream, the incremental patch bitstream is parsed and decoded to recover the residual values ​​of the sparse binary mask region and the corresponding geographic grid positions, which are then fused into the basic meteorological reconstruction data and corrected before denormalization. In this process, extreme outliers such as typhoon center pressure and extreme rainfall values ​​are accurately marked by the sparse binary mask due to large physical errors, and targeted residual compensation is obtained through the incremental patch bitstream. Their reconstruction error is controlled within a preset physical tolerance threshold, thus solving the problem of uncontrollable errors of extreme outliers in existing technologies.

[0123] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0124] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0125] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0126] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0127] The embodiments of the present invention have been described above with reference to the accompanying drawings. However, the present invention is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of the present invention without departing from the spirit and scope of the claims. All of these forms are within the protection scope of the present invention.

Claims

1. A method for compressing and reconstructing meteorological data, characterized in that, include: Obtain meteorological monitoring data; The meteorological monitoring data is preprocessed based on the normalized mapping function and geographic mask information to obtain tensor data; Based on the tensor data, feature extraction and probability distribution prediction are performed using a pre-trained cascaded enhanced hybrid coding network to obtain prediction results. Based on the prediction results, compression is performed to obtain a basic binary code stream, and the basic binary code stream is reconstructed locally at the encoding end to obtain basic meteorological reconstruction data. If the physical error between the meteorological monitoring data and the basic meteorological reconstruction data is less than or equal to a preset threshold, then the basic binary code stream will be used as the compression result. If the physical error between meteorological monitoring data and basic meteorological reconstruction data is greater than a preset threshold, a sparse binary mask is constructed based on the basic meteorological reconstruction data with the physical error greater than the preset threshold and the preset physical tolerance threshold. Extract the residual values ​​of the mask region of the sparse binary mask and quantize them to generate an incremental patch bitstream; The incremental patch bitstream and the basic binary bitstream are used as the compression result; The reconstructed data is obtained by joint decoding and inverse normalization based on the compression results; The network structure of the cascaded enhanced hybrid coding network includes: The front-end feature enhancement module includes at least two cascaded residual dense groups, which are used to extract features from the tensor data and increase the number of channels before outputting them to the core compression module. The core compression module is used to compress tensor data after increasing the number of channels to obtain a compressed tensor; perform probability distribution prediction on the compressed tensor to obtain a prediction result, and perform entropy encoding on the prediction result to obtain a basic binary bitstream; perform entropy decoding on the basic binary bitstream to obtain a reconstruction result; and decompress the reconstruction result to obtain a decompressed tensor. The backend feature reconstruction module includes at least two cascaded residual dense groups, which are used to reduce the number of channels by decompressing the tensor and then reconstruct the basic meteorological reconstruction data.

2. The meteorological data compression and reconstruction method according to claim 1, characterized in that, The meteorological monitoring data is preprocessed based on a normalized mapping function and geographic masking information to obtain tensor data, including: The physical quantity data in the meteorological monitoring data are mapped using a normalized mapping function; Based on the geographic mask information, "0" is filled into the invalid areas of the mapped physical quantity data to obtain tensor data.

3. The meteorological data compression and reconstruction method according to claim 1, characterized in that, The core compression module includes a downsampling layer, a hyperprior network, and an upsampling layer connected in sequence. The downsampling layer includes at least two sequentially connected residual blocks, which are used to compress tensor data after increasing the number of channels by folding spatial dimensions to obtain a compressed tensor. The super-prior network is used to predict the probability distribution of the compressed tensor through statistical modeling to obtain the prediction result, and to perform entropy encoding based on the prediction result to obtain the basic binary code stream, and to perform entropy decoding on the basic binary code stream to obtain the reconstruction result. The upsampling layer includes at least two sequentially connected residual blocks, which are used to decompress the reconstruction results by restoring spatial dimensions to obtain a decompressed tensor and output it to the backend feature reconstruction module.

4. The meteorological data compression and reconstruction method according to claim 1, characterized in that, Based on the basic meteorological reconstruction data with physical errors exceeding a preset threshold and a preset physical tolerance threshold, a sparse binary mask is constructed, including: The physical residual matrix is ​​obtained by calculating the location-by-location residuals of the basic meteorological reconstruction data with physical errors exceeding a preset threshold and the corresponding geographic grid locations of the meteorological monitoring data. The absolute value of the physical residual matrix is ​​then processed to obtain the absolute error distribution. Finally, the location error values ​​in the absolute error distribution are compared with a preset physical tolerance threshold. When the absolute error value corresponding to the geographic grid location is greater than the preset physical tolerance threshold, the sparse binary mask corresponding to the geographic grid location is marked as "1". When the absolute error value corresponding to the geographic grid location is less than or equal to the preset physical tolerance threshold, the sparse binary mask corresponding to the geographic grid location is marked as "0".

5. The meteorological data compression and reconstruction method according to claim 4, characterized in that, Extracting the residual values ​​of the mask region from the sparse binary mask and quantizing them to generate an incremental patch bitstream includes: The residual values ​​corresponding to the geographic grid locations are extracted based on the sparse binary mask; the residual values ​​corresponding to the geographic grid locations are quantized to obtain quantized residual values; the quantized residual values ​​are entropy encoded based on the geographic grid locations corresponding to the sparse binary mask to obtain incremental patch bitstreams.

6. The meteorological data compression and reconstruction method according to claim 5, characterized in that, Based on the compression results, joint decoding and inverse normalization are performed to obtain reconstructed data, including: When the compression result includes only the basic binary bitstream: The basic binary code stream is parsed and decoded to recover the basic meteorological reconstruction data; the basic meteorological reconstruction data is then denormalized to obtain the reconstructed data. When the compression result includes both the incremental patch bitstream and the base binary bitstream: The basic binary code stream is parsed and decoded to recover the basic meteorological reconstruction data; the incremental patch code stream is parsed and decoded to recover the residual values ​​of the mask region of the sparse binary mask and the corresponding geographic grid positions. The residual values ​​of the mask region of the sparse binary mask are fused into the basic meteorological reconstruction data according to the corresponding geographic grid location to obtain the corrected basic meteorological reconstruction data; the corrected basic meteorological reconstruction data is then subjected to inverse normalization to obtain the reconstruction data.

7. The meteorological data compression and reconstruction method according to claim 6, characterized in that, The reconstructed data is represented as follows: ； In the formula, Indicates the first Reconstruction data for each channel, Indicates the first Normalized reconstructed data for each channel after residual correction Indicates the first The maximum value parameter used by each channel during normalization mapping. Indicates the first The minimum parameter used by each channel during normalization mapping. This represents element-wise multiplication. This represents the geographic mask matrix, specifically the land mask matrix. This indicates that the reconstruction results for non-land areas are preserved while land areas are masked.

8. The meteorological data compression and reconstruction method according to claim 1, characterized in that, A method for training the cascaded enhanced hybrid coding network includes: Historical meteorological monitoring data is acquired, and a sample set is constructed after preprocessing the historical meteorological monitoring data based on the normalized mapping function and geographic mask information. The sample set is then divided into training samples and validation samples. Construct a joint loss function, including bit rate loss and mean squared error loss; The training samples are input into the cascaded enhanced hybrid coding network, the joint loss function is minimized through the backpropagation algorithm, the network parameters are iteratively updated until convergence, and the optimal cascaded enhanced hybrid coding network parameters are selected based on the validation samples to obtain the trained cascaded enhanced hybrid coding network.

9. The meteorological data compression and reconstruction method according to claim 8, characterized in that, The joint loss function is expressed as: ； In the formula, Denotes the joint loss function. This represents the estimated bitrate value of the main latent feature portion corresponding to the basic binary bitstream. This represents the quantified principal latent features. This represents the estimated bitrate value of the extra-prior latent features. This represents the quantified prior latent features. Represents meteorological monitoring data With reconstruction data The mean square error between them These are Lagrange multipliers used to balance compressibility and physical accuracy.

Images