A three-dimensional physical field data compression method based on multi-scale vector quantization

Abstract

This invention belongs to the field of information technology and discloses a method for compressing three-dimensional physical field data based on multi-scale vector quantization, comprising the following steps: S1, constructing a multi-scale vector quantization variational autoencoder model; S2, inputting the original three-dimensional physical field data into a multi-layer fully convolutional encoder in the model to obtain downsampled feature maps; S3, performing multi-scale bilinear interpolation on the downsampled feature maps to generate a multi-scale feature map set composed of multiple feature maps of different scales; S4, performing vector quantization on each scale feature map in the multi-scale feature map set to obtain quantization mapping maps corresponding to each scale and using them as the compressed representation of the original three-dimensional physical field data; S5, introducing a residual learning mechanism in the multi-scale vector quantization process and aggregating multi-scale quantization features to obtain a fitted feature map; S6, inputting the fitted feature map into a multi-layer fully convolutional decoder in the model to reconstruct the target three-dimensional physical field data with the same spatial scale as the original data.

Description

Technical Field

[0001] This invention relates to the field of information technology, and specifically to a method for compressing three-dimensional physical field data based on multi-scale vector quantization. Background Technology

[0002] In the fields of scientific computing, engineering simulation, and industrial monitoring, scenarios such as marine climate monitoring, aircraft aerodynamic design, advanced energy equipment thermal management, and computational fluid dynamics (CFD) simulation continuously generate massive amounts of three-dimensional physical field data. These data encompass multi-physical quantities, multi-scale, and transient spatiotemporal distribution information, including temperature, pressure, velocity, density, and turbulent kinetic energy fields. This type of data is characterized by high dimensionality, large volume, strong spatiotemporal correlation, and sensitivity to physical features. High-resolution simulation and monitoring data often reach terabyte-scale, placing enormous cost pressure on data storage, transmission, reading, writing, and downstream analysis applications.

[0003] However, existing 3D physics data compression technologies cannot simultaneously achieve high compression ratios and effective preservation of key physical features when compressing 3D physics fields. Traditional lossless compression has too low a compression ratio; general lossy compression and existing deep learning compression methods destroy key physical information such as temperature gradients, pressure extrema, vorticity, and boundary layers, resulting in insufficient reconstruction accuracy; domain-specific methods and traditional dimensionality reduction methods are difficult to adapt to complex 3D physics fields with multi-scale, multi-variable, and non-stationary distributions, and cannot meet the needs of practical engineering applications. Summary of the Invention

[0004] The purpose of this invention is to provide a three-dimensional physical field data compression method based on multi-scale vector quantization to solve the problem that existing three-dimensional physical field data compression cannot simultaneously achieve high compression ratio and high fidelity of key physical features, and overcome the shortcomings of traditional compression methods such as low compression ratio, easy loss of physical features, and poor adaptability to complex multi-scale physical fields.

[0005] To achieve the above objectives, the present invention adopts the following technical solution: A three-dimensional physics data compression method based on multi-scale vector quantization includes the following steps: S1. Construct a multi-scale vector quantization variational autoencoder model and train the model. S2. Input the original three-dimensional physical field data into the multi-layer fully convolutional encoder in the model, perform feature extraction and downsampling processing, and obtain the downsampled feature map; S3. Perform multi-scale bilinear interpolation on the downsampled feature map to generate a multi-scale feature map set composed of multiple feature maps of different scales; S4. Perform vector quantization on each scale feature map in the multi-scale feature map set, map the feature vectors to the discrete codebook in the model, and obtain the quantized mapping map corresponding to each scale as the compressed representation of the original three-dimensional physical field data. S5. In the process of multi-scale vector quantization, a residual learning mechanism is introduced to optimize the current scale features based on the quantization results of the previous scale, and the multi-scale quantization features are aggregated to obtain a fitted feature map. S6. Input the fitted feature map into the multi-layer fully convolutional decoder in the model, and reconstruct the target three-dimensional physical field data with the same spatial scale as the original data by upsampling and feature fusion layer by layer.

[0006] Preferably, the input to the multilayer fully convolutional encoder in step S2 is the original three-dimensional physical field data. The output is a downsampled feature map. ,in, The original three-dimensional physical field data, These are the length, width, and height dimensions of the data, respectively. , The spatial dimension after downsampling. To quantify the number of spatial channels, This is a downsampled feature map.

[0007] Preferably, step S3 specifically involves: performing multi-scale bilinear interpolation on the downsampled feature map to generate multiple sets of feature maps with scales increasing from small to large, thereby forming a multi-scale feature map set for use by the quantizer. Feature maps at each scale The dimension is ,when At that time, the feature map scale is consistent with the downsampled feature map, that is... , ,in, It is a set of multi-scale feature maps. Indicates scale level. For the first Scale feature map The total number of levels across multiple scales. , For the first Spatial dimensions of scale feature maps , represents the spatial dimension of the downsampled feature map.

[0008] Preferably, the vector quantization in step S4 specifically involves mapping each feature vector in the feature map at each scale to the vector in the discrete codebook that is closest to that feature vector, thereby forming a quantization mapping map for the corresponding scale, and the vector quantization satisfies the following mapping formula: ,in, For the first Scale, coordinates The corresponding quantization mapping value, Vector quantization function For the first Coordinates in the scale feature map ( ) eigenvectors, These are the grid indices in the height and width directions of the feature map, respectively. To find the parameter corresponding to the minimum value, For inclusion A discrete codebook of learnable vectors, For the index in the discrete codebook Learnable vectors, This represents the index of the learnable vectors in the discrete codebook. This represents the total number of learnable vectors in the discrete codebook. This is an L2 norm operation.

[0009] Preferably, the residual learning mechanism in step S5 is specifically as follows: when the scale level When the value is greater than 1, based on the previous scale quantization mapping map and the discrete codebook, the previous scale quantization feature map is obtained by reverse mapping, i.e.: ,in, For the first Scale-quantized feature map For the first Scale quantization mapping, This is a function that retrieves and reconstructs the corresponding feature map from a discrete codebook based on the quantization index mapping value; The quantized feature map from the previous scale is upsampled to the same scale as the downsampled feature map. After processing by a convolutional network, the residual is calculated with the downsampled feature map, and this residual is used as the input for feature extraction at the current scale.

[0010] Preferably, the residual calculation formula is: ,in, For residual feature maps, For downsampled feature maps, For the first Layered convolutional optimization network, This is an upsampling operation.

[0011] Preferably, the fitted feature map in step S5 is obtained by weighted aggregation of multi-scale quantized feature maps after upsampling and convolution optimization, and the aggregation formula is as follows: ,in, To fit the feature map, The total number of levels across multiple scales. Indicates scale level. For the first Layered convolutional optimization network, For upsampling operation, For the first Scale-based quantized feature map, , represents the spatial dimension of the downsampled feature map.

[0012] Preferably, the model training employs a joint loss function to optimize parameters. This joint loss function includes reconstruction loss, codebook optimization loss, and commitment loss, with the specific formula as follows: ,in, For the joint loss function, The original three-dimensional physical field data, For the target three-dimensional physical field data, For gradient stopping operation, For the first Scale feature map For the first Scale-quantized feature map These are the weighting coefficients. The total number of levels across multiple scales. This is an L2 norm operation.

[0013] Preferably, the model training uses the AdamW optimizer with an initial learning rate of 1e-5, a batch size of 256, and a training period of 100 epochs.

[0014] Preferably, in step S4, the quantization mapping map is used as the compressed representation of the original three-dimensional physical field data, and the data compression effect is calculated based on the compressed representation. The compression ratio is calculated using the formula: compression ratio = 1 - total size of quantization mapping map / size of original three-dimensional physical field data. The data compression effect achieved by this compressed representation has a compression ratio of not less than 90%.

[0015] By adopting the above technical solution, the present invention has the following advantages compared with the prior art: 1. This invention provides a three-dimensional physical field data compression method based on multi-scale vector quantization. Through a variational autoencoder architecture that combines multi-scale vector quantization with residual learning, it fully extracts the global structure and local details of the physical field. While achieving a high compression ratio, it effectively reduces quantization errors and accurately preserves key physical features of the three-dimensional physical field, such as temperature gradient, pressure extrema, boundary layer, and turbulence structure, significantly improving the physical fidelity of the reconstructed data. Employing a fully convolutional isotropic processing method, it can uniformly adapt to three-dimensional physical field data of various types, resolutions, and boundary conditions, possessing stronger versatility and adaptability. It can meet the engineering needs of efficient storage, transmission, and application of massive three-dimensional physical field data in scenarios such as marine climate monitoring, aerospace aerodynamic design, energy equipment monitoring, and CFD numerical simulation.

[0016] 2. This invention provides a three-dimensional physical field data compression method based on multi-scale vector quantization. The multi-scale architecture and residual learning work together to preserve the cross-scale correlation information and high-frequency spatial details of the physical field, effectively reduce the accumulation of quantization error, improve the complementarity of features at different levels, and solve the problem that traditional deep learning compression is prone to losing edges, turbulence and small-scale structures. The reconstruction results are highly consistent with the original data in both coarse-grained overall distribution and fine-grained local features. Attached Figure Description

[0017] Figure 1 This is a network framework diagram of the present invention. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0019] Example Please refer to Figure 1 As shown, this invention discloses a three-dimensional physical field data compression method based on multi-scale vector quantization. This method is used for efficient compression and high-precision reconstruction of three-dimensional physical field data such as temperature field, velocity field, and pressure field. Specifically, it includes the following steps: S1. Construct a multi-scale vector quantization variational autoencoder model, which includes six core parts: a multi-layer fully convolutional encoder, a multi-scale bilinear interpolation module, a vector quantizer, a residual learning module, a discrete codebook, and a multi-layer fully convolutional decoder. The modules are connected in sequence to form an end-to-end compression and reconstruction architecture.

[0020] A joint loss function is used to train the model end-to-end to simultaneously optimize the learnable parameters of the encoder, decoder, quantizer and discrete codebook, enabling the model to have stable feature extraction, multi-scale quantization, residual correction and physical field reconstruction capabilities.

[0021] The optimizer used was AdamW, with an initial learning rate of 1e-5, a batch size of 256, and 100 training iterations until convergence.

[0022] The joint loss function includes reconstruction loss, codebook optimization loss, and commitment loss, and the specific formula is as follows: ,in, For the joint loss function, The original three-dimensional physical field data, For the target three-dimensional physical field data, For gradient stopping operation, For the first Scale feature map For the first Scale-quantized feature map These are the weighting coefficients. The total number of levels across multiple scales. This is an L2 norm operation.

[0023] S2, compress the original three-dimensional physical field data. The input is a multi-layer fully convolutional encoder, which performs deep feature extraction and spatial dimension compression through multi-layer convolution and downsampling operations, and outputs a downsampled feature map. This is used for subsequent quantization and compression.

[0024] in, The original three-dimensional physical field data, These are the length, width, and height dimensions of the data, respectively. , The spatial dimension after downsampling. To quantify the number of spatial channels, This is a downsampled feature map.

[0025] The downsampled feature map output by the multilayer fully convolutional encoder can preserve the global structure, gradient distribution, boundary features and local extrema of the three-dimensional physical field, and retain key physical information without loss even at high compression ratios.

[0026] S3. Perform multi-scale bilinear interpolation on the downsampled feature map to generate multiple sets of feature maps with scales increasing from small to large, thus forming a multi-scale feature map set for use by the quantizer. Feature maps at each scale The dimension is ,when At that time, the feature map scale is consistent with the downsampled feature map, that is... , ,in, It is a set of multi-scale feature maps. Indicates scale level. For the first Scale feature map The total number of levels across multiple scales. , For the first Spatial dimensions of scale feature maps , The spatial dimension of the downsampled feature map. =Height (number of rows) of the encoder output feature map = Width (number of columns) of the encoder output feature map.

[0027] S4. Perform vector quantization on each scale feature map in the multi-scale feature map set, map the feature vectors to the discrete codebook in the model, obtain the quantized mapping map corresponding to each scale, and use it as the compressed representation of the original three-dimensional physical field data.

[0028] In step S4, vector quantization specifically involves mapping each feature vector in the feature map at each scale to the vector in the discrete codebook that is closest to that feature vector, thereby forming a quantization mapping map for the corresponding scale. The vector quantization satisfies the following mapping formula: ,in, For the first Scale, coordinates The corresponding quantization mapping value, Vector quantization function For the first Coordinates in the scale feature map ( ) eigenvectors, These are the grid indices in the height and width directions of the feature map, respectively. To find the parameter corresponding to the minimum value, For inclusion A discrete codebook of learnable vectors, For the index in the discrete codebook Learnable vectors, This represents the index of the learnable vectors in the discrete codebook. This represents the total number of learnable vectors in the discrete codebook. This is an L2 norm operation.

[0029] In step S4, the quantization mapping map is used as the compressed representation of the original three-dimensional physical field data. The data compression effect is calculated based on the compressed representation. The compression ratio is calculated using the formula: Compression ratio = 1 − Total size of quantization mapping map / Size of original three-dimensional physical field data. The data compression effect achieved by this compressed representation has a compression ratio of not less than 90%, which meets the needs of massive three-dimensional physical field data storage and transmission in actual engineering.

[0030] S5. In the process of multi-scale vector quantization, a residual learning mechanism is introduced to optimize the current scale features based on the quantization results of the previous scale, and the multi-scale quantization features are aggregated to obtain a fitted feature map.

[0031] The residual learning mechanism described in step S5 is specifically as follows: when the scale level When the value is greater than 1, based on the previous scale quantization mapping map and the discrete codebook, the previous scale quantization feature map is obtained by reverse mapping, i.e.: ,in, For the first Scale-quantized feature map For the first Scale quantization mapping, This is a function that retrieves and reconstructs the corresponding feature map from a discrete codebook based on the quantization index mapping value; The quantized feature map from the previous scale is upsampled to the same scale as the downsampled feature map. After processing by a convolutional network, the residual is calculated with the downsampled feature map, and this residual is used as the input for feature extraction at the current scale.

[0032] The formula for calculating the residual is: ,in, For residual feature maps, For downsampled feature maps, For the first Layered convolutional optimization network, This is an upsampling operation.

[0033] The fitted feature map mentioned in step S5 is obtained by weighted aggregation of multi-scale quantized feature maps after upsampling and convolution optimization. The aggregation formula is as follows: ,in, To fit the feature map, The total number of levels across multiple scales. Indicates scale level. For the first Layered convolutional optimization network, For upsampling operation, For the first Scale-based quantized feature map, , represents the spatial dimension of the downsampled feature map.

[0034] S6. Input the fitted feature map into the multi-layer fully convolutional decoder in the model. Through layer-by-layer upsampling, feature fusion and deconvolution operations, the original spatial scale is gradually restored. Finally, the reconstructed three-dimensional physical field data with the same spatial scale as the original data is output, realizing high-fidelity physical field reconstruction, which can be directly used for subsequent scientific computing, simulation analysis and engineering monitoring.

[0035] In this embodiment, the ocean reanalysis data GLORYS12V1 (1 / 12° resolution) is used as an example. The original data with latitude and longitude of 3°×3° and a depth of 1000 meters (35 layers) is selected, and its size is: Original data size: 3×12×3×12×35, quantization mapping map The scale is set as follows: The size of the quantization mapping is: The compression rate is: .

[0036] The above are merely preferred embodiments of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A three-dimensional physics data compression method based on multi-scale vector quantization, characterized in that, Includes the following steps: S1. Construct a multi-scale vector quantization variational autoencoder model and train the model. S2. Input the original three-dimensional physical field data into the multi-layer fully convolutional encoder in the model, perform feature extraction and downsampling processing, and obtain the downsampled feature map; S3. Perform multi-scale bilinear interpolation on the downsampled feature map to generate multiple sets of feature maps that increase from small scale to large scale, thus forming a multi-scale feature map set. S4. Perform vector quantization on each scale feature map in the multi-scale feature map set, map the feature vectors to the discrete codebook in the model, and obtain the quantized mapping map corresponding to each scale as the compressed representation of the original three-dimensional physical field data. S5. In the process of multi-scale vector quantization, a residual learning mechanism is introduced. When the scale level k>1, the previous scale quantization feature map is obtained by reverse mapping the previous scale quantization mapping map with the discrete codebook. After upsampling, the residual is calculated with the downsampled feature map, and the residual is used as the input for the current scale feature extraction. A fitting feature map is obtained by aggregating multi-scale quantized features; S6. Input the fitted feature map into the multi-layer fully convolutional decoder in the model, and reconstruct the target three-dimensional physical field data with the same spatial scale as the original data by upsampling and feature fusion layer by layer.

2. The three-dimensional physics data compression method based on multi-scale vector quantization as described in claim 1, characterized in that: The input to the multilayer fully convolutional encoder in step S2 is the original three-dimensional physical field data. The output is a downsampled feature map. ,in, The original three-dimensional physical field data, These are the length, width, and height dimensions of the data, respectively. , The spatial dimension after downsampling. To quantify the number of spatial channels, This is a downsampled feature map.

3. The three-dimensional physics data compression method based on multi-scale vector quantization as described in claim 1, characterized in that, Step S3 specifically involves performing multi-scale bilinear interpolation on the downsampled feature map to generate multiple sets of feature maps with scales increasing from small to large, thus forming a multi-scale feature map set for use by the quantizer. Feature maps at each scale The dimension is ,when At that time, the feature map scale is consistent with the downsampled feature map, that is... , ,in, It is a set of multi-scale feature maps. Indicates scale level. For the first Scale feature map The total number of levels across multiple scales. , For the first Spatial dimensions of scale feature maps , represents the spatial dimension of the downsampled feature map.

4. The three-dimensional physics data compression method based on multi-scale vector quantization as described in claim 3, characterized in that, The vector quantization in step S4 specifically involves mapping each feature vector in the feature map at each scale to the vector in the discrete codebook that is closest to that feature vector, thereby forming a quantization mapping map for the corresponding scale. The vector quantization satisfies the following mapping formula: ,in, For the first Scale, coordinates The corresponding quantization mapping value, Vector quantization function For the first Coordinates in the scale feature map ( ) eigenvectors, These are the grid indices in the height and width directions of the feature map, respectively. To find the parameter corresponding to the minimum value, For inclusion A discrete codebook of learnable vectors, For the index in the discrete codebook Learnable vectors, This represents the index of the learnable vectors in the discrete codebook. This represents the total number of learnable vectors in the discrete codebook. This is an L2 norm operation.

5. The three-dimensional physics data compression method based on multi-scale vector quantization as described in claim 1, characterized in that, The residual learning mechanism described in step S5 is specifically as follows: when the scale level When the value is greater than 1, based on the previous scale quantization mapping map and the discrete codebook, the previous scale quantization feature map is obtained by reverse mapping, i.e.: ,in, For the first Scale-quantized feature map For the first Scale quantization mapping, This is a function that retrieves and reconstructs the corresponding feature map from a discrete codebook based on the quantization index mapping value; The quantized feature map from the previous scale is upsampled to the same scale as the downsampled feature map. After processing by a convolutional network, the residual is calculated with the downsampled feature map, and this residual is used as the input for feature extraction at the current scale.

6. The three-dimensional physics data compression method based on multi-scale vector quantization as described in claim 5, characterized in that, The formula for calculating the residual is: ,in, For residual feature maps, For downsampled feature maps, For the first Layered convolutional optimization network, This is an upsampling operation.

7. The three-dimensional physics data compression method based on multi-scale vector quantization as described in claim 1, characterized in that: The fitted feature map mentioned in step S5 is obtained by weighted aggregation of multi-scale quantized feature maps after upsampling and convolution optimization. The aggregation formula is as follows: ,in, To fit the feature map, The total number of levels across multiple scales. Indicates scale level. For the first Layered convolutional optimization network, For upsampling operation, For the first Scale-based quantized feature map, , represents the spatial dimension of the downsampled feature map.

8. The three-dimensional physics data compression method based on multi-scale vector quantization as described in claim 1, characterized in that: The model training employs a joint loss function to optimize parameters. This joint loss function includes reconstruction loss, codebook optimization loss, and commitment loss, and the specific formula is as follows: ,in, For the joint loss function, The original three-dimensional physical field data, For the target three-dimensional physical field data, For gradient stopping operation, For the first Scale feature map For the first Scale-quantized feature map These are the weighting coefficients. The total number of levels across multiple scales. This is an L2 norm operation.

9. The three-dimensional physics data compression method based on multi-scale vector quantization as described in claim 1, characterized in that, The model was trained using the AdamW optimizer with an initial learning rate of 1e-5, a batch size of 256, and a training period of 100 epochs.

10. The three-dimensional physics data compression method based on multi-scale vector quantization as described in claim 1, characterized in that, In step S4, the quantization mapping map is used as the compressed representation of the original three-dimensional physical field data. The data compression effect is calculated based on the compressed representation. The compression ratio is calculated using the formula: compression ratio = 1 - total size of quantization mapping map / size of original three-dimensional physical field data. The data compression effect achieved by this compressed representation has a compression ratio of not less than 90%.

Images