A method for compressing space-time scientific data, computer equipment and storage medium

Through the low-precision compression method and double-precision compression strategy based on neural networks, the feature encoder and probability decoder are used to process spatiotemporal scientific data, which solves the problems of low compression rate and poor accuracy flexibility in spatiotemporal scientific data, and realizes efficient data compression and Accurate data transmission.

CN118631259BActive Publication Date: 2024-11-01CHINA UNIV OF PETROLEUM (EAST CHINA)
View PDF 4 Cites 0 Cited by

Patent Information

Application Number
CN202411087366.7
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-08-09
Publication Date
2024-11-01
Estimated Expiration
2044-08-09

AI Technical Summary

Technical Problem

Existing spatiotemporal and spatial scientific data compression algorithms face the problems of low compression rate and poor accuracy and flexibility, especially in complex downstream tasks, where there are calculation and transmission redundancy problems.

Method used

Using a low-precision compression method based on neural networks, the time and spatial dimensions are divided by time and space, and the time and spatial features are extracted using feature encoder, and combined with probability decoder and arithmetic encoder to form a high-precision compressed bit stream to improve data compression rate and accuracy .

Benefits of technology

The ultra-high compression rate is achieved, and the low-precision compression rate exceeds 278% of the current optimal method. The poor flexibility of single-precision compression is overcome through the double-precision compression strategy, adapt to complex downstream tasks, and improve the turnover efficiency of the overall process.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118631259B_ABST
    Figure CN118631259B_ABST
Patent Text Reader

Abstract

The present invention belongs to the field of data compression technology, and discloses a spatiotemporal scientific data compression method, a computer device, and a storage medium. The spatiotemporal scientific data compression method proposed in the present invention is divided into a low-precision compression method and a double-precision compression method. Both data compression methods are spatiotemporal scientific data compression methods based on neural networks. The low-precision compression method can effectively utilize the spatiotemporal autocorrelation of spatiotemporal scientific data to achieve an ultra-high compression rate. Double-precision compression includes two parts: low-precision compression and high-precision compression. High-precision compression is an incremental compression of data residuals based on low-precision compression. It is a novel compression method based on probability modeling + entropy coding, and its compression rate can also reach the optimal level. The double-precision compression method can flexibly adapt to complex downstream tasks, which is conducive to improving the turnover efficiency of the overall process. The present invention is suitable for application scenarios of spatiotemporal scientific data transmission.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the field of data compression technology, and specifically relates to a spatiotemporal scientific data compression method, a computer device, and a storage medium, and is particularly suitable for spatiotemporal scientific data transmission application scenarios such as marine scientific data. Background Art

[0002] With the advancement of ocean observation, simulation and assimilation technology, people can obtain large-scale, long-period, high-resolution spatiotemporal scientific data (such as ocean spatiotemporal scientific data). The convenience provided by abundant data to scientific research also creates tremendous pressure on data transmission. For example, the bandwidth of marine data transmission is much narrower than that on land, and the traditional VHF segment marine communication provides a maximum data transmission rate of only 100 kbit / s, which cannot transmit TB-level marine scientific data in a timely manner.

[0003] The efficiency of data transmission is mainly determined by hardware equipment and software compression algorithms. It is very difficult to improve hardware equipment, so efficient source compression algorithms have become the key to solving the problem. Unlike discrete image and video data, spatiotemporal scientific data are often continuous floating-point arrays with characteristics such as large data volume, high dynamic range, and high spatiotemporal redundancy. Therefore, how to compress these data is a key research direction in the field of data science. The latest research on spatiotemporal scientific data compression mainly includes compression methods based on prediction, compression methods based on transformation, and compression methods based on dimensionality reduction. However, these traditional marine science data compression methods rely on manually designed feature extraction processes and cannot effectively handle spatiotemporal redundant information.

[0004] The compression method based on neural network uses the powerful representation ability of neural network to extract data features, which can effectively identify and remove redundant information in time and space and has a higher compression rate. However, the current method only supports single-precision compression, and repeated compression content will cause redundancy in calculation and transmission when facing complex downstream tasks. For example, in the early stages of seabed mineral resource exploration, low-precision seabed topography and geological structure data are needed to conduct large-scale searches to identify possible mineral resource distribution areas; once an area with potential development value is determined, high-precision geological exploration technology can be used for detailed resource assessment, and the use of double-precision compression strategies can well support such complex compression tasks.

[0005] In summary, the current spatiotemporal science data compression algorithms face technical problems such as low compression rate and poor accuracy and flexibility.

[0006] Glossary:

[0007] Low-precision compression and high-precision compression are two commonly used terms in the field of data compression. In the present invention, the normalized mean absolute error (NMAE) of low-precision compression is set to be less than 1%; the range-based maximum error limit (REL) of high-precision compression is set to be 0.01%. Summary of the invention

[0008] The purpose of the present invention is to propose a spatiotemporal scientific data compression method, which is a low-precision spatiotemporal scientific data compression method based on a neural network and a low-precision compression strategy, and is beneficial to improving the data compression rate.

[0009] In order to achieve the above object, the present invention adopts the following technical scheme:

[0010] A method for compressing spatiotemporal scientific data comprises the following steps:

[0011] Step 1. Divide the time and space dimensions of the spatiotemporal scientific data to be compressed;

[0012] Step 2. Use the feature encoder to process the divided data, extract the temporal and spatial features, and directly store them as a low-precision compressed bit stream.

[0013] Preferably, in step 2, after the divided data is processed by the feature encoder to extract the temporal features and the spatial features, the estimated data is further reconstructed based on the extracted temporal features and the spatial features by using a reconstruction decoder.

[0014] In addition, after step 2, it further includes:

[0015] Step 3. Define the difference between the divided data and the reconstructed estimated data as the data residual;

[0016] The extracted temporal features and spatial features are input into a probability decoder to obtain parameters of a mixed Gaussian distribution; according to the mixed Gaussian distribution, the data residual is converted into a compressed residual bit stream using an arithmetic encoder;

[0017] Finally, the low-precision compressed bit stream is combined with the compressed residual bit stream to form a high-precision compressed bit stream.

[0018] On the basis of the low-precision spatiotemporal scientific data compression method, the double-precision spatiotemporal scientific data compression method based on neural network and double-precision compression strategy is further proposed, which effectively improves the data compression rate and compression accuracy.

[0019] Preferably, in step 1, based on a meta-inspired time partitioning algorithm, the time series of spatiotemporal scientific data is divided into a number of intervals according to the similarity of time features, and each interval is expressed using a time feature;

[0020] After dividing the spatiotemporal scientific data by time dimension, the spatial dimension is then divided. The spatial partitioning strategy is to cut the data after the time dimension division into slices of the same size, and at the same time perform reflection filling on the edges of the slices.

[0021] Preferably, in step 2, the feature encoder includes a feature extraction module and vector quantization module ;

[0022] Given a partitioned data , which is represented as a time series ,in , , It represents the width and height of the data after division, and t represents the time length of the data after division;

[0023] For each two-dimensional spatial data at a time point , first use Extract spatial features. In the feature space, each point in the space is expressed by a feature vector, and we get:

[0024] ;

[0025] in, Indicates that it corresponds to The characteristic vector of Indicates the dimension of each vector;

[0026] Then through Features Perform vector quantization to obtain discrete spatial features and time characteristics ;

[0027] Finally, the discrete spatial features and time characteristics Stored as low-precision compressed bitstream .

[0028] Preferably, in step 2, the process of reconstructing the estimated data by the reconstruction decoder is as follows:

[0029] First, the bit stream is compressed from the low-precision Get discrete spatial features from and time characteristics , and then convert them into ;

[0030] Will Input into the reconstruction decoder to obtain the reconstructed estimated data .

[0031] Preferably, in step 3, it is assumed that the data residual obey The Gaussian mixture distribution GMM is composed of components, and then the parameters of the Gaussian mixture distribution GMM, that is, the mean, are output through the probability decoder PD. , Standard Deviation , weight ;

[0032] For a residual data , which takes the value The probability density of The component GMM is expressed as follows:

[0033] ;

[0034] in , , Respectively represent The mean, standard deviation, and weight of each component;

[0035] Assume that the quantization level is , all quantified data are , , , ;

[0036] For each quantified data , the probability density is:

[0037] ;

[0038] in and They are [ q i − 1 , q i ] and [ q i , q i + 1 ] The midpoint of

[0039] Assumptions and The cumulative distribution functions are and ,in It is known that according to the relationship between Gaussian density function and distribution function To obtain, to use calculate , the expression formula is:

[0040] ;

[0041] The probability decoder is used to determine the data characteristics at a given time point. Estimate the parameters of GMM under the condition of .

[0042] Preferably, in step 3, in order to accurately control the maximum error and improve the compression rate, quantization and entropy coding are performed only on the data residuals in the interval [-1, 1], and the data residuals outside [-1, 1] are stored separately and completely.

[0043] Preferably, in step 1, the process of dividing the spatiotemporal scientific data according to the similarity of the time characteristics is as follows:

[0044] Given a spatiotemporal scientific dataset ;

[0045] in Indicates Data at time points, i=1,2,…,T;

[0046] The spatiotemporal scientific dataset is divided into time segments, each of which contains time points, namely:

[0047] ;

[0048] set up Indicates time segment;

[0049] in, , Indicates A time series of time slices;

[0050] The optimization objective of the meta-inspiration-based time partitioning algorithm is expressed as:

[0051] ;

[0052] For each fragment , its internal temporal autocorrelation Use the following formula to measure:

[0053] ;

[0054] ;

[0055] in is the Pearson product-power correlation coefficient, It's a fragment Internal Sample and The covariance matrix between is the time series number of the fragment, and Respectively and between, and The covariance matrix between ;

[0056] Assume that two different time series are and , and The correlation between As shown below:

[0057] ;

[0058] Among them, here The formula is the same as above The formula is the same, except In For fragment Internal Sample With fragment Internal Sample The covariance matrix between , Respectively represent the time series of the mth and nth segments;

[0059] Set the minimum time series length threshold , that is, specify any time series ;

[0060] in , , is a hyperparameter used to balance different principles;

[0061] Based on the optimization problem of the optimization target, the meta-heuristic algorithm effectively explores and finds the approximate optimal solution in the search space, that is, obtains unequally spaced time partition data subsets and completes the time dimension partition.

[0062] In addition, based on the above-mentioned spatiotemporal scientific data compression method, the present invention also proposes a computer device, which includes a memory and one or more processors. An executable code is stored in the memory. When the processor executes the executable code, it is used to implement the above-mentioned spatiotemporal scientific data compression method.

[0063] In addition, based on the above-mentioned spatiotemporal scientific data compression method, the present invention also proposes a computer-readable storage medium on which a program is stored. When the program is executed by a processor, it is used to implement the above-mentioned spatiotemporal scientific data compression method.

[0064] The present invention has the following advantages:

[0065] As described above, the present invention relates to a spatiotemporal scientific data compression method, a computer device, and a storage medium. Among them, the spatiotemporal scientific data compression method in the present invention is divided into a low-precision compression method and a double-precision compression method. Both of the above data compression methods are spatiotemporal scientific data compression methods based on neural networks. The low-precision compression method can effectively utilize the spatiotemporal autocorrelation of the data to achieve an ultra-high compression rate, and its compression rate exceeds 278% of the current optimal method. The double-precision compression method includes two parts: low-precision compression and high-precision compression. High-precision compression is an incremental compression of data residuals based on low-precision compression. It is a novel compression method based on probability modeling + entropy coding. The double-precision compression method proposed in the present invention overcomes the problem of poor flexibility of existing single-precision compression methods in the face of complex downstream tasks, can flexibly adapt to complex downstream tasks, reduce calculation and transmission redundancy in compression tasks of different precisions, and improve the turnover efficiency of the overall process. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] Figure 1 Schematic diagram of the flow of the data compression method in Embodiment 1 of the present invention;

[0067] Figure 2 A network structure diagram of a low-precision compression and decompression model in an embodiment of the present invention;

[0068] Figure 3 It is a network structure diagram of a feature encoder FE in an embodiment of the present invention;

[0069] Figure 4 A network structure diagram of a reconstruction decoder RD in an embodiment of the present invention;

[0070] Figure 5 A network structure diagram of a probability decoder PD in an embodiment of the present invention;

[0071] Figure 6 This is a network structure diagram of a single residual block Res Block in an embodiment of the present invention;

[0072] Figure 7 This is a network structure diagram of Tail Block in an embodiment of the present invention;

[0073] Figure 8 Schematic diagram of the flow of the data compression method in Embodiment 3 of the present invention;

[0074] Fig. 9 A schematic diagram comparing a low-precision data compression method in an embodiment of the present invention and a traditional data compression method;

[0075] Fig.10 Schematic diagram for comparing the high-precision data compression method in an embodiment of the present invention with the traditional data compression method. DETAILED DESCRIPTION

[0076] The present invention is further described in detail below with reference to the accompanying drawings and specific embodiments:

[0077] Example 1

[0078] This embodiment 1 proposes a double-precision spatiotemporal scientific data compression method based on a neural network and a double-precision compression strategy to improve the compression rate and compression accuracy of the data.

[0079] like Figure 1 As shown, the double-precision spatiotemporal scientific data compression method in this embodiment includes the following steps:

[0080] Step 1. Divide the spatiotemporal scientific data to be compressed into time and space dimensions.

[0081] Firstly, based on the meta-inspired time partitioning algorithm, the time series of spatiotemporal scientific data are divided into several intervals according to the similarity of temporal features, and each interval is expressed using a temporal feature.

[0082] First, space-time scientific data The time dimension is divided, and the result after the time dimension division is unequal. Its purpose is to divide the data with strong temporal and spatial correlation together to achieve a higher compression rate. This process is achieved by meta-heuristic algorithm.

[0083] in, , Represents height, width, and time length respectively.

[0084] Spatiotemporal scientific data in real scenarios (such as marine scientific data) often have long time series and large changes in temporal characteristics. It is inappropriate to express them using the same temporal characteristics.

[0085] Therefore, the present invention proposes a meta-inspiration-based time partitioning algorithm, which can divide the long time series of marine scientific data in real scenes into several intervals according to the similarity of time characteristics.

[0086] In this way, each interval is expressed using a time feature, which ensures the overall reconstruction accuracy while improving the compression rate.

[0087] Specifically, given a spatiotemporal scientific dataset ;

[0088] in Indicates The data of time points, i=1,2,…,T.

[0089] The spatiotemporal scientific dataset is divided into time segments, each of which contains time points, namely:

[0090] .

[0091] set up Indicates time segment;

[0092] in, ; Indicates A time series of time slices.

[0093] The present invention is expected to meet the following principles:

[0094] 1. Time series with strong temporal autocorrelation are grouped into the same region.

[0095] For each fragment , its internal temporal autocorrelation is as strong as possible, and this process is measured using formulas (1) and (2):

[0096] (1)

[0097] (2)

[0098] in is the Pearson product-power correlation coefficient, It's a fragment Internal Sample and The covariance matrix between is the time series number of the fragment, and Respectively and between, and The covariance matrix between .

[0099] 2. Time series with weak temporal autocorrelation are divided into different zones.

[0100] The correlation between different series should be as weak as possible.

[0101] Assume that two different time series are and , then the correlation calculation formula between them is shown in formula (3):

[0102] (3)

[0103] Among them, here The formula is the same as above The formula is the same, except In It's a fragment Internal Sample With fragment Internal Sample The covariance matrix between , Represent the time series of the m-th and n-th segments respectively.

[0104] 3. Too dense time partitioning will cause excessive computational burden and affect the compression rate.

[0105] Therefore, the present invention sets a minimum time series length threshold , that is, specify any time series .

[0106] Based on the above principles, the present invention expresses the optimization objective as shown in formula (4).

[0107] (4)

[0108] in, is a hyperparameter used to balance different principles.

[0109] This is an optimization problem involving a large number of possible solutions. Genetic algorithms are usually able to quickly and effectively explore and find approximately optimal solutions in the search space for optimization problems, that is, to obtain unequally spaced time-divided data subsets.

[0110] Of course, the meta-heuristic algorithm in this embodiment is not limited to the above-mentioned genetic algorithm. For example, meta-heuristic algorithms such as particle swarm optimization and simulated annealing may also be used. This is only exemplary and is not intended to limit the present invention.

[0111] After the time dimension is divided for the spatiotemporal scientific data set, the space dimension is then divided. The space partitioning strategy of the present invention is to cut it into a number of slices of the same size and perform reflection filling on the edges of the slices.

[0112] Specifically, reflective fill mirrors the bounding elements of the input data to create the fill area.

[0113] For example, for a two-dimensional data, if a reflection fill is used, the pixel values ​​at the top and bottom edges of the image will be mirrored to the fill area, and the pixel values ​​at the left and right edges will also be mirrored to the fill area.

[0114] The benefits of doing this are as follows:

[0115] 1). Reduce the need for video memory and the amount of forward and backward propagation calculations, which helps to increase computing speed.

[0116] 2). It can play the role of data enhancement, increase the number of training samples, and help improve the accuracy of model reconstruction.

[0117] 3). In practical applications, multiple slices can be processed at the same time, which is conducive to improving the degree of parallelism.

[0118] 4). Reflection filling can maintain the continuity of image edges, which is beneficial to improving the effect of edge reconstruction.

[0119] For example, assuming the spatial size of the original data is (2400,3600), it is divided into A slice of size (240,240) is created, and then the edges (including top, bottom, left, and right) are padded with 8 units, and the final data is (150, 256,256).

[0120] Step 2. Build a low-precision compression and decompression model including a feature encoder (FE) and a reconstruction decoder (RD) to perform compression on the divided data. Performs low-precision compression.

[0121] in, , , , They respectively represent the height, width, and time length of the divided data.

[0122] The goal of low-precision compression is to maximize the compression rate while maintaining a visually acceptable error range.

[0123] FE and RD are essentially discrete encoders based on CNN.

[0124] Use the feature encoder FE to process the divided data and extract the time features and spatial characteristics , these features are reconstructible, and RD can reconstruct the estimated data based on the extracted temporal and spatial features .

[0125] In addition, due to the time characteristics and spatial characteristics are compact and discrete features that are stored directly as low-precision compressed bitstreams ,The compression ratio of low-precision compression can exceed 50, and the normalized mean absolute error is less than 1%.

[0126] The low-precision compression process is completed by the feature encoder FE. The network structure of FE is as follows: Figure 2As shown. FE consists of two modules, namely the feature extraction (extractor) module and vector quantization modules .

[0127] like Figure 3 As shown, Includes multiple (e.g. Figure 3 4) convolutional layers Conv and a residual stack, the residual stack consists of n res It is composed of residual blocks Res Block. The process of extracting features is as follows:

[0128] First, it passes through multiple convolutional layers Conv to gradually downsample the input and increase the channel depth to capture more complex features. The total downsampling factor is 4. Next, it passes through a residual stack, which consists of multiple residual blocks to prevent the gradient from disappearing. Finally, it passes through another convolutional layer Conv to change the number of channels to 1.

[0129] The network structure of the residual block Res Block is as follows Figure 6 shown.

[0130] ‌A single residual block Res Block consists of two LeakyReLU activation functions and two convolutional layers Conv.

[0131] Feature extraction module The design of is simple and effective. Compared with the Transformer architecture, it has lighter parameters, is easier to train, and can adapt to inputs of different resolutions.

[0132] like Figure 4 As shown in Figure 1, the structure of RD is almost a mirror image of FE, which includes a convolution layer Conv, a residual stack, and two transposed convolution layers ConvTranspose. The RD processing flow is as follows:

[0133] It first passes through a convolutional layer Conv, which increases the number of channels to 3, and then uses the residual stack to optimize the features in preparation for upsampling. It then passes through two transposed convolutional layers ConvTranspose to gradually enlarge the feature map while reducing the channel depth. The total upsampling factor is also 4, reconstructing the data to its original dimension.

[0134] Among them, the residual stack is also composed of n res The network structure is composed of residual blocks Res Block. Figure 6 shown.

[0135] Given a partitioned data , which is represented as a time series .in , , It represents the width and height of the divided data, and t represents the time length of the divided data.

[0136] For each two-dimensional spatial data at a time point , first use Extract spatial features. In the feature space, each point in the space is expressed by a feature vector, and we get:

[0137] ;

[0138] in, Indicates that it corresponds to The characteristic vector of To represent the dimension of each vector, use The process of extracting spatial features is expressed by formula (5):

[0139] (5)

[0140] Then through the vector quantization module Features Perform vector quantization to obtain discrete spatial features and time characteristics ,use The process of vector quantization is expressed by formula (6):

[0141] (6)

[0142] Specifically, vector quantization consists of two structures: codebook cb and index id, , .in represents the number of vectors in the codebook, where and These are all hyperparameters.

[0143] cb is a set of learnable discrete vectors. After training, each vector becomes an embedding space The reference point in All vectors in can be represented by vectors in cb.

[0144] During training, each spatial data at each time step is input into the network as a batch.

[0145] For each feature at a specific time point , get the corresponding , a total of All time points correspond to the same cb. Therefore, and Represent the temporal and spatial features respectively, namely , .

[0146] Finally, the discrete spatial features and time characteristics Stored as low-precision compressed bitstream .

[0147] Assumptions Each vector in bit storage (using float32), after vector quantization, only bit. Because The value of is usually small, so this step is the key to improving the compression ratio.

[0148] The low-precision decompression process is completed by the reconstruction decoder RD, and the process is as follows:

[0149] First from Get discrete spatial features from and time characteristics , and then convert them into .

[0150] Will Input into the reconstruction decoder to obtain the reconstructed estimated data .

[0151] The process of reconstructing the estimated data using the reconstruction decoder RD is expressed by formula (7):

[0152] (7)

[0153] In this embodiment, FE and RD are trained together, using L2 loss , as shown in formula (8):

[0154] (8)

[0155] Assuming that the original data is stored in float32, the original data storage size is byte.

[0156] After low-precision compression, the storage space is reduced by , and network weights.

[0157] and The sizes are and Since the network weights act as compressors and decompressors, they are reusable and their storage space is much smaller than and Very small, so negligible.

[0158] Finally, in theory, for a time series after partitioning , its lossy compression ratio As shown in formula (9):

[0159] (9)

[0160] in, Indicates the original data size, Indicates the compressed data size.

[0161] In actual situations, in order to prevent edge distortion after reconstruction, the slices need to be filled. In this case, the actual compression rate obtained will be slightly lower than the above theoretical value.

[0162] Step 3. Build a high-precision compression model based on a probabilistic decoder, such as Figure 5 shown.

[0163] The divided data and reconstruct estimated data The difference between them is defined as the data residual The present invention uses a probability modeling + entropy coding (ie, arithmetic coding) method to perform incremental high-precision compression on data residuals.

[0164] Specifically, we first assume that the data residuals obey The components are mixed with Gaussian distribution and modeled using a probabilistic decoder (Probdecoder Decoder, PD, which is a network composed of CNN and fully connected layers).

[0165] The extracted time features and spatial characteristics Input probability decoder PD to obtain the parameters of Gaussian Mixture Models (GMM) ;in represent mean, standard deviation, and weight respectively.

[0166] like Figure 5 The network structure of the probability decoder PD is shown for a given The parameters of GMM are estimated under the condition of ), the first half of the PD network is the same as the reconstruction decoder RD, the difference lies in the tail.

[0167] The tail of the reconstruction decoder RD consists of three parallel tail blocks. Each sequence has a similar structure. The three parallel tail blocks are responsible for generating different parameter values, which correspond to the mean, standard deviation and weight respectively.

[0168] The structure of Tail Block is as follows Figure 7 As shown, it consists of a ReLU activation function, a convolutional layer Conv, and a linear layer Linear.

[0169] Each tail uses a convolutional layer to reduce the number of channels to 1, and then applies a linear transformation to the output size To ensure that the parameters are valid, use the function Make sure the standard deviation is positive and normalize the weights so that they sum to 1 via the softmax function.

[0170] Because the data residual is continuous, and entropy coding can only accept discrete data, the residual needs to be quantized first.

[0171] In this embodiment, uniform quantization is preferably used, and the specific process is as follows:

[0172] First, the data residual r is divided into evenly spaced intervals, H is the quantization level. The length of each interval is ,in and They represent the maximum and minimum values ​​of the residual r respectively.

[0173] Then each residual value Map to the closest discrete value , expressed as formula (10):

[0174] (10)

[0175] in, Represents a rounding function that rounds a number to the nearest integer.

[0176] It should be noted that the above quantization methods are only exemplary. In addition to uniform quantization, there is also non-uniform quantization. Non-uniform quantization uses different quantization intervals for different intervals according to the characteristics of the signal.

[0177] Compared with uniform quantization, the advantage of non-uniform quantization is that it can better preserve the details and dynamic range of the signal, thereby reducing the quantization error. However, the disadvantage of non-uniform quantization is that it will increase the information entropy, resulting in a decrease in entropy coding performance.

[0178] The process of inverse quantization is to convert the quantized data Convert back to the original data estimate , expressed by formula (11):

[0179] (11)

[0180] Statistics show that 99% of the data residuals are concentrated in the interval [-1,1], and the data residuals outside [-1,1] account for only 1%.

[0181] Therefore, in order to accurately control the maximum error and improve the compression rate as much as possible, the present invention only performs quantization and entropy coding on the data residuals in the interval [-1, 1], and stores the data residuals outside [-1, 1] separately and completely.

[0182] In the present invention, it is set , , , at this time The maximum error caused by quantization occurs when the input value is exactly in the middle of two adjacent discrete values. In this case, the input value will be rounded up or down to the nearest discrete value, resulting in a quantization error equal to half the quantization step size.

[0183] Therefore, the maximum error caused by quantization is shown in formula (12):

[0184] (12)

[0185] Unlike the probability model PixelCNN that directly outputs all possible values ​​of each data point, the present invention first assumes obey The Gaussian mixture distribution of the components is then used to output the distribution parameters through the neural network PD to greatly improve the performance.

[0186] For example, assuming that each data point has 1001 quantization levels, using the first method (i.e., the probabilistic model PixelCNN method), the output size of the network is .

[0187] In the method of the present invention, each data point output parameters (each component requires 3 parameters: weight , mean , standard deviation ), the network output size only needs , when n=2, it is less than 1% of the former.

[0188] It is not difficult to find from the above comparison that the present invention reduces the number of parameters calculated and improves the calculation speed.

[0189] For a residual data , which takes the value The probability density of The component GMM is expressed as shown in formula (13):

[0190] (13)

[0191] in , , Respectively represent The mean, standard deviation, and weight of the components.

[0192] The above probability density is continuous. Since the residual after quantization is discrete, it needs to be discretized.

[0193] Assume that the quantization level is , , , , It is all quantified data.

[0194] For each quantified data , the probability density is:

[0195] .

[0196] in , They are [ q i − 1 , q i ] and [ q i , q i + 1 ] The midpoint of , .

[0197] Assumptions and The cumulative distribution functions are and ,in It is known that according to the relationship between Gaussian density function and distribution function To obtain, to use calculate , expressed as formula (14):

[0198] (14)

[0199] The present invention uses maximum likelihood estimation to optimize the probabilistic decoder PD.

[0200] Given the data characteristics at a certain point in time , probability decoder Output the probability of all data points at that time point; consider each point as a sample, then the likelihood function formula is (15):

[0201] (15)

[0202] in Shidi The data value is The probability of By The value with the highest probability among all possible values ​​of the points is determined, that is, .

[0203] The optimization goal is to maximize .

[0204] However, because directly calculating the product may involve very small probability values, underflow problems may easily occur. Therefore, in actual calculations, it is usually converted to minimizing the log-likelihood function, and the final loss function is as shown in formula (16):

[0205] (16)

[0206] The calculation of the probability loss in Equation (16) requires solving the probability density, which in turn requires evaluating the cumulative distribution function (CDF) for each possible quantized value of the data point. This becomes computationally very intensive when dealing with high-resolution data.

[0207] For example, a 512×512 data with each data point quantized to 1001 levels will require 512×512×1002 CDF calculations, which is computationally expensive and time-consuming.

[0208] To solve this challenge, a probability sampling calculation method can be used. This method can quickly solve the approximate probability density. The main idea is to group first and then find the average. Specifically:

[0209] In order to approximately solve the approximate value of the true probability density, first Adjacent values ​​form a group;

[0210] Subsequently, each group is considered as a quantization level to calculate the CDF, and finally the probability density of each group is Divide equally levels, so that the value of each level is This reduces the number of CDF calculations to the original .

[0211] Experiments show that although this approach has a certain tradeoff in accuracy, it significantly speeds up the calculation process.

[0212] According to the mixed Gaussian distribution, Converted to a compressed residual bit stream using an arithmetic encoder ; Finally, the low-precision compressed bit stream The residual bit stream after compression Splicing to form a high-precision compressed bit stream .

[0213] It should be noted here that the arithmetic encoder, also known as entropy coding, is a classic compression method and will not be described in detail here.

[0214] The compression ratio of the above high-precision compression can reach more than 5, and the maximum error limit based on the range is 0.01%.

[0215] In addition, this embodiment also proposes a high-precision compression decompression method for the above-mentioned high-precision compression method, wherein the high-precision compression decompression process is the inverse process of the high-precision compression, which is specifically as follows:

[0216] First take out the high-precision compressed bit stream , which includes and ,Will Input PD to get probability distribution .

[0217] Will and Input the arithmetic encoder AC to get the quantized residual , and then Perform inverse quantization to obtain an approximate residual , the approximate residual Reconstructing estimated data with low-precision compression By adding them together, you can get a high-precision decompression result.

[0218] In addition, in order to verify the effectiveness of the double-precision spatiotemporal scientific data compression method proposed in the present invention, specific experiments were conducted on the compression rates of low-precision compression and high-precision compression. The experimental process is as follows:

[0219] The present invention uses 6 data sets, which are 4 types of variables from the HYCOM data set, with different data ranges, areas, and time lengths to simulate different usage scenarios as much as possible, as shown in Table 1.

[0220] Table 1 Experimental dataset

[0221]

[0222] Among them, the long time series data (YBT, YSE) screened some areas, the specific range is , Because the data in this area changes dramatically over time, it can reflect the dynamic changes of long time series.

[0223] It should be noted that although the data sets selected by the present invention are all scalar field data, the method of the present invention not only supports scalar field data. In fact, vector field data can be easily converted into scalar field data by vector decomposition (for example, decomposing water flow velocity into u direction and v direction).

[0224] The present invention uses the mean absolute error (MAE) to measure the accuracy of low-precision compression rate and the maximum absolute error (MaxAE) to measure the accuracy of high-precision compression. Both indicators are common in marine science and compression technology.

[0225] In order to compare the accuracy on data sets of different scales, the normalized mean absolute error (NMAE) and the range-based maximum absolute error bound (REL) are used, which are calculated as follows:

[0226] NMAE= ,REL= ;

[0227] The Data Range is the difference between the maximum value and the minimum value of the data. The present invention uses the compression ratio (CR) to measure the compression effect. The compression ratio (CR) is calculated as the ratio of the data size before compression to the data size after compression.

[0228] The low-precision compression and high-precision compression in this invention are compared with the four currently popular scientific data compression methods:

[0229] Comparison method 1:

[0230] SZ2.1: It combines the improved Lorenzo prediction method and the optimized linear regression method to compress data.

[0231] Comparison method 2:

[0232] SZ3: This is the latest work in the SZ series, proposing a modular and composable compression framework.

[0233] Comparison method 3:

[0234] ZFP: A compact classical compressor for multidimensional floating point and integer arrays.

[0235] Comparison method 4:

[0236] SPERR: A lossy compressor for scientific data (primarily 2D or 3D floating-point data generated by numerical simulations).

[0237] First, the low-precision compression method of the present invention is compared with four traditional scientific data compression methods.

[0238] This experiment explores the accuracy (with MAE as an indicator) and CR performance of the low-precision compression method proposed in this invention on different data sets, and compares it with the above-mentioned baseline methods. Since the method of this invention has a fixed compression rate but not fixed accuracy, in order to be fair, this invention controls the MAE of other methods to be as close as possible to that of the method of this invention.

[0239] The difference between the SZ3, SZ2.1, and SPERR methods and the low-precision compression method proposed in this invention is less than 0.01, while the ZFP method has a larger difference because it cannot be precisely controlled. Fig. 9 As shown in the graph, the method of the present invention has achieved significant advantages in the CR of low-precision compression, reaching more than three times that of the second-best method SZ3 on the GBT, GST, GSE, YBT, and YSE data sets, and 2.8 times that of SZ3 on the GSS data set, indicating that the method of the present invention can more effectively reduce the data size of MSD compared with the traditional method to improve the performance of massive data transmission.

[0240] In addition, through data analysis, it is found that when the data range spans a larger range, the MAE is usually larger. Among the 6 data sets tested by the present invention, the maximum data range is 36.60, the corresponding MAE is 0.211336, and the NMAE is 0.6%<1%, which still meets the objectives of the present invention.

[0241] Secondly, the high-precision compression method of the present invention is compared with four traditional scientific data compression methods.

[0242] This experiment explores the accuracy (with MaxAE as the index) and CR performance of the high-precision compression method in the present invention on different data sets, and compares it with the baseline method. Unlike low-precision compression, the high-precision compression method can control the error limit to 0.001 (that is, the maximum absolute error does not exceed 0.001, which is exceeded by the ZFP method because it cannot be precisely controlled).

[0243] The experimental results are as follows Fig.10 As shown, the high-precision compression method in the present invention still has significant advantages in compression rate compared with traditional methods, reaching the best on 5 data sets and the second best on 1 data set. The present invention analyzes the data set GSS that does not reach the best, and finds that this is because the data set generally has large values ​​(the mean of the data is 34.47), which leads to large residual values, and more data exceeds the range of [-1,1] and needs to be stored separately.

[0244] In summary, compared with the traditional spatiotemporal scientific data compression method, the low-precision compression method for spatiotemporal scientific data based on neural networks proposed in the present invention can effectively utilize the spatiotemporal autocorrelation of ocean data, thereby achieving an ultra-high compression rate, which can exceed 278% of the current optimal method. The present invention also proposes a novel high-precision compression method based on probability modeling + entropy coding, and its compression rate can also achieve the best. In addition, compared with traditional single-precision compression, the double-precision compression proposed in the present invention enables the compression algorithm to flexibly adapt to complex downstream tasks, thereby improving the turnover efficiency of the overall process.

[0245] Example 2

[0246] This embodiment 2 describes a spatiotemporal scientific data compression system, which is based on the same inventive concept as the spatiotemporal scientific data compression method described in the above embodiment 1.

[0247] Specifically, the spatiotemporal scientific data compression system in this embodiment includes the following modules:

[0248] Partitioning module, used to divide the time and space dimensions of the spatiotemporal scientific data to be compressed;

[0249] Low-precision compression and decompression module, used to build a low-precision compression and decompression model including feature encoder and reconstruction decoder;

[0250] The feature encoder is used to process the divided data, extract the temporal features and spatial features, and directly store them as a low-precision compressed bit stream; the reconstruction decoder reconstructs the estimated data based on the extracted temporal features and spatial features;

[0251] and a high-precision compression module, which is used to build a high-precision compression model based on a probabilistic decoder;

[0252] The difference between the divided data and the reconstructed estimated data is defined as the data residual;

[0253] The extracted temporal features and spatial features are input into a probability decoder to obtain parameters of a mixed Gaussian distribution; according to the mixed Gaussian distribution, the data residual is converted into a compressed residual bit stream using an arithmetic encoder;

[0254] Finally, the low-precision compressed bit stream is concatenated with the compressed residual bit stream to form a high-precision compressed bit stream.

[0255] It should be noted that in the spatiotemporal scientific data compression system in this embodiment 2, the implementation process of the functions and effects of each functional module is specifically described in the implementation process of the corresponding steps in the method in the above embodiment 1, and will not be repeated here.

[0256] Example 3

[0257] like Figure 8 As shown, this embodiment 3 proposes a low-precision spatiotemporal scientific data compression method based on a neural network and a low-precision compression strategy to achieve maximum compression rate while maintaining a visually acceptable error range.

[0258] The difference from the above-mentioned embodiment 1 is that the data compression method of this embodiment only executes the low-precision compression part, and does not execute the decompression of the low-precision compression and the high-precision compression process. The rest of the contents can refer to the above-mentioned embodiment 1.

[0259] like Figure 8 As shown, the low-precision spatiotemporal scientific data compression method in this embodiment includes the following steps:

[0260] Step 1. Divide the time and space dimensions of the spatiotemporal scientific data to be compressed;

[0261] Step 2. Build a low-precision compression model based on the feature encoder; use the feature encoder to process the divided data, extract temporal and spatial features, and directly store them as a low-precision compressed bit stream.

[0262] The spatiotemporal scientific data compression method based on neural network and low-precision compression strategy described in the present invention can effectively utilize the spatiotemporal autocorrelation of ocean data to achieve an ultra-high compression rate, and its compression rate can exceed 278% of the current optimal method.

[0263] Example 4

[0264] This embodiment 4 describes a spatiotemporal scientific data compression system, which is based on the same inventive concept as the spatiotemporal scientific data compression method described in the above embodiment 3.

[0265] Specifically, the spatiotemporal scientific data compression system in this embodiment includes the following modules:

[0266] Partitioning module, used to divide the time and space dimensions of the spatiotemporal scientific data to be compressed;

[0267] The low-precision compression module is used to build a low-precision compression model including a feature encoder; the feature encoder is used to process the divided data, extract temporal features and spatial features, and directly store them as a low-precision compressed bit stream.

[0268] It should be noted that in the data compression system of this embodiment 4, the implementation process of the functions and effects of each functional module is specifically described in the implementation process of the corresponding steps in the method of the above embodiment 1, and will not be repeated here.

[0269] Example 5

[0270] This embodiment 5 describes a computer device, which includes a memory and one or more processors. An executable code is stored in the memory. When the processor executes the executable code, it is used to implement the spatiotemporal scientific data compression method in the above embodiment 1 or 3.

[0271] In this embodiment, the computer device is any device or apparatus with data processing capability, which will not be described in detail here.

[0272] Example 6

[0273] This embodiment 6 describes a computer-readable storage medium on which a program is stored. When the program is executed by a processor, it is used to implement the spatiotemporal scientific data compression method in the above-mentioned embodiment 1 or 3.

[0274] The computer-readable storage medium may be an internal storage unit of any device or apparatus with data processing capabilities, such as a hard disk or memory, or an external storage device of any device with data processing capabilities, such as a plug-in hard disk, a smart media card (SMC), an SD card, a flash card, etc., equipped on the device.

[0275] Of course, the above description is only a preferred embodiment of the present invention, and the present invention is not limited to the above embodiments. It should be noted that all equivalent substitutions and obvious deformation forms made by any technician familiar with the field under the guidance of this specification fall within the essential scope of this specification and should be protected by the present invention.

Claims

1. A method for compressing spatiotemporal scientific data, characterized in that: The steps include: Step 1: Divide the time and space dimensions of the spatiotemporal scientific data to be compressed; Among them, the spatiotemporal scientific data are ocean scientific data in continuous floating-point array format; Firstly, based on the meta-inspired time partitioning algorithm, the time series of spatiotemporal scientific data are divided into several intervals according to the similarity of time features, and each interval is expressed using a time feature. After dividing the spatiotemporal scientific data into time dimensions, we then divide them into space dimensions; Step 2. Use the feature encoder to process the divided data, extract the temporal and spatial features, and directly store them as a low-precision compressed bit stream; the feature encoder includes a feature extraction module FE e and vector quantization module FE vq ; Given a partitioned data x, it is represented as a time series x = {x (1) ,x (2) ,…,x (t) },in w and h represent the width and height of the divided data, and t represents the time length of the divided data; For each two-dimensional spatial data x at a time point (i) , first use FE e Extract spatial features. In the feature space, each point in the space is expressed by a feature vector, and we get: z = {z (1) ,z (2) ,…,z (t) }; in, Indicates that it corresponds to x (i) The feature vector of , d represents the dimension of each vector; Then through FE vq For feature z (i) Perform vector quantization to obtain discrete spatial features z s (i) and time characteristics z t ; Finally, the discrete spatial feature z s (1) ,z s (2) ,…,z s (t) and time characteristics z t Stored as a low-precision compressed bit stream b low .

2. The spatiotemporal scientific data compression method according to claim 1, characterized in that: In step 2, after the divided data is processed by the feature encoder to extract the time feature and the space feature, the estimated data is further reconstructed by the reconstruction decoder based on the extracted time feature and the space feature; After step 2, the method further comprises: Step 3. The difference between the divided data and the reconstructed estimated data is defined as the data residual; The extracted temporal features and spatial features are input into a probability decoder to obtain parameters of a mixed Gaussian distribution; according to the mixed Gaussian distribution, the data residual is converted into a compressed residual bit stream using an arithmetic encoder; Finally, the low-precision compressed bit stream is combined with the compressed residual bit stream to form a high-precision compressed bit stream.

3. The spatiotemporal scientific data compression method according to claim 1 or 2, characterized in that: In step 1, the spatial dimension is divided, and the spatial partitioning strategy is to cut the data divided by the time dimension into slices of the same size, and at the same time perform reflection filling on the edges of the slices.

4. The spatiotemporal scientific data compression method according to claim 2, characterized in that: In step 2, the process of reconstructing the estimated data by the reconstruction decoder is as follows: First, from the low-precision compressed bit stream b low Get the discrete spatial feature z s (1) ,z s (2) ,…,z s (t) and time characteristics z t , and then transform them into z = {z (1) ,z (2) ,…,z (t) }; Set z = {z (1) ,z (2) ,…,z (t) } Input to the reconstruction decoder to obtain the reconstructed estimated data 5. The spatiotemporal scientific data compression method according to claim 2, characterized in that: In step 3, it is assumed that the data residual r obeys n comp The Gaussian mixture distribution GMM component is then output through the probability decoder PD. The parameters of the Gaussian mixture distribution GMM, namely the mean α, standard deviation μ, and weight σ; For a residual data r, it takes the value r i The probability density of comp The component GMM is expressed as follows: where α i , μ i , σ i Respectively represent the mean, standard deviation and weight of the i-th component; Assume that the quantization level is H, and all quantized data are q1, q2, ..., q H ; For each quantized data q i , the probability density is: where a i and b i They are [q i-1 ,q i ] and [q i ,q i+1 ]; Assume that the cumulative distribution functions of f1 and f2 are F1 and F2 respectively, where F1 is known and is obtained based on the relationship between the Gaussian density function and the distribution function and f1, and F2 is calculated using F1. The expression formula is: The probability decoder is used to estimate the parameters of the GMM given the data features z at a certain time point.

6. The spatiotemporal scientific data compression method according to claim 2, characterized in that: In step 3, in order to accurately control the maximum error and improve the compression rate, quantization and entropy coding are performed only on the data residuals in the interval [-1, 1], and the data residuals outside [-1, 1] are stored separately and completely.

7. The spatiotemporal scientific data compression method according to claim 1, characterized in that: In step 1, the process of dividing the spatiotemporal scientific data according to the similarity of the temporal characteristics is as follows: Given a spatiotemporal scientific dataset X = {X1,X2,...,X T }; Where X i Represents the data at the i-th time point, i=1,2,…,T; The spatiotemporal scientific dataset is divided into M time segments, each of which contains t i time points, namely: set up represents the kth time segment; Where k = 1, 2, ..., M, t k represents the time series of the kth time segment; The optimization objective of the meta-inspiration-based time partitioning algorithm is expressed as: For each segment S k , its internal temporal autocorrelation ξ k Use the following formula to measure: Where R ij is the Pearson product-power correlation coefficient, C ij It is fragment S k Inner sample X i With X j The covariance matrix between k is the number of time series of segments, C ii and C jj Respectively represent X i and X i Between, X j and X j The covariance matrix between ; Suppose two different time series are S m and S n , S m and S n The correlation between m,n As shown below: Here, R ij The formula is the same as above R ij The formula is the same, except that R ij C ij For fragment S m Inner sample X i With fragment S n Inner sample X j The covariance matrix between m ,t n Respectively represent the time series of the mth and nth segments; Set the minimum time series length threshold S, that is, specify any time series t i ≥S; Among them, λ, S, and M are hyperparameters used to balance different principles; Based on the optimization problem of the optimization target, the meta-heuristic algorithm effectively explores and finds the approximate optimal solution in the search space, that is, obtains unequally spaced time partition data subsets and completes the time dimension partition.

8. A computer device comprising a memory and one or more processors; an executable code is stored in the memory; characterized in that: When the processor executes the executable code, it is used to implement the spatiotemporal scientific data compression method described in any one of claims 1 or 2.

9. A computer-readable storage medium having a program stored thereon; characterized in that: When the program is executed by a processor, it is used to implement the spatiotemporal scientific data compression method described in any one of claims 1 or 2.

Citation Information

Patent Citations

  • Storage medium and data processing method

    CN111666708A

  • Feature space context video compression method and system based on optical flow guidance

    CN116684622A

  • Image subjective optimization coding and decoding method

    CN117036515A

  • Image compression device, image decompression device, image compression / Decompression device, image compression method, image decompression method, program, and recording medium recording the program

    JP2003339047A