Method and device for compressing multi-dimensional data based on column storage and self-adaption

Abstract

The invention discloses a method and equipment for compressing multi-dimensional data based on column storage and self-adaption, and the method comprises the steps: recombining original multi-dimensional data into a column data set structure, and organizing data block storage for the column data set structure according to a hierarchical structure; each data block stores observation grid data of a single variable of multi-dimensional data at a certain moment so as to support column type storage and self-adaptive compression; and carrying out real-time statistical analysis on characteristics or parameters of variables in the data blocks, and dynamically selecting and configuring a filter and parameters thereof to compress the multi-dimensional data. The data of the same variable are stored together through column storage, so that the data access efficiency is greatly improved, especially during variable-level analysis. Meanwhile, the self-adaptive compression technology can dynamically select a compression mode according to the distribution characteristics of the data, so that the compression rate of the data is effectively improved, and the occupied storage space is reduced. The processing speed of the meteorological grid point data can be obviously improved, and the storage cost is reduced.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of satellite, more particularly, relates to a method and device for storing and compressing multi-dimensional data based on column storage and self-adaptation. BACKGROUND

[0002] Meteorological grid data is the core carrier for monitoring atmospheric and surface meteorological elements in the field of satellite remote sensing, which refers to a time-space data set formed by discretizing temperature, humidity, wind speed and other elements to a regular grid point. It has distinct structural characteristics: high dimension and multi-variable characteristics are prominent, including time, variable, longitude and latitude, etc. For example, single-satellite overflight data can cover multiple spectral channels, multiple observation times and thousands of grid points. The time-space resolution is high (kilometer-level grid points, minute-level step), the data volume of a single time is large and the adjacent data are strongly correlated. The data redundancy is obvious, with high correlation between spatial and spectral dimensions and great compression potential. Some variables (such as cloud detection) have sparse structure, and some regions (such as high-temperature fire points) have extreme value mutations, which need special processing.

[0003] Traditional meteorological grid data processing methods mostly use row storage, which has low data access efficiency and poor compression effect when processing high-dimensional and multi-variable data.

[0004] Current processing of meteorological grid data in the field of satellite remote sensing mainly relies on two types of traditional HDF5 and GeoTIFF schemes, but both have obvious limitations. Traditional HDF5 is a powerful general scientific data container with a hierarchical data model and rich programming interfaces, which can effectively organize multi-dimensional data sets. However, it uses a static passive compression strategy, all parameters need to be preset before data writing, cannot be dynamically adjusted with content, and users need to have expert experience to configure reasonably. GeoTIFF, as a standard in the field of geographic information and remote sensing, realizes geographic reference storage by embedding geographic labels in TIFF, has strong interoperability and supports most professional software. However, it is essentially a two-dimensional image format, and when processing multi-variable and multi-time sequence multi-dimensional remote sensing data, it needs to use the strategy of "one file for each variable and each time point", which leads to extreme fragmentation of data, greatly increases the difficulty and efficiency loss of file management, transmission and query, and is difficult to adapt to the efficient processing needs of large-scale meteorological grid data.

[0005] Therefore, it is urgent to overcome the technical defects of the prior art. SUMMARY

[0006] In view of the above defects or improvement needs of the prior art, the present application provides a method and device for multi-dimensional data based on columnar storage and adaptive compression, which aims to solve the efficiency bottleneck of the traditional meteorological grid data processing mode in dealing with high-dimensional and multi-variable data by deep integration of columnar storage architecture and adaptive compression algorithm, and overcome the technical problems of lack of flexibility of static compression strategy and serious data fragmentation in the prior art.

[0007] To achieve the above object, according to one aspect of the present application, a method for multi-dimensional data based on columnar storage and adaptive compression is provided, which comprises:

[0008] reorganizing the original multi-dimensional data into a columnar data set structure, and storing the data blocks in a hierarchical structure for the columnar data set structure;

[0009] each data block stores the observation grid data of a single variable of the multi-dimensional data at a certain time, to support columnar storage and adaptive compression;

[0010] by real-time statistical analysis of the characteristics or parameters of the variables in the data block, dynamically selecting and configuring filters and their parameters to compress the multi-dimensional data.

[0011] As a further improvement and supplement to the above scheme, the present application further comprises the following additional technical features.

[0012] Preferably, the method of reorganizing the original multi-dimensional data into a columnar data set structure comprises:

[0013] classifying the data blocks according to the types of variables, and storing the data of the same variable in the same columnar data set.

[0014] Preferably, the method of storing the data blocks in a hierarchical structure for the columnar data set structure comprises:

[0015] the columnar data set stores files according to time points, i.e. one columnar data set stores the data of a variable at different time points.

[0016] Preferably, the two dimensions of the data block are fixed variables and time.

[0017] Preferably, the method of real-time analysis of the statistical characteristics of the grid data in the data block comprises:

[0018] configuring the corresponding filter chain based on the category of the variable in the data block;

[0019] and / or, calculating the preset indicators of each data block, and selecting the corresponding filter chain according to the calculation results.

[0020] Preferably, the method further comprises:

[0021] When receiving the user query, the dimension information in the query condition is parsed, which triggers the index query of variables, time and space, locates to the file containing the target data, and obtains the path of the corresponding columnar data set and the block index in the file;

[0022] According to the file path, data set path and block index, the corresponding data block on the storage device and the filter chain applied when creating the data block are located;

[0023] Reverse decompression and de-filtering are performed to restore the compressed byte stream to the original numerical value;

[0024] According to the user's demand or pre-configuration, the original numerical value is converted into a preset output format, and finally returned to the user for subsequent analysis or visualization.

[0025] Preferably, the preset indicators at least include:

[0026] One or more combinations of zero value density, spatial gradient entropy, value range dynamic range or autocorrelation coefficient.

[0027] Preferably, the categories of variables in the data block at least include:

[0028] One of the following: continuous variable data with strong spatial continuity, categorical variable or mask variable high sparsity data, and data with strong multivariate correlation.

[0029] Preferably, the method further comprises:

[0030] The index metadata in the original multidimensional data is stored in an external index database and is associated with spatial index, variable index and time index;

[0031] The index metadata includes one or more combinations of the following: latitude and longitude boundaries of the data block, variable attribute information, and timestamp.

[0032] To achieve the above-mentioned purposes, according to another aspect of the present application, a columnar storage and adaptive compression multidimensional data processing device is provided, the device comprising:

[0033] One or more processors;

[0034] A storage device for storing one or more programs, when the one or more programs are executed by the one or more processors, so that the one or more processors implement the columnar storage and adaptive compression multidimensional data processing method as described in the first aspect.

[0035] Overall, the above technical solutions conceived by the present application have the following beneficial effects compared with the prior art:

[0036] Columnar storage stores data of the same variable together, greatly improving the efficiency of data access, especially when performing variable-level analysis. At the same time, adaptive compression technology can dynamically select the compression method according to the distribution characteristics of the data, effectively improving the data compression rate and reducing the occupation of storage space. In practical application, this method can significantly improve the processing speed of meteorological grid data and reduce the storage cost, providing strong technical support for meteorological monitoring in the field of satellite remote sensing. BRIEF DESCRIPTION OF DRAWINGS

[0037] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings needed to be used in the embodiments of the present application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor on the premise of not departing from the concept of the present application.

[0038] Figure 1 is a method for storing and compressing multi-dimensional data based on columnar storage and adaptive compression provided by the first embodiment of the present application;

[0039] Figure 2 is a device for storing and compressing multi-dimensional data based on columnar storage and adaptive compression provided by the second embodiment of the present application. DETAILED DESCRIPTION

[0040] In order to make the purpose, technical solutions and advantages of the present application more clear, the present application will be further described in detail below in combination with the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and do not limit the present application. In addition, the technical features involved in each embodiment of the present application described below can be combined with each other as long as there is no conflict between them.

[0041] Traditional HDF5 has many significant shortcomings in the field of large-scale satellite remote sensing data compression, from storage efficiency, query experience to use threshold multi-dimensional constraints data value play: in compression efficiency, it adopts rigid passive global compression strategy, requires users to preset unified compression algorithm and parameters for the entire large data set before data writing, this "one size fits all" mode completely ignores the internal heterogeneity of satellite data - continuous land / sea surface temperature, sparse distribution of precipitation rate, strong correlation of sea surface / atmospheric wind field and other data types should be significantly different in the best compression strategy, but the traditional scheme cannot be adjusted flexibly, and the final compression effect is dragged down by the "most difficult to compress" part of the data set, the overall compression rate is difficult to optimize, causing a large amount of storage space waste, and the storage cost of PB-level satellite remote sensing data is high; At the same time, slow query speed is another core defect, traditional HDF5 lacks native intelligent indexing mechanism for geographic space, time and specific variables, when users need to query data in a specific area or time, the system cannot locate the data position in advance, and can only adopt "needle in a haystack" type full file scanning, which not only reads the entire large file, but also needs to be completely decompressed before extracting a small amount of target data. In this process, more than 99% of I / O bandwidth and computing resources are consumed on irrelevant data processing, resulting in a sharp increase in query response time from milliseconds to minutes or even hours, which cannot support interactive data analysis and real-time application requirements; More importantly, its configuration process is extremely complex and highly dependent on expert experience, users must manually preset the entire set of key configurations such as block shape and size, compression algorithm, compression level and precision control parameters before compressing data, and the block strategy needs to accurately predict the future main data access mode in advance, the compression level needs to be finely balanced between compression rate and speed, and for highly heterogeneous satellite remote sensing data, it is also necessary to understand the spatial characteristics of data under different weather systems to develop a reasonable scheme, which essentially requires users to have professional compression knowledge and data awareness, which greatly raises the technical use threshold and further limits its application in more extensive satellite remote sensing data processing scenarios.

[0042] Embodiment one:

[0043] The embodiment one provides a method for storing and adaptively compressing multi-dimensional data based on columnar storage, as shown in Figure 1 The method comprises the following steps:

[0044] S101: reorganize the original multi-dimensional data into a columnar data set structure, and store the data blocks according to the hierarchical structure for the columnar data set structure.

[0045] The first embodiment uses HDF5 as the core storage format, and reorganizes the original multi-dimensional data into a dataset structure in the HDF5 file according to the columnar storage idea. The types of columns are distinguished according to variables v, time t, latitude y, and longitude x, for example. Each HDF5 file uses a hierarchical group structure to organize data, which is consistent with the multi-dimensional characteristics of satellite remote sensing data. In the HDF5 file, a column block is implemented as a chunked dataset, which is stored as an independent dataset in the HDF5 file.

[0046] S102: Each data block stores the observation grid data of a single variable of multi-dimensional data at a certain time, to support columnar storage and adaptive compression.

[0047] A column block is a physical storage unit of a subset of the original multi-dimensional data space (variables v, time t, latitude y, and longitude x). It fixes the two dimensions of variables v and time t, and contains the data of the entire observation area (all latitudes y and longitudes x) of the variable at that time. From the data structure, a column block is a two-dimensional floating-point array or a quantized integer array.

[0048] All column blocks belonging to the same variable are physically organized together to form a "column family". For example, all column blocks representing the near-infrared band, regardless of their timestamps, are stored and managed together. The great advantage of this approach is that it greatly improves data locality. Data of the same variable are stored in the same collection. HDF5 and the file system will try to store these related collections in physically contiguous or close disk areas. When reading all variable A data, the storage system can sequentially and minimally jump to read a large amount of continuous data, maximizing I / O efficiency and greatly reducing data access time. This is particularly effective in scenarios where data of a variable at consecutive times is obtained.

[0049] S103: By analyzing the distribution characteristics of the variables in the data block in real time, dynamically selecting and configuring filters and their parameters to compress the multi-dimensional data.

[0050] In the first embodiment, the built-in and third-party registered compression filters of HDF5 are intelligently matched with the physical characteristics of satellite remote sensing data. By dynamically configuring the filter pipeline when creating the HDF5 dataset, adaptive compression of different characteristic data blocks is achieved.

[0051] In combination with the embodiments of the present application, there is also a preferred implementation scheme, specifically, the filter chain preset based on variable categories is selected. Due to the differences in physical characteristics and scientific meanings, satellite remote sensing data presents complex characteristics such as self-organization, high dimension and multi-scale, which requires that the compression mechanism must be able to intelligently adapt to the internal structure and scientific value of different variables. The adaptive compression mechanism based on the HDF5 filter pipeline realizes this goal by pre-configuring differentiated filter chains for different categories of variables.

[0052] The filter and its parameters are configured and set through the HDF5 attribute list interface when creating an HDF5 dataset. The system intelligently selects the most suitable compression strategy according to the scientific characteristics and precision requirements of the variables, ensuring the usability of scientific data and maximizing compression efficiency.

[0053] In combination with the embodiments of the present application, there is also a preferred implementation scheme, specifically, block-level statistical analysis and dynamic decision-making are selected.

[0054] Before the data is actually written into the HDF5 dataset, the system will perform real-time statistical analysis at the block level to provide the basis for dynamic decision-making. This process calculates a series of key indicators for each data block, such as zero value density, spatial gradient entropy (used to quantify spatial continuity), value range dynamic range, and autocorrelation coefficient, etc.

[0055] Spatial gradient entropy and autocorrelation coefficient are highly related to satellite observation grid data. Regular grid data obtained by satellite observation usually has strong spatial continuity, that is, the numerical value of adjacent grid points changes smoothly. Spatial gradient entropy can effectively quantify spatial continuity by calculating the distribution of horizontal / vertical gradient values, and the lower the entropy value, the smoother the data. Autocorrelation coefficient, such as Pearson correlation coefficient, directly measures the statistical correlation between adjacent pixels. Both of these indicators are classic statistical quantities that describe spatial structure characteristics, and are very suitable for analyzing the compression characteristics of satellite grid data and providing decision-making basis for adaptive selection of compression algorithms.

[0056] The output of these analyses is no longer directed to custom codecs, but is converted into a set of explicit HDF5 filter chain identifiers and corresponding parameter sets, such as assigning "ZSTD Level 3" configuration to a cloud detection data block with high zero value density.

[0057] This dynamic decision-making mechanism ensures that each data block can apply the most suitable compression algorithm for its data characteristics, thereby maximizing compression efficiency while ensuring data quality. In practical applications, this adaptive compression strategy significantly reduces the storage space occupation, while maintaining the key characteristics of the data, providing high-quality data support for subsequent meteorological analysis and visualization.

[0058] In combination with the embodiment of the present application, there is also a preferred implementation scheme, specifically, the method for reorganizing the original multi-dimensional data into a columnar data set structure comprises:

[0059] The data blocks are classified according to the types of variables, and the data of the same variable are stored in the same columnar data set.

[0060] In combination with the embodiment of the present application, there is also a preferred implementation scheme, specifically, the method for storing the data blocks in a hierarchical structure according to the columnar data set structure comprises:

[0061] The columnar data set stores files according to time points, that is, a columnar data set stores the data of a variable at different time points in a block. The traditional HDF5 file is stored in a row, and when reading the data of a single variable at multiple time points, different data sets need to be accessed by jumping in the traditional HDF5 file, and the I / O efficiency is low. Embodiment I no longer stores all data in a large traditional HDF5 file, and embodiment I creates an independent physical file for each variable, and in the independent file of each variable, the structure of the internal HDF5 data set is organized and blocked according to the time dimension. For an HDF5 file, there is only one layer of time dimension, and the variable dimension is taken out of the HDF5 file.

[0062] In combination with the embodiment of the present application, there is also a preferred implementation scheme, specifically, the two dimensions of the fixed variable and the time in the data block.

[0063] In combination with the embodiment of the present application, there is also a preferred implementation scheme, specifically, the method for analyzing the statistical characteristics of the grid data in the data block in real time comprises:

[0064] Based on the category of the variable in the data block, a corresponding filter chain is configured;

[0065] And / or, a preset index of each data block is calculated, and a corresponding filter chain is selected according to the calculation result.

[0066] In combination with the embodiment of the present application, there is also a preferred implementation scheme, specifically, the method further comprises:

[0067] When receiving a user query, the dimension information in the query condition is analyzed, the dimension information triggers the index query of the variable, the time and the space, the file containing the target data is located, and the path of the corresponding columnar data set in the file and the block index are obtained;

[0068] According to the file path, the data set path and the block index, the corresponding data block on the storage device and the filter chain applied when the data block is created are located;

[0069] Performing reverse decompression and de-filtering to restore the compressed byte stream to the original numerical values;

[0070] According to the user's needs or pre-configuration, the original numerical values are converted into a preset output format, and finally returned to the user for subsequent analysis or visualization.

[0071] In combination with the embodiments of the application, there is also a preferred implementation scheme, specifically, the preset index at least includes:

[0072] One or more combinations of zero value density, spatial gradient entropy, value range dynamic range, or autocorrelation coefficient.

[0073] In the first embodiment, before the multi-dimensional data is actually written into the HDF5 dataset, the system performs real-time statistical analysis at the block level to provide a basis for dynamic decision-making.

[0074] In combination with the embodiments of the application, there is also a preferred implementation scheme, specifically, the categories of variables in the data block at least include:

[0075] One of the following: continuous variable data with strong spatial continuity, categorical variables, or high sparsity data of mask variables, and data with strong multivariate correlation.

[0076] In combination with the embodiments of the application, there is also a preferred implementation scheme, specifically, the method further includes:

[0077] Storing the index metadata in the original multi-dimensional data in an external index database, and associating it with spatial index, variable index, and time index;

[0078] The index metadata includes, but is not limited to, one or more combinations of the following: latitude and longitude boundaries of the data block, variable attribute information, and time stamp.

[0079] In the first embodiment, the decompression and query strategy based on HDF5 transparent decompression is the core of making full use of the efficient data management capabilities and hierarchical structure of the HDF5 library itself, accurately positioning the data block through multi-level indexing, and relying on the built-in filter pipeline of HDF5 to realize automatic decompression of compressed data, and finally returning the processing result to the user. The whole process aims to efficiently respond to complex queries and minimize unnecessary decompression calculations and I / O overhead.

[0080] Index query and data location are the starting point of the whole process. When the system receives a user query, it parses the key dimension information in the query condition, such as variable name, time range, and geographic spatial boundary. The parsed parameters trigger multi-level index linkage query: use variable index to quickly narrow down the data range, time index to quickly filter out the data file list that meets the time condition, and spatial index to further accurately locate the specific HDF5 file containing the target data according to the latitude and longitude range, and obtain the path of the corresponding data set in the file and the specific chunk index, directly locking the smallest physical unit that stores the required data.

[0081] After accurately obtaining the data block position information, the process enters the HDF5 on-demand transparent decompression phase. The application program calls the efficient read function of the HDF5 library, directly locates the specific data block on the storage device based on the file path, data set path, and chunk index obtained in the previous step. The HDF5 library automatically identifies the filter chain applied when creating the data block, which may include byte rearrangement, scalarization, differential encoding, and lossless compression, etc.

[0082] Subsequently, the HDF5 library transparently performs reverse decompression and de-filtering operations in sequence to restore the compressed byte stream to the original numerical array. This process is completely transparent to the application program, and developers do not need to write complex decompression code or concern about the specific parameters of the underlying compression algorithm, significantly improving development efficiency and program reliability. For large-scale data access, the system supports parallel prefetching mechanism, which can concurrently initiate multiple H5Dread_chunk requests, utilize the parallel I / O capability of modern multi-core processors and HDF5 library to simultaneously read and decompress multiple data blocks, thereby greatly shortening the overall data acquisition time.

[0083] Finally, the data reorganization and return phase. Since the user's query range may span multiple consecutive column blocks, it is necessary to splice multiple two-dimensional sub-grid data decompressed from different data blocks in memory according to their inherent spatial range (such as latitude and longitude index) to assemble a complete and continuous data array that accurately matches the user's requested spatial area. After assembly, the system will convert the original data array in memory to the specified output format according to the user's requirements or pre-configuration, and finally return it to the user for subsequent analysis or visualization.

[0084] Compared with traditional HDF5 compression schemes, the advantages of this embodiment in satellite remote sensing data processing are as follows.

[0085] 1. High degree of intelligence and automation, reducing the use of threshold: the traditional HDF5 scheme requires users to manually predefine all compression parameters (such as compression algorithm type, level, block size), which requires users to have deep knowledge and technical background of satellite remote sensing data. Especially in the scene of processing complex and variable data characteristics such as fine structure of typhoon eye area, anvil cloud at the top of strong convective cloud system, or spectral characteristics of specific ground features (such as water, vegetation, and fire area), it is difficult for users to make globally optimal configuration based on experience. However, the adaptive decision tree system of the embodiment I can automatically analyze the characteristics of each data block and dynamically configure the HDF5 filter chain, greatly reducing the technical threshold, enabling non-professional users to achieve near-optimal compression effect, and improving the convenience of HDF5 use.

[0086] 2. Higher compression efficiency: the static compression strategy adopted by traditional HDF5 (once set, applied to the entire data set) lacks the necessary flexibility. Satellite remote sensing data often shows obvious spatio-temporal and spectral heterogeneity (for example, high sparsity of cloud detection bands and high correlation of multispectral data), and the compression characteristics of data under different regions, variables, and ground cover are significantly different. The static compression scheme cannot adapt to such internal changes, which may result in poor compression effect of part of the data set. The embodiment I can dynamically match and configure the most suitable HDF5 filter combination for data blocks of different characteristics (such as selecting a chain that efficiently handles zero values for sparse cloud detection data, and selecting a chain that utilizes spatial redundancy for continuous temperature field) based on data block characteristics, achieving more balanced and efficient overall compression effect, and effectively solving the challenge of internal heterogeneity of satellite remote sensing data to compression.

[0087] 3. Fast query speed: the embodiment I adopts columnar storage idea to improve the locality of storage, and builds an external multi-dimensional index system to solve the slow query speed defect of traditional scheme. By establishing an independent relational database index, the system records the precise spatial range, timestamp, and variable attribute of each data block. When the user submits a query, the system first performs millisecond-level precise positioning in the index database, directly locks a small amount of related data blocks, and then only reads and decompresses these specific data blocks. This mode avoids unnecessary I / O overhead, greatly improving query response speed.

[0088] Embodiment II:

[0089] The embodiment II provides a multi-dimensional data processing device based on columnar storage and adaptive compression, as shown in Figure 2 The device comprises:

[0090] one or more processors;

[0091] a storage device for storing one or more programs, when executed by one or more processors, cause the one or more processors to implement the method for processing multi-dimensional data based on columnar storage and adaptive compression as in Embodiment One.

[0092] Figure 2 A structure diagram of the equipment for processing multi-dimensional data based on columnar storage and adaptive compression is provided for Embodiment Two. Figure 2 An exemplary equipment block diagram for processing multi-dimensional data based on columnar storage and adaptive compression is shown, which is suitable for implementing the embodiments of the present application. Figure 2 The equipment for processing multi-dimensional data based on columnar storage and adaptive compression shown is merely an example, and should not bring any limitation to the function and use range of the embodiments of the present application.

[0093] As shown in Figure 2 The equipment for processing multi-dimensional data based on columnar storage and adaptive compression is in the form of a general-purpose device. The components of the equipment for processing multi-dimensional data based on columnar storage and adaptive compression can include, but are not limited to, one or more processors or processing units, a memory, and a bus connecting different system components, including the memory and the processing unit.

[0094] The bus represents one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, a graphics acceleration port, a processor or local bus using any of a variety of bus architectures. By way of example, these architectures include Industry Standard Architecture (ISA) bus, Micro Channel Architecture (MCA) bus, Enhanced ISA bus, Video Electronics Standards Association (VESA) local bus, and Peripheral Component Interconnect (PCI) bus.

[0095] The equipment for processing multi-dimensional data based on columnar storage and adaptive compression typically includes a number of computer system readable media. These media can be any available media that can be accessed by the intelligent well interpretation model modification device and includes both volatile and nonvolatile media, removable and non-removable media.

[0096] The memory can include computer system readable media in the form of volatile memory, such as random access memory (RAM) and / or cache memory. The equipment for processing multi-dimensional data based on columnar storage and adaptive compression can further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, storage system can be used for reading from and writing to non-removable, non-volatile magnetic media (e.g. Figure 2 not shown, commonly referred to as a "hard disk drive"). Although Figure 2A disk drive can also be provided for reading from or writing to a removable nonvolatile magnetic disk (e.g., a "floppy disk"), and an optical disk drive can be provided for reading from or writing to a removable nonvolatile optical disk (e.g., a CD-ROM, DVD-ROM, or other optical media). In these instances, each drive can be connected to the bus by one or more data media interfaces. The memory can include at least one program product having a set (e.g., at least one) of program modules that are configured to carry out the functions of embodiments of the application.

[0097] Program / utility, having a set (at least one) of program modules, can be stored in memory, for example, having a set (at least one) of program modules that are configured to carry out the functions of embodiments of the application. The program modules can include, but are not limited to, an operating system, one or more application programs, other program modules, and program data, each or some combination thereof, likely including implementation of a network environment. The various program modules can perform functions and / or methodologies described in embodiments of the application.

[0098] The columnar storage and adaptive compression multi-dimensional data processing device can also communicate with one or more external devices such as a keyboard, a pointing device, a display, etc. through input / output (I / O) interface(s). Furthermore, the columnar storage and adaptive compression multi-dimensional data processing device can communicate with one or more devices such as a printer or plotter, a computer, or the like, via an output interface(s). As well, the columnar storage and adaptive compression multi-dimensional data processing device can communicate with one or more devices such as a keyboard, a pointing device, a display, etc. through input / output (I / O) interface(s). In this regard, the columnar storage and adaptive compression multi-dimensional data processing device can implement any one or more of numerous Figure 2 As such, the network adapter can include a digital signal processor (DSP), a respective modem, and numerous other like components. The columnar storage and adaptive compression multi-dimensional data processing device can communicate with one or more devices over a network in accordance with a transmission control protocol (TCP), a user datagram protocol (UDP), a hypertext transfer protocol (HTTP), a file transfer protocol (FTP), a simple mail protocol (SMTP), a wireless application protocol (WAP), etc. In this regard, the network adapter can include a digital signal processor (DSP), a respective modem, and numerous other like components.

[0099] The processing unit can execute a set of instructions that are stored in the memory to perform various functions as described in embodiments of the application. In this regard, the memory can include data storage media having patterns of magnetically or optically changeable patterns or other physical patterns that are written to by an elastic computing device, such as a magnetic hard disk drive, a floppy disk drive, a CD-ROM drive, a DVD-ROM drive, a Blu-ray drive, a flash memory drive, etc.

[0100] The foregoing is considered as illustrative only of the principles of the application. Further, since numerous modifications and changes will readily occur to those skilled in the art, it is not desired to limit the application to the exact construction and operation described. Accordingly, all such variations are intended to fall within the scope of the application.

Claims

1. A method for storing and adaptively compressing multi-dimensional data based on columnar storage, the method comprising: The method comprises: reorganizing the original multidimensional data into a columnar data set structure, and storing data blocks in a hierarchical structure for the columnar data set structure; each data block stores the observation grid data of a single variable of the multidimensional data at a time to support columnar storage and adaptive compression; dynamically selecting and configuring filters and their parameters for the multidimensional data compression by analyzing the characteristics or parameters of the variables in the data block in real time.

2. The method for columnar storage based and adaptive compression of multi-dimensional data as claimed in claim 1, wherein, The method of reorganizing the original multidimensional data into a columnar data set structure comprises: classifying the data blocks according to the types of variables, and storing the data of the same variable in the same columnar data set.

3. The method for columnar storage based and adaptive compression of multi-dimensional data as claimed in claim 2, wherein, The method of storing data blocks in a hierarchical structure for the columnar data set structure comprises: The columnar data set stores files according to time points, that is, a columnar data set stores the data of a variable at different time points.

4. The method for columnar storage based and adaptive compression of multi-dimensional data as claimed in claim 3, wherein, The data block has two dimensions of fixed variables and time.

5. The method for columnar storage based and adaptive compression of multi-dimensional data as claimed in claim 1, wherein, The method of analyzing the statistical characteristics of the grid data in the data block in real time comprises: configuring the corresponding filter chain based on the category of the variable in the data block; and / or, calculating the preset indicators of each data block, and selecting the corresponding filter chain according to the calculation result.

6. The method for columnar storage based and adaptive compression of multi-dimensional data as claimed in claim 1, wherein, The method further comprises: After receiving a user query, parse the dimension information in the query condition, which triggers the index query of variables, time and space, locates the file containing the target data, and obtains the path of the corresponding columnar data set and the block index in the file; According to the file path, data set path and block index, locate the corresponding data block on the storage device and the filter chain applied when creating the data block; Perform reverse decompression and de-filtering to restore the compressed byte stream to the original numerical value; According to the user's demand or pre-configuration, convert the original numerical value into a preset output format, and finally return it to the user for subsequent analysis or visualization.

7. The method for columnar storage based and adaptive compression of multi-dimensional data as claimed in claim 5, wherein, The preset indicators at least include: one or more combinations of zero value density, spatial gradient entropy, value range dynamic range or autocorrelation coefficient.

8. The method for columnar storage based and adaptive compression of multi-dimensional data as claimed in claim 5, wherein, The category of the variable in the data block at least includes: one of continuous variable data with strong spatial continuity, categorical variable or mask variable high sparsity data, and data with strong multivariate correlation.

9. The method for columnar storage based and adaptive compression of multi-dimensional data as claimed in claim 1, wherein, The method further comprises: store the index metadata in the original multidimensional data in an external index database, and associate it with spatial index, variable index and time index; The index metadata includes but is not limited to one or more combinations of latitude and longitude boundaries of the data block, variable attribute information, and timestamp.

10. A multidimensional data processing device based on columnar storage and adaptive compression, characterized in that the device... It comprises: one or more processors; storage device for storing one or more programs, when the one or more programs are executed by the one or more processors, the one or more processors implement the method of storing and compressing multidimensional data based on columnar storage and adaptive compression according to any one of claims 1 to 9.

Images