Compression method and compression program

Abstract

Provided is a compression method executed by a compression device 1, wherein the following are carried out in the compression method: a step S1 in which a data group observed at a plurality of positions is divided by area, and a plurality of data items included in each area are scanned in a zigzag manner and aligned such that data items at nearby positions are continuous; a step S3 in which a plurality of areas having the same or similar area attributes are grouped together, and a plurality of aligned data strings included in the plurality of grouped areas are integrated; and a step S4 in which the integrated data strings are compressed by XOR delta encoding.

Description

Compression method and compression program

[0001] The present disclosure relates to a compression method and a compression program.

[0002] A series of software called Earth System Models (ESMs) is used to predict climate change and analyze climate phenomena. This series of software consists of atmospheric circulation models (AGCMs), ocean circulation models (OGCMs), and land surface models (LSMs).

[0003] The results calculated by ESM contain not only observational data but also three-dimensional spatial and temporal information such as latitude, longitude, and height, and are large in size with high resolution in both time and space, so data writing limits the calculation speed.

[0004] In the meteorological field, the NetCDF (Network Common Data Form) file format is widely used. The NetCDF format also compresses data before writing, reducing data volume and improving writing speed (see Non-Patent Documents 1-5).

[0005] “What is NetCDF?” [online] [searched April 18, 2024], <URL: https: / / docs.unidata.ucar.edu / netcdf-c / current / > “14 Chunking and Deflating Data with NetCDF-4”, [online] [searched April 18, 2024], <URL: https: / / www.unidata.ucar.edu / software / netcdf / workshops / 2011 / nc4chunking / > “9.1 Components of a NetCDF File”, [online] [searched April 18, 2024], <URL: https: / / www.gfd-dennou.org / arch / netcdf / netcdf-jman / guidec / netCDF_ja80.html> “NetCDF Compression Tools”, [online] [searched April 18, 2024], <URL: http: / / climate-cms.wikis.unsw.edu.au / NetCDF_Compression_Tools>“Chunking Data: Why it Matters”, [online], [Retrieved April 18, 2020], <URL: https: / / www.unidata.ucar.edu / blogs / developer / entry / chunking_data_why_it_matters>

[0006] However, the NetCDF format uses general-purpose compression algorithms (e.g., GZIP and ZSTD) interchangeably, which poses challenges in both data compression speed and data writing speed. In other words, the use of a general-purpose compression method poses challenges in terms of compression ratio. Furthermore, adopting a method that increases the compression ratio increases the compression processing time.

[0007] The present disclosure has been made in consideration of the above circumstances, and an object of the present disclosure is to provide a technology that can compress data at high speed and reduce the time required to write data.

[0008] A compression method according to one aspect of the present disclosure is a compression method performed by a compression device, and includes a first step of dividing a group of data observed at multiple locations into areas and aligning the multiple data contained in each area by zigzag scanning so that data located in close proximity are continuous; a second step of grouping multiple areas having the same or similar area attributes and integrating the aligned data strings contained in the grouped multiple areas; and a third step of compressing the integrated data strings using XOR delta encoding.

[0009] A compression program according to one aspect of the present disclosure causes a computer to execute the compression method.

[0010] According to the present disclosure, it is possible to provide a technology that can compress data at high speed and reduce the time required to write data.

[0011] Fig. 1 is a diagram showing the functional block configuration of a compression device. Fig. 2 is a diagram showing an example of scanning measurement data. Fig. 3 is a diagram showing an example of merging square areas. Fig. 4 is a diagram showing the compression flow of observation data. Fig. 5 is a diagram showing the hardware configuration of a compression device.

[0012] Hereinafter, embodiments of the present disclosure will be described with reference to the drawings. In the description of the drawings, the same parts are designated by the same reference numerals and the description thereof will be omitted.

[0013] [Summary of the Disclosure] The present disclosure relates to the field of scientific computing for analyzing climate phenomena using models to predict future climate change due to global warming, etc. The disclosure also relates to the storage of global-scale observation data used for such analysis.

[0014] This disclosure provides a compression technology that writes globally acquired calculation / observation data at high speed and in low volume. In other words, we propose a method that can reduce data volume while proceeding with compression processing at high speed and shortening data writing time. This includes both reductions by reducing data volume and by reducing the compression time required for that purpose.

[0015] [Configuration of Compression Apparatus] FIG. 1 is a diagram showing the functional block configuration of a compression apparatus 1 according to this embodiment.

[0016] The compression device 1 compresses observation data observed at multiple locations on Earth (on a world map). The observation data includes, for example, temperature, humidity, wind speed, thermal radiation, evapotranspiration, precipitation, snowfall, and wave height. The observation data includes three-dimensional spatial information (latitude, longitude, and altitude) of the observation location and time information of the observation time.

[0017] The compression device 1 includes an integration unit 11 that integrates observation data, a compression unit 12 that compresses the integrated observation data, and a storage unit 13 that stores the compressed observation data.

[0018] The integration unit 11 has a function of dividing a group of observation data observed at multiple positions into square areas, and zigzag-scanning and aligning the multiple observation data contained in each square area so that observation data at nearby positions are continuous.

[0019] For example, when generating a data string from within one rectangular area, the integration unit 11 scans and aligns multiple observation data in a zigzag manner so that two adjacent data in the data string have similar values ​​and so that as many zeros as possible line up when XORing.

[0020] The integrating unit 11 has a function of grouping a plurality of rectangular areas having the same and / or similar area attributes and integrating a plurality of sorted observation data strings included in the grouped plurality of rectangular areas. Area attributes include, for example, climate classification and land use type.

[0021] The compression unit 12 has a function of compressing the integrated observation data string using XOR delta encoding and storing the compressed observation data string in the storage unit 13. The integrated observation data string is a single unit (chunk) for which compression processing is performed.

[0022] The storage unit 13 has a function of storing, for example, the compressed observation data string, setting data in which similarity information of area attributes is set, and various data required for compression.

[0023] [Method of Compressing Observation Data] First, a method of compressing observation data will be described in advance.

[0024] In this embodiment, the observed data is compressed using an encoding method called delta encoding using XOR. Hereinafter, this encoding method will be referred to as XOR'd encoding.

[0025] XOR'd encoding has the property that when floating-point numbers with similar values ​​are XORed, the most significant bits of the sign, exponent, and mantissa tend to become 0. Floating-point numbers are a form of numerical representation used in computers, and each number is expressed by combining a sign that indicates whether it is positive or negative, a mantissa that indicates the order of digits, and an exponent that indicates the position of the decimal point.

[0026] For example, if there are four consecutive numbers, 12.0, 24.0, 15.0, and 12.0, the value obtained by XORing the first and second numbers in order from the beginning is as follows.

[0027] 12.0^24.0: 00000000000100000000000000000000000000000000000000000000000000000 24.0^15.0: 00000000000101100000000000000000000000000000000000000000000000000 15.0^12.0: 00000000000001100000000000000000000000000000000000000000000000000000 "^" means XOR.

[0028] The leading zeros are called leading zeros, the non-zero areas are called meaningful bits, and the trailing zeros are called trailing zeros. In the XOR of 12.0^24.0, the first 11 zeros are leading zeros, the ones after that are meaningful bits, and all the zeros that follow are trailing zeros.

[0029] The previous value, 12.0, is not encoded because it is the first value. The next value, 24.0, has only significant bits, the meaningful bits 1, so the remaining 0 can be encoded. The next value, 15.0, has only significant bits, the meaningful bits 1011, so the remaining 0 can be encoded. Therefore, the leading and trailing zeros, which are 0, can be encoded.

[0030] Next, we will explain the encoding method of XOR'd encoding. The value is encoded in the following order using the XOR of the two values ​​before and after.

[0031] If the XOR with the previous value is 0 (= same as the previous value), write 0.

[0032] If the XOR with the previous value is not 0 (= different from the previous value), first write 1. Then, if the meaningful bits are the same as the meaningful bits of the previous value, write 0 and then write the meaningful bits.

[0033] On the other hand, if the meaningful bits are different from the meaningful bits of the previous value, write a 1, then write the next three in order: the number of leading zeros in 5 bits, the length of the meaningful bits in 6 bits, and the meaningful bits themselves.

[0034] The above encoding method will be explained using the above example.

[0035] The value 12.0 is written as is because it is the first value, i.e. 12.0 is not encoded.

[0036] The value 24.0 is XORed with the previous value (= 12.0), and since it is different from the previous value, a 1 is written. After that, the meaningful bits of the previous value are checked, and since they are not the same, a 1 is written. After that, the number of leading 0 bits (11) is written in 5 bits. After that, the length of the meaningful bits (1) is written in 6 bits. Finally, the meaningful bit itself (1) is written.

[0037] The value 15.0 is XORed with the previous value (= 24.0), and since it is different from the previous value, a 1 is written. After that, the meaningful bits of the previous value are checked, and since they are not the same, a 1 is written. After that, the number of leading 0 bits (11) is written in 5 bits. After that, the length of the meaningful bits (4) is written in 6 bits. Finally, the meaningful bits themselves (1011) are written.

[0038] As a result, if the data type handled by the computer is a 64-bit double type, a total of 95 bits of data capacity (64 bits + 14 bits + 17 bits) is sufficient to write the first three numbers.

[0039] In this way, only two values, the previous and next values, are required for encoding, and since append processing is performed, high-speed encoding (i.e., compression) is possible. In this embodiment, observation data is compressed using XOR'd encoding, which has these characteristics.

[0040] XOR'd encoding is described in "Tuomas Pelkonen and six others, "Gorilla: A Fast, Scalable, In-Memory Time Series Database," [online], [Retrieved April 18, 2024], <URL: https: / / www.vldb.org / pvldb / vol8 / p1816-teller.pdf>."

[0041] [Observation Data Integration Method] Next, the observation data integration method will be described.

[0042] In this embodiment, the observed data is integrated before being compressed by taking advantage of the above-mentioned characteristic that "the XOR of floating-point numbers with similar values ​​tends to result in a result with many zeros."

[0043] In this embodiment, in order to collect observation data having values ​​that are as close as possible, the compression unit (chunk) of the observation data is defined based on the following two characteristics.

[0044] (First characteristic) The first is to collect spatially nearby observation data, based on the fact that spatially nearby observation data are likely to have similar values. For example, it is assumed that precipitation amounts are all zero in non-precipitation areas, with large differences between non-precipitation and precipitation areas, and that precipitation amounts change somewhat continuously within precipitation areas.

[0045] (Second characteristic) The second characteristic is to further gather together observational data that are highly similar to each other among the observational data collected in the first characteristic, taking into account the fact that observational data in the same climate zone, etc., are likely to have similar values. For example, it makes use of characteristics such as the high probability that observational data in areas with the same climate zone or land use pattern have similar values.

[0046] (Alignment of observation data using the first characteristic) Based on the first characteristic, as shown in Fig. 2, the integration unit 11 aligns observation data that are spatially close in the latitude and longitude coordinate system by zigzag scanning. The "circles" shown in Fig. 2 represent observation data at a specific latitude and longitude. The arrows indicate the scanning order of the observation data.

[0047] The square area shown in Figure 2 is the unit of collection of observation data to be included in the same chunk. However, a square area is not necessarily equal to a chunk, but can be part or all of a chunk. In other words, a chunk is a collection of square areas. The vertical and horizontal sizes of the observation data included in a square area can be different or the same.

[0048] Any rectangular area can be created using any of the four corner observation coordinates and the latitude and longitude range. The range can be positive or negative. If the reference point is the lowest latitude and longitude point among the four corners, both ranges will be positive. If the reference point is the highest latitude and longitude point, both ranges will be negative.

[0049] Within the square area, the observation data is scanned in a zigzag pattern as shown in Figure 2, so that the observation data are arranged as close as possible to each other. In other words, taking into account the characteristics of XOR'd encoding described above, multiple observation data are scanned and arranged in a zigzag pattern so that two adjacent values ​​in the data string to be compressed are close to each other, i.e., so that the list of 0s and 1s in the floating-point number sequence is as close as possible.

[0050] The zigzag direction may be longitude->latitude, or longitude->latitude. The scanning direction may also be determined arbitrarily. For example, it may start from the bottom right and end at the top right.

[0051] In the first step, the square areas are arranged so that the entire world map is filled with them like a patchwork. The definition and arrangement information of these square areas can be placed at any coordinate and with any size, and can be stored as a dictionary in the compression algorithm or written as metadata in the file.

[0052] (Integration of Observation Data Using the Second Property) Next, the integration unit 11 generates chunks that collect similar observation data by grouping identical and / or similar rectangular areas, thereby further increasing the compression rate.

[0053] (Integration of Observation Data Using Second Property: Example 1) In Example 1, square areas are grouped using climate classifications.

[0054] To satisfy the second characteristic, the first-stage rectangular area can be roughly set according to the climate zone. However, there are particularly rapid changes in climate zones in western North America and western South America, making it difficult to collect observation data for the same climate zone in the west.

[0055] Therefore, if a square area were treated as a chunk, a series of small chunks would result. The first observation data in a chunk would not be encoded using XOR'd encoding, resulting in overhead.

[0056] Therefore, we group square areas by the same or similar climate zones to generate a single chunk. This reduces the amount of observation data at the beginning, thereby reducing the overhead. However, when grouping square areas, we do not mix areas from the northern and southern hemispheres. This is because the seasons are reversed in the northern and southern hemispheres.

[0057] A method for setting the setting data (metadata) required for processing will be described below. The setting data is set in advance in the storage unit 13 by the user.

[0058] The similar climate zones can be selected arbitrarily. For example, each column of the Köppen climate classification can be divided into groups A, B, Cs, Cw, Cf, Ds, Dw, Df, and E. In an extreme example, if all observations are grouped into the same similarity group, the observations for the entire hemisphere would be one chunk.

[0059] The information on which climate zone each observation data belongs to is predefined. In this case, any definition of climate zone can be used. For the ocean, the climate zone can be set independently from the climate zone for land, or it can simply be set as the climate zone for ocean.

[0060] The climate zone classification of a given square area is determined using the observation data contained in that square area. For example, it is determined by taking a majority vote among the observation data contained in that square area. It is determined by referring to the first land climate classification in the series of observation data. However, this determination method is arbitrary and is not limited. Furthermore, this determination may be performed in advance before compressing the observation data, or may be performed only once when compression of the observation data begins.

[0061] The rectangles belonging to similar climate zones are combined in order of geographic proximity for each hemisphere. This ordering can be done beforehand before compressing the observation data, or it can be done only once when compressing the observation data.

[0062] Collecting observation data that utilizes climate classification is suitable for, for example, precipitation and temperature.

[0063] (Integration of Observation Data Using Second Property: Example 2) In Example 2, rectangular areas are grouped using land use types.

[0064] Any land use type may be assumed. For example, cities, farmland, forests, grasslands, deserts, frozen soil, ice sheets, etc. Two or more land use types may be arbitrarily combined. As in the case of the climate zones described above, rectangular areas of the same or similar land use types are grouped together to generate a single chunk.

[0065] A single rectangular area may contain multiple land use types. In this case, for example, the rectangular areas are connected by aligning the land use type of the last observation data collected in one rectangular area and the land use type to be collected first in another rectangular area. However, any other method may be used.

[0066] Similarity can also be set across land use types. This is a way to deal with cases where there is not enough data when collecting data for only a single land use type. For example, it collects data that is highly similar in terms of the degree of impact of human activity, vegetation, and wetness, such as "urban, rural, farmland" versus "forest, grassland, desert" versus "frozen soil, ice sheet."

[0067] Land use types can be further classified, for example, by further classifying forests into smaller categories, which may have an effect similar to that of climate zonation.

[0068] Collecting observation data that makes use of land use patterns is suitable for measuring, for example, the amount of thermal radiation and evapotranspiration from the ground. Urban land use patterns are heat sources, and radiation tends to be higher than in other areas.

[0069] Although the sea is not technically land, it is not limited to being included in the square area. In other words, the sea is treated as land use on the same level as land, and there can be square areas where land and sea coexist.

[0070] (Further Integration of Observation Data) To further reduce overhead, a single square area or a set of square areas may be further integrated in the time direction, as shown in Figure 3. For example, each square area may be integrated in the time direction and then added to a chunk. Chunks that combine square areas for multiple time periods are prepared, and then these are combined into a single chunk.

[0071] [Observation Data Compression Method] FIG. 4 is a diagram showing a flow of compressing observation data.

[0072] The compression device 1 acquires in advance a group of observation data observed at multiple locations on the earth (for example, temperatures around the world observed every 0.5 seconds on a certain day).

[0073] Step S1: The integration unit 11 divides the group of observation data observed at each position into square areas, and aligns the multiple observation data contained in each square area by zigzag scanning so that observation data at nearby positions are continuous.

[0074] Step S2: The integration unit 11 integrates each square area in the time direction. Step S2 does not have to be executed.

[0075] Step S3: The integration unit 11 reads the setting data from the memory unit 13, and based on the similarity information of the climate zones set in the setting data, groups each square area by hemisphere and by similar climate zone, and generates chunks for each grouped area by integrating the observation data strings of each grouped area into a single row.

[0076] Step S4: The compression unit 12 compresses the observation data set by applying XOR'd encoding to each chunk, and stores the compressed observation data set in the storage unit 13.

[0077] The chunks compressed with XOR'd encoding can be further compressed with a conventional general-purpose compression algorithm. Since the amount of data to be compressed is reduced by XOR'd encoding, and the time required to apply the compression algorithm is also reduced, the compression time is less than the total time required for the conventional general-purpose compression algorithm and the XOR'd encoding.

[0078] [Effects] According to this embodiment, a group of observation data observed at multiple locations is divided into square areas, and the multiple observation data contained in each square area are zigzag scanned and aligned so that observation data at nearby locations are continuous. Multiple square areas with the same or similar area attributes are grouped, and the aligned observation data strings contained in the grouped multiple square areas are integrated. The integrated observation data strings (chunks) are compressed using XOR delta encoding, thereby enabling high-speed compression of the observation data and reducing the time it takes to write the observation data.

[0079] Specifically, observation data is integrated based on spatial characteristics and the characteristics of climate classification and land use type, and the integrated observation data string (chunk) is compressed using XOR delta encoding, which makes it possible to reduce data volume while erasing unnecessary data areas compared to when general-purpose compression algorithms were used.

[0080] Although the definition information for the square area increases as metadata, it is merely snapshot-like definition information, and the data size is extremely small compared to the size of high-resolution spatiotemporal data. The upper limit is the number of measurement data, and the impact decreases as the time resolution and period handled become longer.

[0081] [Others] The present disclosure is not limited to the above-described embodiment. The present disclosure can be modified in many ways within the scope of the gist of the present disclosure. For example, Examples 1 and 2 may be combined.

[0082] The compression device 1 of the present embodiment described above can be realized, for example, by using a general-purpose computer system including a CPU 901, a memory 902, a storage 903, a communication device 904, an input device 905, and an output device 906, as shown in Fig. 5. The memory 902 and the storage 903 are storage devices. In this computer system, the CPU 901 executes a predetermined program loaded onto the memory 902, thereby realizing each function of the compression device 1.

[0083] The compression device 1 may be implemented by one computer, or by multiple computers, or may be a virtual machine implemented on a computer.

[0084] The program for the compression device 1 can be stored in a computer-readable recording medium such as a HDD, SSD, USB memory, CD, or DVD. The computer-readable recording medium is, for example, a non-transitory recording medium. The program for the compression device 1 can also be distributed via a communication network.

[0085] REFERENCE SIGNS LIST 1 Compression device 11 Integration unit 12 Compression unit 13 Storage unit 901 CPU 902 Memory 903 Storage 904 Communication device 905 Input device 906 Output device

Claims

1. A compression method performed by a compression device, comprising: a first step of dividing a group of data observed at multiple locations into areas, and aligning the multiple data contained in each area by zigzag scanning so that data in nearby locations are continuous; a second step of grouping multiple areas with the same or similar area attributes, and integrating the aligned data strings contained in the grouped multiple areas; and a third step of compressing the integrated data string using XOR delta encoding.

2. The compression method according to claim 1, wherein the area attribute is a climate zone or a land use type.

3. A compression method according to claim 1, wherein in the first step, a plurality of data are zigzag scanned and aligned so that two adjacent pieces of data in a data string to be compressed within one area are as close as possible to each other, and the XOR value of the two adjacent pieces of data contains as many zeros as possible.

4. A compression program that causes a computer to execute the compression method according to claim 1.

Images